2025-12-04T08:25:45.0972959Z Current runner version: '2.330.0' 2025-12-04T08:25:45.0978079Z Runner name: 'i-02dcdd5317aed8049' 2025-12-04T08:25:45.0978702Z Runner group name: 'default' 2025-12-04T08:25:45.0979619Z Machine name: 'ip-10-0-12-49' 2025-12-04T08:25:45.0981945Z ##[group]GITHUB_TOKEN Permissions 2025-12-04T08:25:45.0984031Z Contents: read 2025-12-04T08:25:45.0984484Z Metadata: read 2025-12-04T08:25:45.0984870Z ##[endgroup] 2025-12-04T08:25:45.0986697Z Secret source: Actions 2025-12-04T08:25:45.0987305Z Prepare workflow directory 2025-12-04T08:25:45.1411706Z Prepare all required actions 2025-12-04T08:25:45.1445324Z Getting action download info 2025-12-04T08:25:45.4773563Z Download action repository 'pytorch/test-infra@main' (SHA:39aa74d619174326f4e2fb0e216151c2f29d9ffd) 2025-12-04T08:25:48.1337976Z Download action repository 'pytorch/pytorch@main' (SHA:ffd9b0fb4355e97af82fc42cf185c3ffa0fc0a32) 2025-12-04T08:26:04.4389719Z Download action repository 'actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065' (SHA:a26af69be951a213d495a4c3e4e4022e16d87065) 2025-12-04T08:26:04.8643247Z Download action repository 'aws-actions/configure-aws-credentials@ececac1a45f3b08a01d2dd070d28d111c5fe6722' (SHA:ececac1a45f3b08a01d2dd070d28d111c5fe6722) 2025-12-04T08:26:05.1524938Z Download action repository 'aws-actions/amazon-ecr-login@062b18b96a7aff071d4dc91bc00c4c1a7945b076' (SHA:062b18b96a7aff071d4dc91bc00c4c1a7945b076) 2025-12-04T08:26:05.3601818Z Download action repository 'seemethere/download-artifact-s3@1da556a7aa0a088e3153970611f6c432d58e80e6' (SHA:1da556a7aa0a088e3153970611f6c432d58e80e6) 2025-12-04T08:26:05.5917690Z Download action repository 'seemethere/upload-artifact-s3@baba72d0712b404f646cebe0730933554ebce96a' (SHA:baba72d0712b404f646cebe0730933554ebce96a) 2025-12-04T08:26:05.9201778Z Getting action download info 2025-12-04T08:26:06.0418870Z Download action repository 'actions/checkout@v4' (SHA:34e114876b0b11c390a56381ad16ebd13914f8d5) 2025-12-04T08:26:06.2665694Z Getting action download info 2025-12-04T08:26:06.3844138Z Download action repository 'nick-fields/retry@v3.0.0' (SHA:7152eba30c6575329ac0576536151aca5a72780e) 2025-12-04T08:26:06.5708785Z Getting action download info 2025-12-04T08:26:06.7077557Z Download action repository 'nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482' (SHA:3e91a01664abd3c5cd539100d10d33b9c5b68482) 2025-12-04T08:26:06.8738718Z Getting action download info 2025-12-04T08:26:07.0144242Z Uses: pytorch/pytorch/.github/workflows/_linux-test.yml@refs/heads/main (ffd9b0fb4355e97af82fc42cf185c3ffa0fc0a32) 2025-12-04T08:26:07.0147886Z ##[group] Inputs 2025-12-04T08:26:07.0148259Z build-environment: linux-jammy-py3.10-gcc11-build 2025-12-04T08:26:07.0150244Z test-matrix: {"include": [{"config": "cpu_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.2xlarge.amx"}, {"config": "cpu_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.2xlarge.amx"}, {"config": "dynamic_cpu_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.2xlarge.amx"}, {"config": "dynamic_cpu_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.2xlarge.amx"}, {"config": "dynamic_cpu_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.2xlarge.amx"}, {"config": "dynamic_cpu_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.2xlarge.amx"}, {"config": "dynamic_cpu_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.2xlarge.amx"}, {"config": "inductor_torchbench_cpu_smoketest_perf", "shard": 1, "num_shards": 1, "runner": "linux.24xl.spr-metal"}]} 2025-12-04T08:26:07.0152553Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-f0cd68561080d537ef3d3d6f81b25a6416ad600a 2025-12-04T08:26:07.0153241Z sync-tag: 2025-12-04T08:26:07.0153974Z timeout-minutes: 240 2025-12-04T08:26:07.0154201Z use-gha: 2025-12-04T08:26:07.0154438Z dashboard-tag: 2025-12-04T08:26:07.0154702Z s3-bucket: gha-artifacts 2025-12-04T08:26:07.0155245Z aws-role-to-assume: 2025-12-04T08:26:07.0155808Z disable-monitor: false 2025-12-04T08:26:07.0156144Z monitor-log-interval: 5 2025-12-04T08:26:07.0156446Z monitor-data-collect-interval: 1 2025-12-04T08:26:07.0156735Z ##[endgroup] 2025-12-04T08:26:07.0157222Z Complete job name: inductor-cpu-test / test (dynamic_cpu_inductor_huggingface, 1, 1, linux.2xlarge.amx) 2025-12-04T08:26:07.0533706Z A job started hook has been configured by the self-hosted runner administrator 2025-12-04T08:26:07.0612006Z ##[group]Run '/home/ec2-user/runner-scripts/before_job.sh' 2025-12-04T08:26:07.0618887Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T08:26:07.0619327Z ##[endgroup] 2025-12-04T08:26:07.9947972Z Runner Type: linux.2xlarge.amx 2025-12-04T08:26:08.0029914Z Instance Type: m7i-flex.2xlarge 2025-12-04T08:26:08.0030204Z AMI Name: unknown 2025-12-04T08:26:08.0030408Z AMI ID: ami-08982f1c5bf93d976 2025-12-04T08:26:12.3344835Z ##[group]Run pytorch/test-infra/.github/actions/setup-ssh@main 2025-12-04T08:26:12.3345169Z with: 2025-12-04T08:26:12.3345741Z github-secret: *** 2025-12-04T08:26:12.3346209Z instructions: All testing is done inside the container, to start an interactive session run: docker exec -it $(docker container ps --format '{{.ID}}') bash 2025-12-04T08:26:12.3346688Z activate-with-label: false 2025-12-04T08:26:12.3346884Z label: with-ssh 2025-12-04T08:26:12.3347072Z remove-existing-keys: true 2025-12-04T08:26:12.3347268Z fail-silently: true 2025-12-04T08:26:12.3347445Z env: 2025-12-04T08:26:12.3347609Z GIT_DEFAULT_BRANCH: main 2025-12-04T08:26:12.3347822Z ##[endgroup] 2025-12-04T08:26:12.4521553Z Please see https://github.com/pytorch/pytorch/wiki/Debugging-using-with-ssh-for-Github-Actions for more info. 2025-12-04T08:26:12.4522738Z Not on pull request and ciflow reference could not be extracted, skipping adding ssh keys 2025-12-04T08:26:12.4664186Z ##[group]Run pytorch/pytorch/.github/actions/checkout-pytorch@main 2025-12-04T08:26:12.4664520Z with: 2025-12-04T08:26:12.4664706Z no-sudo: true 2025-12-04T08:26:12.4664917Z submodules: recursive 2025-12-04T08:26:12.4665121Z fetch-depth: 0 2025-12-04T08:26:12.4665306Z env: 2025-12-04T08:26:12.4665481Z GIT_DEFAULT_BRANCH: main 2025-12-04T08:26:12.4665683Z ##[endgroup] 2025-12-04T08:26:12.4741261Z ##[group]Run echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2025-12-04T08:26:12.4741915Z echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2025-12-04T08:26:12.4749603Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T08:26:12.4749888Z env: 2025-12-04T08:26:12.4750087Z GIT_DEFAULT_BRANCH: main 2025-12-04T08:26:12.4750321Z ##[endgroup] 2025-12-04T08:26:12.4842234Z ##[group]Run # Use all available CPUs for fetching 2025-12-04T08:26:12.4842588Z # Use all available CPUs for fetching 2025-12-04T08:26:12.4842842Z cd "${GITHUB_WORKSPACE}" 2025-12-04T08:26:12.4843084Z git config --global fetch.parallel 0 2025-12-04T08:26:12.4843371Z git config --global submodule.fetchJobs 0 2025-12-04T08:26:12.4843614Z  2025-12-04T08:26:12.4843865Z # Clean workspace. The default checkout action should also do this, but 2025-12-04T08:26:12.4844260Z # do it here as well just in case 2025-12-04T08:26:12.4844492Z if [[ -d .git ]]; then 2025-12-04T08:26:12.4844704Z  if [ -z "${NO_SUDO}" ]; then 2025-12-04T08:26:12.4844930Z  sudo git clean -ffdx 2025-12-04T08:26:12.4845137Z  else 2025-12-04T08:26:12.4845318Z  git clean -ffdx 2025-12-04T08:26:12.4845506Z  fi 2025-12-04T08:26:12.4845671Z fi 2025-12-04T08:26:12.4850261Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T08:26:12.4850523Z env: 2025-12-04T08:26:12.4850807Z GIT_DEFAULT_BRANCH: main 2025-12-04T08:26:12.4851029Z NO_SUDO: true 2025-12-04T08:26:12.4851197Z ##[endgroup] 2025-12-04T08:26:12.4971808Z ##[group]Run actions/checkout@v4 2025-12-04T08:26:12.4972062Z with: 2025-12-04T08:26:12.4972266Z ref: ffd9b0fb4355e97af82fc42cf185c3ffa0fc0a32 2025-12-04T08:26:12.4972501Z fetch-depth: 0 2025-12-04T08:26:12.4972687Z submodules: recursive 2025-12-04T08:26:12.4972884Z show-progress: false 2025-12-04T08:26:12.4973080Z repository: pytorch/pytorch 2025-12-04T08:26:12.4973384Z token: *** 2025-12-04T08:26:12.4973549Z ssh-strict: true 2025-12-04T08:26:12.4973722Z ssh-user: git 2025-12-04T08:26:12.4973899Z persist-credentials: true 2025-12-04T08:26:12.4974099Z clean: true 2025-12-04T08:26:12.4974298Z sparse-checkout-cone-mode: true 2025-12-04T08:26:12.4974508Z fetch-tags: false 2025-12-04T08:26:12.4974681Z lfs: false 2025-12-04T08:26:12.4974858Z set-safe-directory: true 2025-12-04T08:26:12.4975059Z env: 2025-12-04T08:26:12.4975224Z GIT_DEFAULT_BRANCH: main 2025-12-04T08:26:12.4975424Z ##[endgroup] 2025-12-04T08:26:12.5888206Z Syncing repository: pytorch/pytorch 2025-12-04T08:26:12.5889429Z ##[group]Getting Git version info 2025-12-04T08:26:12.5889787Z Working directory is '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2025-12-04T08:26:12.5890261Z [command]/usr/bin/git version 2025-12-04T08:26:12.6113421Z git version 2.50.1 2025-12-04T08:26:12.6135823Z ##[endgroup] 2025-12-04T08:26:12.6148267Z Copying '/home/ec2-user/.gitconfig' to '/home/ec2-user/actions-runner/_work/_temp/b6945cef-5298-4412-90ff-6fec16575ae2/.gitconfig' 2025-12-04T08:26:12.6169355Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/b6945cef-5298-4412-90ff-6fec16575ae2' before making global git config changes 2025-12-04T08:26:12.6170087Z Adding repository directory to the temporary git global config as a safe directory 2025-12-04T08:26:12.6174976Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2025-12-04T08:26:12.6232875Z Deleting the contents of '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2025-12-04T08:26:12.6234274Z ##[group]Initializing the repository 2025-12-04T08:26:12.6238354Z [command]/usr/bin/git init /home/ec2-user/actions-runner/_work/pytorch/pytorch 2025-12-04T08:26:12.6301093Z hint: Using 'master' as the name for the initial branch. This default branch name 2025-12-04T08:26:12.6301557Z hint: is subject to change. To configure the initial branch name to use in all 2025-12-04T08:26:12.6301945Z hint: of your new repositories, which will suppress this warning, call: 2025-12-04T08:26:12.6302254Z hint: 2025-12-04T08:26:12.6302500Z hint: git config --global init.defaultBranch 2025-12-04T08:26:12.6302742Z hint: 2025-12-04T08:26:12.6302981Z hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and 2025-12-04T08:26:12.6303381Z hint: 'development'. The just-created branch can be renamed via this command: 2025-12-04T08:26:12.6303680Z hint: 2025-12-04T08:26:12.6304009Z hint: git branch -m 2025-12-04T08:26:12.6304227Z hint: 2025-12-04T08:26:12.6304493Z hint: Disable this message with "git config set advice.defaultBranchName false" 2025-12-04T08:26:12.6309575Z Initialized empty Git repository in /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/ 2025-12-04T08:26:12.6316926Z [command]/usr/bin/git remote add origin https://github.com/pytorch/pytorch 2025-12-04T08:26:12.6355969Z ##[endgroup] 2025-12-04T08:26:12.6357289Z ##[group]Disabling automatic garbage collection 2025-12-04T08:26:12.6362162Z [command]/usr/bin/git config --local gc.auto 0 2025-12-04T08:26:12.6388915Z ##[endgroup] 2025-12-04T08:26:12.6389254Z ##[group]Setting up auth 2025-12-04T08:26:12.6393087Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2025-12-04T08:26:12.6416283Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2025-12-04T08:26:12.6797185Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2025-12-04T08:26:12.6830325Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2025-12-04T08:26:12.7171394Z [command]/usr/bin/git config --local --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T08:26:12.7202114Z [command]/usr/bin/git submodule foreach --recursive git config --local --show-origin --name-only --get-regexp remote.origin.url 2025-12-04T08:26:12.7504426Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic *** 2025-12-04T08:26:12.7557124Z ##[endgroup] 2025-12-04T08:26:12.7557522Z ##[group]Fetching the repository 2025-12-04T08:26:12.7567734Z [command]/usr/bin/git -c protocol.version=2 fetch --prune --no-recurse-submodules origin +refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/* 2025-12-04T08:27:06.9914509Z From https://github.com/pytorch/pytorch 2025-12-04T08:27:06.9914944Z * [new branch] 2.6.0.dev20241004+ -> origin/2.6.0.dev20241004+ 2025-12-04T08:27:06.9915613Z * [new branch] 2.9.1 -> origin/2.9.1 2025-12-04T08:27:06.9916160Z * [new branch] AaronWang04_addmmfusion_perftest -> origin/AaronWang04_addmmfusion_perftest 2025-12-04T08:27:06.9916559Z * [new branch] Flamefire-patch-1 -> origin/Flamefire-patch-1 2025-12-04T08:27:06.9917096Z * [new branch] HDCharles-2.6.0-release-notes -> origin/HDCharles-2.6.0-release-notes 2025-12-04T08:27:06.9917616Z * [new branch] HOPrintFunc -> origin/HOPrintFunc 2025-12-04T08:27:06.9918085Z * [new branch] IvanKobzarev/stack/1 -> origin/IvanKobzarev/stack/1 2025-12-04T08:27:06.9918806Z * [new branch] NicoshevSVE128 -> origin/NicoshevSVE128 2025-12-04T08:27:06.9919174Z * [new branch] PR-AOTInductorNoneBug -> origin/PR-AOTInductorNoneBug 2025-12-04T08:27:06.9919558Z * [new branch] PR-AOTInductorNoneBugFix -> origin/PR-AOTInductorNoneBugFix 2025-12-04T08:27:06.9919959Z * [new branch] PR-FixConfigsIssue -> origin/PR-FixConfigsIssue 2025-12-04T08:27:06.9920360Z * [new branch] PR-NoneBugFix-viable -> origin/PR-NoneBugFix-viable 2025-12-04T08:27:06.9920855Z * [new branch] PR-ResetToZero -> origin/PR-ResetToZero 2025-12-04T08:27:06.9921244Z * [new branch] Update-Flash-Packaging -> origin/Update-Flash-Packaging 2025-12-04T08:27:06.9921603Z * [new branch] VLA_exp -> origin/VLA_exp 2025-12-04T08:27:06.9921925Z * [new branch] activation_bench -> origin/activation_bench 2025-12-04T08:27:06.9922259Z * [new branch] addmm-heuristic -> origin/addmm-heuristic 2025-12-04T08:27:06.9925001Z * [new branch] adi/onednn_aarch64 -> origin/adi/onednn_aarch64 2025-12-04T08:27:06.9925337Z * [new branch] adi/test -> origin/adi/test 2025-12-04T08:27:06.9925640Z * [new branch] adi/test_bgemm -> origin/adi/test_bgemm 2025-12-04T08:27:06.9925949Z * [new branch] adi/test_m8g -> origin/adi/test_m8g 2025-12-04T08:27:06.9926251Z * [new branch] adi/test_onednn -> origin/adi/test_onednn 2025-12-04T08:27:06.9926576Z * [new branch] adi/test_onednn_v3.9 -> origin/adi/test_onednn_v3.9 2025-12-04T08:27:06.9926918Z * [new branch] adi/test_presve_change -> origin/adi/test_presve_change 2025-12-04T08:27:06.9927246Z * [new branch] adi/test_timm -> origin/adi/test_timm 2025-12-04T08:27:06.9929620Z * [new branch] adi/testpresve_change -> origin/adi/testpresve_change 2025-12-04T08:27:06.9930011Z * [new branch] aditew01/test/vec_bf16 -> origin/aditew01/test/vec_bf16 2025-12-04T08:27:06.9930520Z * [new branch] ah-globalfeedback-hook -> origin/ah-globalfeedback-hook 2025-12-04T08:27:06.9930943Z * [new branch] albanD-patch-1 -> origin/albanD-patch-1 2025-12-04T08:27:06.9931275Z * [new branch] also-surround-shimh -> origin/also-surround-shimh 2025-12-04T08:27:06.9931636Z * [new branch] angelayi/aot_compile -> origin/angelayi/aot_compile 2025-12-04T08:27:06.9932026Z * [new branch] angelayi/aoti_additional_files -> origin/angelayi/aoti_additional_files 2025-12-04T08:27:06.9935732Z * [new branch] angelayi/benchmark -> origin/angelayi/benchmark 2025-12-04T08:27:06.9936273Z * [new branch] angelayi/change_pytree_serialization -> origin/angelayi/change_pytree_serialization 2025-12-04T08:27:06.9936850Z * [new branch] angelayi/cpp_loader -> origin/angelayi/cpp_loader 2025-12-04T08:27:06.9937338Z * [new branch] angelayi/inductor_const -> origin/angelayi/inductor_const 2025-12-04T08:27:06.9937691Z * [new branch] angelayi/lstm -> origin/angelayi/lstm 2025-12-04T08:27:06.9938019Z * [new branch] angelayi/no_so_weight -> origin/angelayi/no_so_weight 2025-12-04T08:27:06.9938363Z * [new branch] angelayi/scan_layers -> origin/angelayi/scan_layers 2025-12-04T08:27:06.9938699Z * [new branch] angelayi/side_eff -> origin/angelayi/side_eff 2025-12-04T08:27:06.9941488Z * [new branch] angelayi/state_dict -> origin/angelayi/state_dict 2025-12-04T08:27:06.9941914Z * [new branch] angelayi/symint_input -> origin/angelayi/symint_input 2025-12-04T08:27:06.9948402Z * [new branch] angelayi/symm_mem -> origin/angelayi/symm_mem 2025-12-04T08:27:06.9949334Z * [new branch] angelayi/test_cpp -> origin/angelayi/test_cpp 2025-12-04T08:27:06.9949820Z * [new branch] angelayi/torch_size -> origin/angelayi/torch_size 2025-12-04T08:27:06.9950185Z * [new branch] annotate_assert -> origin/annotate_assert 2025-12-04T08:27:06.9950535Z * [new branch] annotate_fallback_kernel -> origin/annotate_fallback_kernel 2025-12-04T08:27:06.9950896Z * [new branch] annotation_deepcopy -> origin/annotation_deepcopy 2025-12-04T08:27:06.9951228Z * [new branch] annotation_dynamo -> origin/annotation_dynamo 2025-12-04T08:27:06.9951562Z * [new branch] aot_eager_stack_trace -> origin/aot_eager_stack_trace 2025-12-04T08:27:06.9951899Z * [new branch] aoti-cuda-alloc -> origin/aoti-cuda-alloc 2025-12-04T08:27:06.9952227Z * [new branch] aoti_const_device -> origin/aoti_const_device 2025-12-04T08:27:06.9952564Z * [new branch] aoti_fqn_name_interface -> origin/aoti_fqn_name_interface 2025-12-04T08:27:06.9952925Z * [new branch] aoti_package_weights_binary -> origin/aoti_package_weights_binary 2025-12-04T08:27:06.9953273Z * [new branch] aoti_target_windows -> origin/aoti_target_windows 2025-12-04T08:27:06.9954442Z * [new branch] arsh/feat/inductor_check_profiling -> origin/arsh/feat/inductor_check_profiling 2025-12-04T08:27:06.9954858Z * [new branch] async_tp -> origin/async_tp 2025-12-04T08:27:06.9955215Z * [new branch] atalman-inductor-perf-cu124 -> origin/atalman-inductor-perf-cu124 2025-12-04T08:27:06.9955776Z * [new branch] atalman-inductor-perf-cu124.1 -> origin/atalman-inductor-perf-cu124.1 2025-12-04T08:27:06.9956286Z * [new branch] atalman-patch-2 -> origin/atalman-patch-2 2025-12-04T08:27:06.9956738Z * [new branch] atalman-patch-3 -> origin/atalman-patch-3 2025-12-04T08:27:06.9957208Z * [new branch] atalman-patch-4 -> origin/atalman-patch-4 2025-12-04T08:27:06.9957873Z * [new branch] atalman-patch-5 -> origin/atalman-patch-5 2025-12-04T08:27:06.9958484Z * [new branch] atalman-patch-6 -> origin/atalman-patch-6 2025-12-04T08:27:06.9959026Z * [new branch] atalman-patch-7 -> origin/atalman-patch-7 2025-12-04T08:27:06.9959550Z * [new branch] atalman-patch-8 -> origin/atalman-patch-8 2025-12-04T08:27:06.9960034Z * [new branch] atalman_inductor_2.3.1 -> origin/atalman_inductor_2.3.1 2025-12-04T08:27:06.9960433Z * [new branch] atalman_inductor_2.4.0 -> origin/atalman_inductor_2.4.0 2025-12-04T08:27:06.9960817Z * [new branch] atalman_inductor_2.4.x -> origin/atalman_inductor_2.4.x 2025-12-04T08:27:06.9961328Z * [new branch] attention_benchmarking_clean -> origin/attention_benchmarking_clean 2025-12-04T08:27:06.9961743Z * [new branch] bahuang/dt_fix_scalar_add -> origin/bahuang/dt_fix_scalar_add 2025-12-04T08:27:06.9962163Z * [new branch] bahuang/fix_debug_mode -> origin/bahuang/fix_debug_mode 2025-12-04T08:27:06.9962508Z * [new branch] bahuang/fix_expand -> origin/bahuang/fix_expand 2025-12-04T08:27:06.9962836Z * [new branch] bahuang/test -> origin/bahuang/test 2025-12-04T08:27:06.9963133Z * [new branch] base/1.5 -> origin/base/1.5 2025-12-04T08:27:06.9965125Z * [new branch] batching_sdpa_efficient_attention -> origin/batching_sdpa_efficient_attention 2025-12-04T08:27:06.9965808Z * [new branch] bench_scaled_mm_ops -> origin/bench_scaled_mm_ops 2025-12-04T08:27:06.9968987Z * [new branch] benchmark-updates -> origin/benchmark-updates 2025-12-04T08:27:06.9969551Z * [new branch] benchmarking-script -> origin/benchmarking-script 2025-12-04T08:27:06.9974083Z * [new branch] bertmaher/pinbump26 -> origin/bertmaher/pinbump26 2025-12-04T08:27:06.9974501Z * [new branch] bertrand/cutlass -> origin/bertrand/cutlass 2025-12-04T08:27:06.9974854Z * [new branch] bf/bug-static-input -> origin/bf/bug-static-input 2025-12-04T08:27:06.9975172Z * [new branch] bf/cg-backend -> origin/bf/cg-backend 2025-12-04T08:27:06.9975482Z * [new branch] bf/cg-nccl-test -> origin/bf/cg-nccl-test 2025-12-04T08:27:06.9975793Z * [new branch] bf/cg-remove-check -> origin/bf/cg-remove-check 2025-12-04T08:27:06.9976134Z * [new branch] bf/clean-torchbench-hf -> origin/bf/clean-torchbench-hf 2025-12-04T08:27:06.9976475Z * [new branch] bf/combo-debug-log -> origin/bf/combo-debug-log 2025-12-04T08:27:06.9976791Z * [new branch] bf/cudagraph -> origin/bf/cudagraph 2025-12-04T08:27:06.9977182Z * [new branch] bf/cudagraph-disable-input-mutation -> origin/bf/cudagraph-disable-input-mutation 2025-12-04T08:27:06.9977775Z * [new branch] bf/cudagraph-enable-input-mutation-support-benchmark -> origin/bf/cudagraph-enable-input-mutation-support-benchmark 2025-12-04T08:27:06.9978289Z * [new branch] bf/cudagraph-partition -> origin/bf/cudagraph-partition 2025-12-04T08:27:06.9978648Z * [new branch] bf/donated-buffer-bench -> origin/bf/donated-buffer-bench 2025-12-04T08:27:06.9978992Z * [new branch] bf/dynamo-partition -> origin/bf/dynamo-partition 2025-12-04T08:27:06.9979298Z * [new branch] bf/lite -> origin/bf/lite 2025-12-04T08:27:06.9979618Z * [new branch] bf/pa-non-divisible -> origin/bf/pa-non-divisible 2025-12-04T08:27:06.9980018Z * [new branch] bf/partition-cache-free-symbols -> origin/bf/partition-cache-free-symbols 2025-12-04T08:27:06.9980433Z * [new branch] bf/partition-memory-plan -> origin/bf/partition-memory-plan 2025-12-04T08:27:06.9980976Z * [new branch] bf/partition-move-cpu -> origin/bf/partition-move-cpu 2025-12-04T08:27:06.9981350Z * [new branch] bf/partition-view-fallback -> origin/bf/partition-view-fallback 2025-12-04T08:27:06.9981741Z * [new branch] bf/remove-check-55b0c39d -> origin/bf/remove-check-55b0c39d 2025-12-04T08:27:06.9982170Z * [new branch] bf/timm-nov-26-2025 -> origin/bf/timm-nov-26-2025 2025-12-04T08:27:06.9982962Z * [new branch] bf/transformer-pin-4-57-3 -> origin/bf/transformer-pin-4-57-3 2025-12-04T08:27:06.9983608Z * [new branch] bisect_perf_hf_T5_3acc6eac492 -> origin/bisect_perf_hf_T5_3acc6eac492 2025-12-04T08:27:06.9984271Z * [new branch] bisect_perf_hf_T5_3fcf66f61fb -> origin/bisect_perf_hf_T5_3fcf66f61fb 2025-12-04T08:27:06.9984942Z * [new branch] bisect_perf_hf_T5_4009d154129 -> origin/bisect_perf_hf_T5_4009d154129 2025-12-04T08:27:06.9985602Z * [new branch] bisect_perf_hf_T5_40d0740e73d -> origin/bisect_perf_hf_T5_40d0740e73d 2025-12-04T08:27:06.9986241Z * [new branch] bisect_perf_hf_T5_5268754e -> origin/bisect_perf_hf_T5_5268754e 2025-12-04T08:27:06.9986821Z * [new branch] bisect_perf_hf_T5_7d89a8d385c -> origin/bisect_perf_hf_T5_7d89a8d385c 2025-12-04T08:27:06.9987650Z * [new branch] bisect_perf_hf_T5_b7a25c1ee7c -> origin/bisect_perf_hf_T5_b7a25c1ee7c 2025-12-04T08:27:06.9988276Z * [new branch] bisect_perf_hf_T5_c25b201583f -> origin/bisect_perf_hf_T5_c25b201583f 2025-12-04T08:27:06.9988966Z * [new branch] bisect_perf_hf_T5_c93e57efac0 -> origin/bisect_perf_hf_T5_c93e57efac0 2025-12-04T08:27:06.9989764Z * [new branch] bisect_perf_hf_T5_ca9813ea149 -> origin/bisect_perf_hf_T5_ca9813ea149 2025-12-04T08:27:06.9990374Z * [new branch] bisect_perf_hf_T5_d65f194a -> origin/bisect_perf_hf_T5_d65f194a 2025-12-04T08:27:06.9991079Z * [new branch] bisect_perf_hf_T5_da94ab0b -> origin/bisect_perf_hf_T5_da94ab0b 2025-12-04T08:27:06.9991751Z * [new branch] bisect_perf_hf_T5_da94ab0b_new -> origin/bisect_perf_hf_T5_da94ab0b_new 2025-12-04T08:27:06.9992423Z * [new branch] bisect_perf_hf_T5_db4e8a1d8a8 -> origin/bisect_perf_hf_T5_db4e8a1d8a8 2025-12-04T08:27:06.9992988Z * [new branch] bisect_perf_hf_T5_e0d97e936a2 -> origin/bisect_perf_hf_T5_e0d97e936a2 2025-12-04T08:27:06.9993660Z * [new branch] bisect_perf_hf_T5_f23621ec563 -> origin/bisect_perf_hf_T5_f23621ec563 2025-12-04T08:27:06.9997473Z * [new branch] brister/fx_device_type -> origin/brister/fx_device_type 2025-12-04T08:27:06.9997988Z * [new branch] brister/test_inductor_all_fx -> origin/brister/test_inductor_all_fx 2025-12-04T08:27:06.9998589Z * [new branch] brister/tiled_reduction_no_numel_check -> origin/brister/tiled_reduction_no_numel_check 2025-12-04T08:27:06.9999058Z * [new branch] bwd-backup -> origin/bwd-backup 2025-12-04T08:27:06.9999394Z * [new branch] c57382a49 -> origin/c57382a49 2025-12-04T08:27:06.9999745Z * [new branch] ca_0431d47eaa -> origin/ca_0431d47eaa 2025-12-04T08:27:07.0000073Z * [new branch] ca_fix_0431d47eaa -> origin/ca_fix_0431d47eaa 2025-12-04T08:27:07.0000580Z * [new branch] camyllh/test_setup_hooks_push -> origin/camyllh/test_setup_hooks_push 2025-12-04T08:27:07.0001401Z * [new branch] cccclai-patch-1 -> origin/cccclai-patch-1 2025-12-04T08:27:07.0002296Z * [new branch] cherry-pick-159969-by-pytorch_bot_bot_ -> origin/cherry-pick-159969-by-pytorch_bot_bot_ 2025-12-04T08:27:07.0002983Z * [new branch] cherry-pick-160586-by-pytorch_bot_bot_ -> origin/cherry-pick-160586-by-pytorch_bot_bot_ 2025-12-04T08:27:07.0003712Z * [new branch] cherry-pick-162208-by-pytorch_bot_bot_ -> origin/cherry-pick-162208-by-pytorch_bot_bot_ 2025-12-04T08:27:07.0004574Z * [new branch] cherry-pick-163169-by-pytorch_bot_bot_ -> origin/cherry-pick-163169-by-pytorch_bot_bot_ 2025-12-04T08:27:07.0005654Z * [new branch] cherry-pick-165086-by-pytorch_bot_bot_ -> origin/cherry-pick-165086-by-pytorch_bot_bot_ 2025-12-04T08:27:07.0006280Z * [new branch] cherry-pick-165514-by-pytorch_bot_bot_ -> origin/cherry-pick-165514-by-pytorch_bot_bot_ 2025-12-04T08:27:07.0007059Z * [new branch] cherry-pick-165601-by-pytorch_bot_bot_ -> origin/cherry-pick-165601-by-pytorch_bot_bot_ 2025-12-04T08:27:07.0007772Z * [new branch] cherry-pick-165667-by-pytorch_bot_bot_ -> origin/cherry-pick-165667-by-pytorch_bot_bot_ 2025-12-04T08:27:07.0008569Z * [new branch] cherry-pick-165815-by-pytorch_bot_bot_ -> origin/cherry-pick-165815-by-pytorch_bot_bot_ 2025-12-04T08:27:07.0009410Z * [new branch] cherry-pick-165922-by-pytorch_bot_bot_ -> origin/cherry-pick-165922-by-pytorch_bot_bot_ 2025-12-04T08:27:07.0010116Z * [new branch] cherry-pick-166148-by-pytorch_bot_bot_ -> origin/cherry-pick-166148-by-pytorch_bot_bot_ 2025-12-04T08:27:07.0010772Z * [new branch] cherry-pick-166181-by-pytorch_bot_bot_ -> origin/cherry-pick-166181-by-pytorch_bot_bot_ 2025-12-04T08:27:07.0011648Z * [new branch] cherry-pick-166404-by-pytorch_bot_bot_ -> origin/cherry-pick-166404-by-pytorch_bot_bot_ 2025-12-04T08:27:07.0012375Z * [new branch] cherry-pick-166427-by-pytorch_bot_bot_ -> origin/cherry-pick-166427-by-pytorch_bot_bot_ 2025-12-04T08:27:07.0013336Z * [new branch] cherry-pick-166480-by-pytorch_bot_bot_ -> origin/cherry-pick-166480-by-pytorch_bot_bot_ 2025-12-04T08:27:07.0013842Z * [new branch] cherry-pick-166570-by-pytorch_bot_bot_ -> origin/cherry-pick-166570-by-pytorch_bot_bot_ 2025-12-04T08:27:07.0014546Z * [new branch] cherry-pick-166993-by-pytorch_bot_bot_ -> origin/cherry-pick-166993-by-pytorch_bot_bot_ 2025-12-04T08:27:07.0015316Z * [new branch] cherry-pick-167111-by-pytorch_bot_bot_ -> origin/cherry-pick-167111-by-pytorch_bot_bot_ 2025-12-04T08:27:07.0016826Z * [new branch] cherry-pick-167478-by-pytorch_bot_bot_ -> origin/cherry-pick-167478-by-pytorch_bot_bot_ 2025-12-04T08:27:07.0017259Z * [new branch] cherry_pick_166036_166040 -> origin/cherry_pick_166036_166040 2025-12-04T08:27:07.0017870Z * [new branch] cherry_pick_166457 -> origin/cherry_pick_166457 2025-12-04T08:27:07.0018617Z * [new branch] cherrypick_166338 -> origin/cherrypick_166338 2025-12-04T08:27:07.0019330Z * [new branch] cherrypick_166458 -> origin/cherrypick_166458 2025-12-04T08:27:07.0020051Z * [new branch] cherrypick_166586 -> origin/cherrypick_166586 2025-12-04T08:27:07.0020927Z * [new branch] cherrypick_166956 -> origin/cherrypick_166956 2025-12-04T08:27:07.0021667Z * [new branch] ci_attn -> origin/ci_attn 2025-12-04T08:27:07.0022328Z * [new branch] codex-testing -> origin/codex-testing 2025-12-04T08:27:07.0027064Z * [new branch] codex/add-check_memory_overlap-helper-functions -> origin/codex/add-check_memory_overlap-helper-functions 2025-12-04T08:27:07.0027701Z * [new branch] codex/fix-issue-121219-in-pytorch -> origin/codex/fix-issue-121219-in-pytorch 2025-12-04T08:27:07.0028314Z * [new branch] codex/investigate-segfaults-in-get_tensor_storage_id -> origin/codex/investigate-segfaults-in-get_tensor_storage_id 2025-12-04T08:27:07.0029019Z * [new branch] codex/refactor-lintrunner-config-to-use-uv-run -> origin/codex/refactor-lintrunner-config-to-use-uv-run 2025-12-04T08:27:07.0029536Z * [new branch] compatiblpy39util -> origin/compatiblpy39util 2025-12-04T08:27:07.0030160Z * [new branch] cond_hop_device -> origin/cond_hop_device 2025-12-04T08:27:07.0030497Z * [new branch] context_test -> origin/context_test 2025-12-04T08:27:07.0030937Z * [new branch] copilot/code-style-cleanup-python-pip -> origin/copilot/code-style-cleanup-python-pip 2025-12-04T08:27:07.0031394Z * [new branch] cpio/fix_new_ami_tests -> origin/cpio/fix_new_ami_tests 2025-12-04T08:27:07.0031814Z * [new branch] cpp-docs-dependency-upgrade -> origin/cpp-docs-dependency-upgrade 2025-12-04T08:27:07.0032743Z * [new branch] csl/always_produce_xml -> origin/csl/always_produce_xml 2025-12-04T08:27:07.0033208Z * [new branch] csl/build_test_more_procs -> origin/csl/build_test_more_procs 2025-12-04T08:27:07.0033641Z * [new branch] csl/build_test_more_procs2 -> origin/csl/build_test_more_procs2 2025-12-04T08:27:07.0034417Z * [new branch] csl/clean_up -> origin/csl/clean_up 2025-12-04T08:27:07.0035242Z * [new branch] csl/fix_retry_segfault_exit -> origin/csl/fix_retry_segfault_exit 2025-12-04T08:27:07.0035908Z * [new branch] csl/katex -> origin/csl/katex 2025-12-04T08:27:07.0037073Z * [new branch] csl/larger_runner -> origin/csl/larger_runner 2025-12-04T08:27:07.0037793Z * [new branch] csl/lint_testing -> origin/csl/lint_testing 2025-12-04T08:27:07.0038965Z * [new branch] csl/lint_thing -> origin/csl/lint_thing 2025-12-04T08:27:07.0039456Z * [new branch] csl/lintrunner_stuff -> origin/csl/lintrunner_stuff 2025-12-04T08:27:07.0040630Z * [new branch] csl/manually_gen_json -> origin/csl/manually_gen_json 2025-12-04T08:27:07.0041152Z * [new branch] csl/mps_sharding -> origin/csl/mps_sharding 2025-12-04T08:27:07.0044001Z * [new branch] csl/multistage_docker -> origin/csl/multistage_docker 2025-12-04T08:27:07.0044393Z * [new branch] csl/print_timing -> origin/csl/print_timing 2025-12-04T08:27:07.0044747Z * [new branch] csl/remove_experiment -> origin/csl/remove_experiment 2025-12-04T08:27:07.0045137Z * [new branch] csl/remove_maybe_unused_var -> origin/csl/remove_maybe_unused_var 2025-12-04T08:27:07.0045572Z * [new branch] csl/remove_repo_specific_autolabel -> origin/csl/remove_repo_specific_autolabel 2025-12-04T08:27:07.0046005Z * [new branch] csl/remove_run_parallel -> origin/csl/remove_run_parallel 2025-12-04T08:27:07.0046379Z * [new branch] csl/remove_unused_vars -> origin/csl/remove_unused_vars 2025-12-04T08:27:07.0051059Z * [new branch] csl/revert_open -> origin/csl/revert_open 2025-12-04T08:27:07.0051382Z * [new branch] csl/skip_build -> origin/csl/skip_build 2025-12-04T08:27:07.0051720Z * [new branch] csl/smaller_avx_amx_runenrs -> origin/csl/smaller_avx_amx_runenrs 2025-12-04T08:27:07.0052053Z * [new branch] csl/td_job_level -> origin/csl/td_job_level 2025-12-04T08:27:07.0052397Z * [new branch] csl/test_cuda_build_large_runner -> origin/csl/test_cuda_build_large_runner 2025-12-04T08:27:07.0052813Z * [new branch] csl/test_owners_autograd_dispatch_nn -> origin/csl/test_owners_autograd_dispatch_nn 2025-12-04T08:27:07.0053244Z * [new branch] csl/test_owners_higher_confidence -> origin/csl/test_owners_higher_confidence 2025-12-04T08:27:07.0053625Z * [new branch] csl/upload_json_running -> origin/csl/upload_json_running 2025-12-04T08:27:07.0053955Z * [new branch] csl/win_sccache -> origin/csl/win_sccache 2025-12-04T08:27:07.0056481Z * [new branch] csl/xml_stuff -> origin/csl/xml_stuff 2025-12-04T08:27:07.0056885Z * [new branch] cublasrelax2 -> origin/cublasrelax2 2025-12-04T08:27:07.0057191Z * [new branch] cuda_mempool -> origin/cuda_mempool 2025-12-04T08:27:07.0057513Z * [new branch] custom_lowering_dict -> origin/custom_lowering_dict 2025-12-04T08:27:07.0057873Z * [new branch] d4l3k/debug_plane_frtrace -> origin/d4l3k/debug_plane_frtrace 2025-12-04T08:27:07.0058199Z * [new branch] daxia6/2.8o3 -> origin/daxia6/2.8o3 2025-12-04T08:27:07.0058514Z * [new branch] debug-guard -> origin/debug-guard 2025-12-04T08:27:07.0061422Z * [new branch] delete-quant-docs -> origin/delete-quant-docs 2025-12-04T08:27:07.0062041Z * [new branch] dependabot/pip/dot-ci/docker/ci_commit_pins/main/transformers-4.57.0 -> origin/dependabot/pip/dot-ci/docker/ci_commit_pins/main/transformers-4.57.0 2025-12-04T08:27:07.0062798Z * [new branch] dependabot/pip/dot-ci/docker/ci_commit_pins/main/transformers-4.57.1 -> origin/dependabot/pip/dot-ci/docker/ci_commit_pins/main/transformers-4.57.1 2025-12-04T08:27:07.0063397Z * [new branch] desertfire/test_cpp_wrapper -> origin/desertfire/test_cpp_wrapper 2025-12-04T08:27:07.0063850Z * [new branch] desertfire/triton-cpu-for-aarch64 -> origin/desertfire/triton-cpu-for-aarch64 2025-12-04T08:27:07.0066133Z * [new branch] dev/dhruva/flex_attn_opt -> origin/dev/dhruva/flex_attn_opt 2025-12-04T08:27:07.0066541Z * [new branch] dev/joona/MPSNDArrayAdd -> origin/dev/joona/MPSNDArrayAdd 2025-12-04T08:27:07.0066911Z * [new branch] dev/joona/Unranked -> origin/dev/joona/Unranked 2025-12-04T08:27:07.0067322Z * [new branch] dev/joona/cat -> origin/dev/joona/cat 2025-12-04T08:27:07.0071611Z * [new branch] dev/joona/embeddingbag -> origin/dev/joona/embeddingbag 2025-12-04T08:27:07.0072118Z * [new branch] dev/joona/fix_sdpa_memtest -> origin/dev/joona/fix_sdpa_memtest 2025-12-04T08:27:07.0072548Z * [new branch] dev/joona/getTensorsString -> origin/dev/joona/getTensorsString 2025-12-04T08:27:07.0073000Z * [new branch] dev/joona/mps_linear_macos14 -> origin/dev/joona/mps_linear_macos14 2025-12-04T08:27:07.0073415Z * [new branch] dev/joona/scalar_clamp -> origin/dev/joona/scalar_clamp 2025-12-04T08:27:07.0073791Z * [new branch] dev/joona/sdpa -> origin/dev/joona/sdpa 2025-12-04T08:27:07.0075670Z * [new branch] dev/joona/sdpa_api -> origin/dev/joona/sdpa_api 2025-12-04T08:27:07.0076121Z * [new branch] dev/joona/type_inf -> origin/dev/joona/type_inf 2025-12-04T08:27:07.0076542Z * [new branch] dev/joona/ulpAssertClose -> origin/dev/joona/ulpAssertClose 2025-12-04T08:27:07.0076938Z * [new branch] dev/joona/upsize3d -> origin/dev/joona/upsize3d 2025-12-04T08:27:07.0077295Z * [new branch] disp_counter -> origin/disp_counter 2025-12-04T08:27:07.0077702Z * [new branch] divyanshk-patch-1 -> origin/divyanshk-patch-1 2025-12-04T08:27:07.0078207Z * [new branch] docs -> origin/docs 2025-12-04T08:27:07.0078558Z * [new branch] documentation -> origin/documentation 2025-12-04T08:27:07.0078959Z * [new branch] eager_model_benchmarks -> origin/eager_model_benchmarks 2025-12-04T08:27:07.0079413Z * [new branch] embg/test_inductor_ci_control -> origin/embg/test_inductor_ci_control 2025-12-04T08:27:07.0080112Z * [new branch] embg/triton_l2_prefetch_128B -> origin/embg/triton_l2_prefetch_128B 2025-12-04T08:27:07.0080705Z * [new branch] embg/triton_l2_prefetch_256B -> origin/embg/triton_l2_prefetch_256B 2025-12-04T08:27:07.0082336Z * [new branch] eqy-patch-1 -> origin/eqy-patch-1 2025-12-04T08:27:07.0083254Z * [new branch] eqy-patch-2 -> origin/eqy-patch-2 2025-12-04T08:27:07.0083593Z * [new branch] eqy-patch-3 -> origin/eqy-patch-3 2025-12-04T08:27:07.0083969Z * [new branch] eqy-patch-4 -> origin/eqy-patch-4 2025-12-04T08:27:07.0084980Z * [new branch] eqy-patch-5 -> origin/eqy-patch-5 2025-12-04T08:27:07.0085547Z * [new branch] eqy-patch-6 -> origin/eqy-patch-6 2025-12-04T08:27:07.0086894Z * [new branch] exclamaforte/amd-ma -> origin/exclamaforte/amd-ma 2025-12-04T08:27:07.0087404Z * [new branch] exclamaforte/combo-kernels-perf-run -> origin/exclamaforte/combo-kernels-perf-run 2025-12-04T08:27:07.0088148Z * [new branch] exclamaforte/do_bench_refactor -> origin/exclamaforte/do_bench_refactor 2025-12-04T08:27:07.0088917Z * [new branch] exclamaforte/enable-mem-dep-fusion -> origin/exclamaforte/enable-mem-dep-fusion 2025-12-04T08:27:07.0089546Z * [new branch] exclamaforte/fix-exhaustive-autotuning -> origin/exclamaforte/fix-exhaustive-autotuning 2025-12-04T08:27:07.0090831Z * [new branch] exclamaforte/fix-trace-parsing-fx-svg -> origin/exclamaforte/fix-trace-parsing-fx-svg 2025-12-04T08:27:07.0091417Z * [new branch] exclamaforte/force-pointwise-cat-perf-run -> origin/exclamaforte/force-pointwise-cat-perf-run 2025-12-04T08:27:07.0092290Z * [new branch] exclamaforte/fusion-data -> origin/exclamaforte/fusion-data 2025-12-04T08:27:07.0092974Z * [new branch] exclamaforte/gemm-benchmark-run -> origin/exclamaforte/gemm-benchmark-run 2025-12-04T08:27:07.0093779Z * [new branch] exclamaforte/gemm-export-model -> origin/exclamaforte/gemm-export-model 2025-12-04T08:27:07.0094351Z * [new branch] exclamaforte/gemm-model -> origin/exclamaforte/gemm-model 2025-12-04T08:27:07.0095060Z * [new branch] exclamaforte/gemm-model-all-data-collection -> origin/exclamaforte/gemm-model-all-data-collection 2025-12-04T08:27:07.0095861Z * [new branch] exclamaforte/gemm-to-amd -> origin/exclamaforte/gemm-to-amd 2025-12-04T08:27:07.0096588Z * [new branch] exclamaforte/just-gemm-model -> origin/exclamaforte/just-gemm-model 2025-12-04T08:27:07.0097340Z * [new branch] exclamaforte/just-gemm-model-no-refactor -> origin/exclamaforte/just-gemm-model-no-refactor 2025-12-04T08:27:07.0098095Z * [new branch] exclamaforte/profile-diff-algo -> origin/exclamaforte/profile-diff-algo 2025-12-04T08:27:07.0098846Z * [new branch] exclamaforte/profiler-visualization -> origin/exclamaforte/profiler-visualization 2025-12-04T08:27:07.0099538Z * [new branch] exclamaforte/test_cpp_wrapper_mode -> origin/exclamaforte/test_cpp_wrapper_mode 2025-12-04T08:27:07.0100333Z * [new branch] exclamaforte/update-autotune-configs -> origin/exclamaforte/update-autotune-configs 2025-12-04T08:27:07.0100934Z * [new branch] exclamaforte/update-autotune-configs-2 -> origin/exclamaforte/update-autotune-configs-2 2025-12-04T08:27:07.0101720Z * [new branch] exec -> origin/exec 2025-12-04T08:27:07.0103132Z * [new branch] experimental-mosaic -> origin/experimental-mosaic 2025-12-04T08:27:07.0103817Z * [new branch] export-D61047529 -> origin/export-D61047529 2025-12-04T08:27:07.0104376Z * [new branch] export-D71412006 -> origin/export-D71412006 2025-12-04T08:27:07.0104909Z * [new branch] export-D73042989 -> origin/export-D73042989 2025-12-04T08:27:07.0105872Z * [new branch] export-D78957093 -> origin/export-D78957093 2025-12-04T08:27:07.0106469Z * [new branch] export-D78996107 -> origin/export-D78996107 2025-12-04T08:27:07.0107332Z * [new branch] export-D80823877 -> origin/export-D80823877 2025-12-04T08:27:07.0108773Z * [new branch] export-D80958642 -> origin/export-D80958642 2025-12-04T08:27:07.0109595Z * [new branch] export-D81054193 -> origin/export-D81054193 2025-12-04T08:27:07.0110026Z * [new branch] export-D81204584 -> origin/export-D81204584 2025-12-04T08:27:07.0115948Z * [new branch] export-D81429090 -> origin/export-D81429090 2025-12-04T08:27:07.0116387Z * [new branch] export-D82250826 -> origin/export-D82250826 2025-12-04T08:27:07.0116734Z * [new branch] export-D82253817 -> origin/export-D82253817 2025-12-04T08:27:07.0117103Z * [new branch] export-D83541846 -> origin/export-D83541846 2025-12-04T08:27:07.0117449Z * [new branch] export-D83627170 -> origin/export-D83627170 2025-12-04T08:27:07.0117791Z * [new branch] export-D83766701 -> origin/export-D83766701 2025-12-04T08:27:07.0118300Z * [new branch] export-D83768878 -> origin/export-D83768878 2025-12-04T08:27:07.0118660Z * [new branch] export-D83769447 -> origin/export-D83769447 2025-12-04T08:27:07.0119005Z * [new branch] export-D84089824 -> origin/export-D84089824 2025-12-04T08:27:07.0119345Z * [new branch] export-D84213020 -> origin/export-D84213020 2025-12-04T08:27:07.0119685Z * [new branch] export-D84373821 -> origin/export-D84373821 2025-12-04T08:27:07.0120035Z * [new branch] export-D84612194 -> origin/export-D84612194 2025-12-04T08:27:07.0120547Z * [new branch] export-D84890985 -> origin/export-D84890985 2025-12-04T08:27:07.0121261Z * [new branch] export-D85122326 -> origin/export-D85122326 2025-12-04T08:27:07.0122016Z * [new branch] export-D86256198 -> origin/export-D86256198 2025-12-04T08:27:07.0122701Z * [new branch] export-D86460608 -> origin/export-D86460608 2025-12-04T08:27:07.0123339Z * [new branch] export-D86474796 -> origin/export-D86474796 2025-12-04T08:27:07.0125363Z * [new branch] export-D86712396 -> origin/export-D86712396 2025-12-04T08:27:07.0125780Z * [new branch] export-D87022129 -> origin/export-D87022129 2025-12-04T08:27:07.0130353Z * [new branch] export-D87838959 -> origin/export-D87838959 2025-12-04T08:27:07.0134725Z * [new branch] export-D88319437 -> origin/export-D88319437 2025-12-04T08:27:07.0139763Z * [new branch] exported-model-train-idempotent -> origin/exported-model-train-idempotent 2025-12-04T08:27:07.0140280Z * [new branch] ezyang-titan-october -> origin/ezyang-titan-october 2025-12-04T08:27:07.0145423Z * [new branch] ezyang-titan-october2 -> origin/ezyang-titan-october2 2025-12-04T08:27:07.0145847Z * [new branch] ezyang-war -> origin/ezyang-war 2025-12-04T08:27:07.0146230Z * [new branch] ezyang/wip-aot-descriptors -> origin/ezyang/wip-aot-descriptors 2025-12-04T08:27:07.0146606Z * [new branch] fa_u8_brgemm -> origin/fa_u8_brgemm 2025-12-04T08:27:07.0146976Z * [new branch] fadeputr/sequence_fbgemm -> origin/fadeputr/sequence_fbgemm 2025-12-04T08:27:07.0147354Z * [new branch] fastmath_baseline -> origin/fastmath_baseline 2025-12-04T08:27:07.0147692Z * [new branch] fbcode/warm -> origin/fbcode/warm 2025-12-04T08:27:07.0147997Z * [new branch] fca -> origin/fca 2025-12-04T08:27:07.0148288Z * [new branch] fca2_ca5984c -> origin/fca2_ca5984c 2025-12-04T08:27:07.0148832Z * [new branch] fca5 -> origin/fca5 2025-12-04T08:27:07.0149165Z * [new branch] feature/justknobs-cpp -> origin/feature/justknobs-cpp 2025-12-04T08:27:07.0149558Z * [new branch] feature/numa-forkserver -> origin/feature/numa-forkserver 2025-12-04T08:27:07.0149931Z * [new branch] ffast_math_baseline -> origin/ffast_math_baseline 2025-12-04T08:27:07.0150274Z * [new branch] ffast_math_target -> origin/ffast_math_target 2025-12-04T08:27:07.0150614Z * [new branch] findhao/base_commit -> origin/findhao/base_commit 2025-12-04T08:27:07.0150974Z * [new branch] findhao/base_commit1 -> origin/findhao/base_commit1 2025-12-04T08:27:07.0151321Z * [new branch] findhao/multistream2 -> origin/findhao/multistream2 2025-12-04T08:27:07.0151665Z * [new branch] findhao/multistream5 -> origin/findhao/multistream5 2025-12-04T08:27:07.0152001Z * [new branch] findhao/multistream6 -> origin/findhao/multistream6 2025-12-04T08:27:07.0152444Z * [new branch] findhao/operatorbench3 -> origin/findhao/operatorbench3 2025-12-04T08:27:07.0152845Z * [new branch] findhao/operatorbench5 -> origin/findhao/operatorbench5 2025-12-04T08:27:07.0153205Z * [new branch] findhao/tritonparse -> origin/findhao/tritonparse 2025-12-04T08:27:07.0153582Z * [new branch] fix-ck-gemm-template-format -> origin/fix-ck-gemm-template-format 2025-12-04T08:27:07.0153965Z * [new branch] fix-config-ignore -> origin/fix-config-ignore 2025-12-04T08:27:07.0154312Z * [new branch] fix-dict-guard -> origin/fix-dict-guard 2025-12-04T08:27:07.0154715Z * [new branch] fix_addmm_issue -> origin/fix_addmm_issue 2025-12-04T08:27:07.0155110Z * [new branch] fix_amd_missing_cluster_dims -> origin/fix_amd_missing_cluster_dims 2025-12-04T08:27:07.0155500Z * [new branch] fix_bench_bwd_pass -> origin/fix_bench_bwd_pass 2025-12-04T08:27:07.0155869Z * [new branch] fix_mem_profiler_config -> origin/fix_mem_profiler_config 2025-12-04T08:27:07.0156234Z * [new branch] fix_nvrtc_discovery -> origin/fix_nvrtc_discovery 2025-12-04T08:27:07.0156575Z * [new branch] fix_op_runner -> origin/fix_op_runner 2025-12-04T08:27:07.0156899Z * [new branch] fix_ubn_159469 -> origin/fix_ubn_159469 2025-12-04T08:27:07.0157243Z * [new branch] fixes-triage -> origin/fixes-triage 2025-12-04T08:27:07.0157575Z * [new branch] fixflashinfer -> origin/fixflashinfer 2025-12-04T08:27:07.0158564Z * [new branch] flash_decoding_cpu -> origin/flash_decoding_cpu 2025-12-04T08:27:07.0158999Z * [new branch] flex-flash -> origin/flex-flash 2025-12-04T08:27:07.0159399Z * [new branch] flex_attention_functorch_grad -> origin/flex_attention_functorch_grad 2025-12-04T08:27:07.0159786Z * [new branch] flex_flash -> origin/flex_flash 2025-12-04T08:27:07.0160363Z * [new branch] fmassa/fix_memeff_sharding_rule -> origin/fmassa/fix_memeff_sharding_rule 2025-12-04T08:27:07.0160973Z * [new branch] fmassa/tests_comm_compute_scheduler -> origin/fmassa/tests_comm_compute_scheduler 2025-12-04T08:27:07.0161403Z * [new branch] forkserver_fix -> origin/forkserver_fix 2025-12-04T08:27:07.0162107Z * [new branch] fsdp2_trace_rules -> origin/fsdp2_trace_rules 2025-12-04T08:27:07.0163079Z * [new branch] fx_cpp -> origin/fx_cpp 2025-12-04T08:27:07.0166267Z * [new branch] fy/fix-win -> origin/fy/fix-win 2025-12-04T08:27:07.0166844Z * [new branch] galv-patch-1 -> origin/galv-patch-1 2025-12-04T08:27:07.0167633Z * [new branch] galv/cudagraphs-conditional-nodes-4 -> origin/galv/cudagraphs-conditional-nodes-4 2025-12-04T08:27:07.0168172Z * [new branch] georgehong/cmakelists-patch -> origin/georgehong/cmakelists-patch 2025-12-04T08:27:07.0168588Z * [new branch] gh/AlnisM/1/base -> origin/gh/AlnisM/1/base 2025-12-04T08:27:07.0169209Z * [new branch] gh/AlnisM/1/head -> origin/gh/AlnisM/1/head 2025-12-04T08:27:07.0173332Z * [new branch] gh/EikanWang/67/base -> origin/gh/EikanWang/67/base 2025-12-04T08:27:07.0173808Z * [new branch] gh/EikanWang/67/head -> origin/gh/EikanWang/67/head 2025-12-04T08:27:07.0174220Z * [new branch] gh/Gasoonjia/1/base -> origin/gh/Gasoonjia/1/base 2025-12-04T08:27:07.0174581Z * [new branch] gh/Gasoonjia/1/head -> origin/gh/Gasoonjia/1/head 2025-12-04T08:27:07.0174942Z * [new branch] gh/H-Huang/131/base -> origin/gh/H-Huang/131/base 2025-12-04T08:27:07.0178726Z * [new branch] gh/H-Huang/131/head -> origin/gh/H-Huang/131/head 2025-12-04T08:27:07.0179991Z * [new branch] gh/H-Huang/131/orig -> origin/gh/H-Huang/131/orig 2025-12-04T08:27:07.0180358Z * [new branch] gh/H-Huang/132/base -> origin/gh/H-Huang/132/base 2025-12-04T08:27:07.0180694Z * [new branch] gh/H-Huang/132/head -> origin/gh/H-Huang/132/head 2025-12-04T08:27:07.0181095Z * [new branch] gh/H-Huang/132/orig -> origin/gh/H-Huang/132/orig 2025-12-04T08:27:07.0181421Z * [new branch] gh/H-Huang/180/base -> origin/gh/H-Huang/180/base 2025-12-04T08:27:07.0181912Z * [new branch] gh/H-Huang/180/head -> origin/gh/H-Huang/180/head 2025-12-04T08:27:07.0182280Z * [new branch] gh/H-Huang/180/orig -> origin/gh/H-Huang/180/orig 2025-12-04T08:27:07.0183068Z * [new branch] gh/H-Huang/182/base -> origin/gh/H-Huang/182/base 2025-12-04T08:27:07.0183466Z * [new branch] gh/H-Huang/182/head -> origin/gh/H-Huang/182/head 2025-12-04T08:27:07.0183798Z * [new branch] gh/H-Huang/182/orig -> origin/gh/H-Huang/182/orig 2025-12-04T08:27:07.0184129Z * [new branch] gh/H-Huang/226/base -> origin/gh/H-Huang/226/base 2025-12-04T08:27:07.0189265Z * [new branch] gh/H-Huang/226/head -> origin/gh/H-Huang/226/head 2025-12-04T08:27:07.0189674Z * [new branch] gh/H-Huang/226/orig -> origin/gh/H-Huang/226/orig 2025-12-04T08:27:07.0190003Z * [new branch] gh/H-Huang/228/base -> origin/gh/H-Huang/228/base 2025-12-04T08:27:07.0190350Z * [new branch] gh/H-Huang/228/head -> origin/gh/H-Huang/228/head 2025-12-04T08:27:07.0190680Z * [new branch] gh/H-Huang/228/orig -> origin/gh/H-Huang/228/orig 2025-12-04T08:27:07.0191048Z * [new branch] gh/IvanKobzarev/150/base -> origin/gh/IvanKobzarev/150/base 2025-12-04T08:27:07.0191423Z * [new branch] gh/IvanKobzarev/150/head -> origin/gh/IvanKobzarev/150/head 2025-12-04T08:27:07.0191997Z * [new branch] gh/IvanKobzarev/150/orig -> origin/gh/IvanKobzarev/150/orig 2025-12-04T08:27:07.0192843Z * [new branch] gh/IvanKobzarev/157/base -> origin/gh/IvanKobzarev/157/base 2025-12-04T08:27:07.0193284Z * [new branch] gh/IvanKobzarev/157/head -> origin/gh/IvanKobzarev/157/head 2025-12-04T08:27:07.0193661Z * [new branch] gh/IvanKobzarev/157/orig -> origin/gh/IvanKobzarev/157/orig 2025-12-04T08:27:07.0194078Z * [new branch] gh/IvanKobzarev/159/base -> origin/gh/IvanKobzarev/159/base 2025-12-04T08:27:07.0194477Z * [new branch] gh/IvanKobzarev/159/head -> origin/gh/IvanKobzarev/159/head 2025-12-04T08:27:07.0195031Z * [new branch] gh/IvanKobzarev/159/orig -> origin/gh/IvanKobzarev/159/orig 2025-12-04T08:27:07.0195970Z * [new branch] gh/IvanKobzarev/162/base -> origin/gh/IvanKobzarev/162/base 2025-12-04T08:27:07.0196696Z * [new branch] gh/IvanKobzarev/162/head -> origin/gh/IvanKobzarev/162/head 2025-12-04T08:27:07.0197370Z * [new branch] gh/IvanKobzarev/162/orig -> origin/gh/IvanKobzarev/162/orig 2025-12-04T08:27:07.0198901Z * [new branch] gh/IvanKobzarev/163/base -> origin/gh/IvanKobzarev/163/base 2025-12-04T08:27:07.0199335Z * [new branch] gh/IvanKobzarev/163/head -> origin/gh/IvanKobzarev/163/head 2025-12-04T08:27:07.0199739Z * [new branch] gh/IvanKobzarev/163/orig -> origin/gh/IvanKobzarev/163/orig 2025-12-04T08:27:07.0201205Z * [new branch] gh/IvanKobzarev/166/base -> origin/gh/IvanKobzarev/166/base 2025-12-04T08:27:07.0201621Z * [new branch] gh/IvanKobzarev/166/head -> origin/gh/IvanKobzarev/166/head 2025-12-04T08:27:07.0203052Z * [new branch] gh/IvanKobzarev/166/orig -> origin/gh/IvanKobzarev/166/orig 2025-12-04T08:27:07.0204667Z * [new branch] gh/IvanKobzarev/167/base -> origin/gh/IvanKobzarev/167/base 2025-12-04T08:27:07.0205058Z * [new branch] gh/IvanKobzarev/167/head -> origin/gh/IvanKobzarev/167/head 2025-12-04T08:27:07.0205444Z * [new branch] gh/IvanKobzarev/167/orig -> origin/gh/IvanKobzarev/167/orig 2025-12-04T08:27:07.0206320Z * [new branch] gh/IvanKobzarev/168/base -> origin/gh/IvanKobzarev/168/base 2025-12-04T08:27:07.0206789Z * [new branch] gh/IvanKobzarev/168/head -> origin/gh/IvanKobzarev/168/head 2025-12-04T08:27:07.0208078Z * [new branch] gh/IvanKobzarev/168/orig -> origin/gh/IvanKobzarev/168/orig 2025-12-04T08:27:07.0208945Z * [new branch] gh/IvanKobzarev/169/base -> origin/gh/IvanKobzarev/169/base 2025-12-04T08:27:07.0209470Z * [new branch] gh/IvanKobzarev/169/head -> origin/gh/IvanKobzarev/169/head 2025-12-04T08:27:07.0209868Z * [new branch] gh/IvanKobzarev/169/orig -> origin/gh/IvanKobzarev/169/orig 2025-12-04T08:27:07.0210516Z * [new branch] gh/IvanKobzarev/170/base -> origin/gh/IvanKobzarev/170/base 2025-12-04T08:27:07.0216279Z * [new branch] gh/IvanKobzarev/170/head -> origin/gh/IvanKobzarev/170/head 2025-12-04T08:27:07.0216884Z * [new branch] gh/IvanKobzarev/170/orig -> origin/gh/IvanKobzarev/170/orig 2025-12-04T08:27:07.0219395Z * [new branch] gh/IvanKobzarev/171/base -> origin/gh/IvanKobzarev/171/base 2025-12-04T08:27:07.0219800Z * [new branch] gh/IvanKobzarev/171/head -> origin/gh/IvanKobzarev/171/head 2025-12-04T08:27:07.0220219Z * [new branch] gh/IvanKobzarev/171/orig -> origin/gh/IvanKobzarev/171/orig 2025-12-04T08:27:07.0220664Z * [new branch] gh/IvanKobzarev/172/base -> origin/gh/IvanKobzarev/172/base 2025-12-04T08:27:07.0221308Z * [new branch] gh/IvanKobzarev/172/head -> origin/gh/IvanKobzarev/172/head 2025-12-04T08:27:07.0221701Z * [new branch] gh/IvanKobzarev/172/orig -> origin/gh/IvanKobzarev/172/orig 2025-12-04T08:27:07.0222067Z * [new branch] gh/IvanKobzarev/173/base -> origin/gh/IvanKobzarev/173/base 2025-12-04T08:27:07.0222419Z * [new branch] gh/IvanKobzarev/173/head -> origin/gh/IvanKobzarev/173/head 2025-12-04T08:27:07.0222781Z * [new branch] gh/IvanKobzarev/173/orig -> origin/gh/IvanKobzarev/173/orig 2025-12-04T08:27:07.0223149Z * [new branch] gh/IvanKobzarev/174/base -> origin/gh/IvanKobzarev/174/base 2025-12-04T08:27:07.0223640Z * [new branch] gh/IvanKobzarev/174/head -> origin/gh/IvanKobzarev/174/head 2025-12-04T08:27:07.0224029Z * [new branch] gh/IvanKobzarev/174/orig -> origin/gh/IvanKobzarev/174/orig 2025-12-04T08:27:07.0224414Z * [new branch] gh/IvanKobzarev/175/base -> origin/gh/IvanKobzarev/175/base 2025-12-04T08:27:07.0225002Z * [new branch] gh/IvanKobzarev/175/head -> origin/gh/IvanKobzarev/175/head 2025-12-04T08:27:07.0225745Z * [new branch] gh/IvanKobzarev/175/orig -> origin/gh/IvanKobzarev/175/orig 2025-12-04T08:27:07.0230935Z * [new branch] gh/IvanKobzarev/176/base -> origin/gh/IvanKobzarev/176/base 2025-12-04T08:27:07.0233785Z * [new branch] gh/IvanKobzarev/176/head -> origin/gh/IvanKobzarev/176/head 2025-12-04T08:27:07.0234244Z * [new branch] gh/IvanKobzarev/176/orig -> origin/gh/IvanKobzarev/176/orig 2025-12-04T08:27:07.0234638Z * [new branch] gh/IvanKobzarev/177/base -> origin/gh/IvanKobzarev/177/base 2025-12-04T08:27:07.0235058Z * [new branch] gh/IvanKobzarev/177/head -> origin/gh/IvanKobzarev/177/head 2025-12-04T08:27:07.0235449Z * [new branch] gh/IvanKobzarev/177/orig -> origin/gh/IvanKobzarev/177/orig 2025-12-04T08:27:07.0235848Z * [new branch] gh/IvanKobzarev/178/base -> origin/gh/IvanKobzarev/178/base 2025-12-04T08:27:07.0236233Z * [new branch] gh/IvanKobzarev/178/head -> origin/gh/IvanKobzarev/178/head 2025-12-04T08:27:07.0236617Z * [new branch] gh/IvanKobzarev/178/orig -> origin/gh/IvanKobzarev/178/orig 2025-12-04T08:27:07.0236994Z * [new branch] gh/IvanKobzarev/179/base -> origin/gh/IvanKobzarev/179/base 2025-12-04T08:27:07.0237373Z * [new branch] gh/IvanKobzarev/179/head -> origin/gh/IvanKobzarev/179/head 2025-12-04T08:27:07.0237752Z * [new branch] gh/IvanKobzarev/179/orig -> origin/gh/IvanKobzarev/179/orig 2025-12-04T08:27:07.0238514Z * [new branch] gh/IvanKobzarev/180/base -> origin/gh/IvanKobzarev/180/base 2025-12-04T08:27:07.0238917Z * [new branch] gh/IvanKobzarev/180/head -> origin/gh/IvanKobzarev/180/head 2025-12-04T08:27:07.0239306Z * [new branch] gh/IvanKobzarev/180/orig -> origin/gh/IvanKobzarev/180/orig 2025-12-04T08:27:07.0239708Z * [new branch] gh/IvanKobzarev/181/base -> origin/gh/IvanKobzarev/181/base 2025-12-04T08:27:07.0245002Z * [new branch] gh/IvanKobzarev/181/head -> origin/gh/IvanKobzarev/181/head 2025-12-04T08:27:07.0246768Z * [new branch] gh/IvanKobzarev/181/orig -> origin/gh/IvanKobzarev/181/orig 2025-12-04T08:27:07.0247182Z * [new branch] gh/IvanKobzarev/182/base -> origin/gh/IvanKobzarev/182/base 2025-12-04T08:27:07.0252161Z * [new branch] gh/IvanKobzarev/182/head -> origin/gh/IvanKobzarev/182/head 2025-12-04T08:27:07.0256538Z * [new branch] gh/IvanKobzarev/182/orig -> origin/gh/IvanKobzarev/182/orig 2025-12-04T08:27:07.0261784Z * [new branch] gh/IvanKobzarev/183/base -> origin/gh/IvanKobzarev/183/base 2025-12-04T08:27:07.0262339Z * [new branch] gh/IvanKobzarev/183/head -> origin/gh/IvanKobzarev/183/head 2025-12-04T08:27:07.0262745Z * [new branch] gh/IvanKobzarev/183/orig -> origin/gh/IvanKobzarev/183/orig 2025-12-04T08:27:07.0263095Z * [new branch] gh/IvanKobzarev/184/base -> origin/gh/IvanKobzarev/184/base 2025-12-04T08:27:07.0263442Z * [new branch] gh/IvanKobzarev/184/head -> origin/gh/IvanKobzarev/184/head 2025-12-04T08:27:07.0263777Z * [new branch] gh/IvanKobzarev/184/orig -> origin/gh/IvanKobzarev/184/orig 2025-12-04T08:27:07.0264138Z * [new branch] gh/NikhilAPatel/1/base -> origin/gh/NikhilAPatel/1/base 2025-12-04T08:27:07.0264496Z * [new branch] gh/NikhilAPatel/1/head -> origin/gh/NikhilAPatel/1/head 2025-12-04T08:27:07.0264843Z * [new branch] gh/NikhilAPatel/2/base -> origin/gh/NikhilAPatel/2/base 2025-12-04T08:27:07.0265186Z * [new branch] gh/NikhilAPatel/2/head -> origin/gh/NikhilAPatel/2/head 2025-12-04T08:27:07.0265679Z * [new branch] gh/NikhilAPatel/4/base -> origin/gh/NikhilAPatel/4/base 2025-12-04T08:27:07.0266027Z * [new branch] gh/NikhilAPatel/4/head -> origin/gh/NikhilAPatel/4/head 2025-12-04T08:27:07.0266368Z * [new branch] gh/NikhilAPatel/5/base -> origin/gh/NikhilAPatel/5/base 2025-12-04T08:27:07.0266714Z * [new branch] gh/NikhilAPatel/5/head -> origin/gh/NikhilAPatel/5/head 2025-12-04T08:27:07.0267056Z * [new branch] gh/NikhilAPatel/5/orig -> origin/gh/NikhilAPatel/5/orig 2025-12-04T08:27:07.0267405Z * [new branch] gh/PaliC/17/base -> origin/gh/PaliC/17/base 2025-12-04T08:27:07.0267721Z * [new branch] gh/PaliC/17/head -> origin/gh/PaliC/17/head 2025-12-04T08:27:07.0268040Z * [new branch] gh/PaliC/17/orig -> origin/gh/PaliC/17/orig 2025-12-04T08:27:07.0268352Z * [new branch] gh/PaliC/18/base -> origin/gh/PaliC/18/base 2025-12-04T08:27:07.0268662Z * [new branch] gh/PaliC/18/head -> origin/gh/PaliC/18/head 2025-12-04T08:27:07.0268960Z * [new branch] gh/PaliC/18/orig -> origin/gh/PaliC/18/orig 2025-12-04T08:27:07.0269266Z * [new branch] gh/PaliC/20/base -> origin/gh/PaliC/20/base 2025-12-04T08:27:07.0269576Z * [new branch] gh/PaliC/20/head -> origin/gh/PaliC/20/head 2025-12-04T08:27:07.0269875Z * [new branch] gh/PaliC/20/orig -> origin/gh/PaliC/20/orig 2025-12-04T08:27:07.0270184Z * [new branch] gh/PaliC/21/base -> origin/gh/PaliC/21/base 2025-12-04T08:27:07.0270581Z * [new branch] gh/PaliC/21/head -> origin/gh/PaliC/21/head 2025-12-04T08:27:07.0270951Z * [new branch] gh/PaliC/21/orig -> origin/gh/PaliC/21/orig 2025-12-04T08:27:07.0271254Z * [new branch] gh/PaliC/23/base -> origin/gh/PaliC/23/base 2025-12-04T08:27:07.0271563Z * [new branch] gh/PaliC/23/head -> origin/gh/PaliC/23/head 2025-12-04T08:27:07.0271874Z * [new branch] gh/PaliC/23/orig -> origin/gh/PaliC/23/orig 2025-12-04T08:27:07.0272191Z * [new branch] gh/PaliC/24/base -> origin/gh/PaliC/24/base 2025-12-04T08:27:07.0272505Z * [new branch] gh/PaliC/24/head -> origin/gh/PaliC/24/head 2025-12-04T08:27:07.0272813Z * [new branch] gh/PaliC/24/orig -> origin/gh/PaliC/24/orig 2025-12-04T08:27:07.0273119Z * [new branch] gh/PaliC/25/head -> origin/gh/PaliC/25/head 2025-12-04T08:27:07.0273585Z * [new branch] gh/PaliC/25/next -> origin/gh/PaliC/25/next 2025-12-04T08:27:07.0274019Z * [new branch] gh/PaliC/25/orig -> origin/gh/PaliC/25/orig 2025-12-04T08:27:07.0274338Z * [new branch] gh/PaliC/26/head -> origin/gh/PaliC/26/head 2025-12-04T08:27:07.0274658Z * [new branch] gh/PaliC/26/next -> origin/gh/PaliC/26/next 2025-12-04T08:27:07.0274969Z * [new branch] gh/PaliC/26/orig -> origin/gh/PaliC/26/orig 2025-12-04T08:27:07.0275274Z * [new branch] gh/PaliC/27/next -> origin/gh/PaliC/27/next 2025-12-04T08:27:07.0275585Z * [new branch] gh/PaliC/28/head -> origin/gh/PaliC/28/head 2025-12-04T08:27:07.0275897Z * [new branch] gh/PaliC/28/next -> origin/gh/PaliC/28/next 2025-12-04T08:27:07.0276213Z * [new branch] gh/PaliC/28/orig -> origin/gh/PaliC/28/orig 2025-12-04T08:27:07.0276523Z * [new branch] gh/PaliC/29/head -> origin/gh/PaliC/29/head 2025-12-04T08:27:07.0277309Z * [new branch] gh/PaliC/29/next -> origin/gh/PaliC/29/next 2025-12-04T08:27:07.0277717Z * [new branch] gh/PaliC/29/orig -> origin/gh/PaliC/29/orig 2025-12-04T08:27:07.0278480Z * [new branch] gh/PaliC/30/head -> origin/gh/PaliC/30/head 2025-12-04T08:27:07.0283790Z * [new branch] gh/PaliC/30/next -> origin/gh/PaliC/30/next 2025-12-04T08:27:07.0284152Z * [new branch] gh/PaliC/30/orig -> origin/gh/PaliC/30/orig 2025-12-04T08:27:07.0284501Z * [new branch] gh/PaliC/31/head -> origin/gh/PaliC/31/head 2025-12-04T08:27:07.0284842Z * [new branch] gh/PaliC/31/next -> origin/gh/PaliC/31/next 2025-12-04T08:27:07.0285186Z * [new branch] gh/PaliC/31/orig -> origin/gh/PaliC/31/orig 2025-12-04T08:27:07.0285590Z * [new branch] gh/PaulZhang12/25/base -> origin/gh/PaulZhang12/25/base 2025-12-04T08:27:07.0285989Z * [new branch] gh/PaulZhang12/25/head -> origin/gh/PaulZhang12/25/head 2025-12-04T08:27:07.0286387Z * [new branch] gh/PaulZhang12/25/orig -> origin/gh/PaulZhang12/25/orig 2025-12-04T08:27:07.0287973Z * [new branch] gh/PaulZhang12/28/base -> origin/gh/PaulZhang12/28/base 2025-12-04T08:27:07.0288383Z * [new branch] gh/PaulZhang12/28/head -> origin/gh/PaulZhang12/28/head 2025-12-04T08:27:07.0288758Z * [new branch] gh/PaulZhang12/28/orig -> origin/gh/PaulZhang12/28/orig 2025-12-04T08:27:07.0289149Z * [new branch] gh/PaulZhang12/31/base -> origin/gh/PaulZhang12/31/base 2025-12-04T08:27:07.0289522Z * [new branch] gh/PaulZhang12/31/head -> origin/gh/PaulZhang12/31/head 2025-12-04T08:27:07.0289922Z * [new branch] gh/PaulZhang12/31/orig -> origin/gh/PaulZhang12/31/orig 2025-12-04T08:27:07.0292633Z * [new branch] gh/PaulZhang12/37/base -> origin/gh/PaulZhang12/37/base 2025-12-04T08:27:07.0293078Z * [new branch] gh/PaulZhang12/37/head -> origin/gh/PaulZhang12/37/head 2025-12-04T08:27:07.0293447Z * [new branch] gh/PaulZhang12/37/orig -> origin/gh/PaulZhang12/37/orig 2025-12-04T08:27:07.0293830Z * [new branch] gh/PaulZhang12/40/base -> origin/gh/PaulZhang12/40/base 2025-12-04T08:27:07.0294214Z * [new branch] gh/PaulZhang12/40/head -> origin/gh/PaulZhang12/40/head 2025-12-04T08:27:07.0294596Z * [new branch] gh/PaulZhang12/40/orig -> origin/gh/PaulZhang12/40/orig 2025-12-04T08:27:07.0295405Z * [new branch] gh/PaulZhang12/42/base -> origin/gh/PaulZhang12/42/base 2025-12-04T08:27:07.0295861Z * [new branch] gh/PaulZhang12/42/head -> origin/gh/PaulZhang12/42/head 2025-12-04T08:27:07.0296642Z * [new branch] gh/PaulZhang12/43/base -> origin/gh/PaulZhang12/43/base 2025-12-04T08:27:07.0297325Z * [new branch] gh/PaulZhang12/43/head -> origin/gh/PaulZhang12/43/head 2025-12-04T08:27:07.0298040Z * [new branch] gh/PaulZhang12/43/orig -> origin/gh/PaulZhang12/43/orig 2025-12-04T08:27:07.0298924Z * [new branch] gh/PaulZhang12/44/base -> origin/gh/PaulZhang12/44/base 2025-12-04T08:27:07.0299590Z * [new branch] gh/PaulZhang12/44/head -> origin/gh/PaulZhang12/44/head 2025-12-04T08:27:07.0300923Z * [new branch] gh/PaulZhang12/45/base -> origin/gh/PaulZhang12/45/base 2025-12-04T08:27:07.0301347Z * [new branch] gh/PaulZhang12/45/head -> origin/gh/PaulZhang12/45/head 2025-12-04T08:27:07.0301998Z * [new branch] gh/PaulZhang12/45/orig -> origin/gh/PaulZhang12/45/orig 2025-12-04T08:27:07.0303193Z * [new branch] gh/PaulZhang12/46/base -> origin/gh/PaulZhang12/46/base 2025-12-04T08:27:07.0303645Z * [new branch] gh/PaulZhang12/46/head -> origin/gh/PaulZhang12/46/head 2025-12-04T08:27:07.0304404Z * [new branch] gh/PaulZhang12/46/orig -> origin/gh/PaulZhang12/46/orig 2025-12-04T08:27:07.0305589Z * [new branch] gh/PaulZhang12/47/base -> origin/gh/PaulZhang12/47/base 2025-12-04T08:27:07.0307002Z * [new branch] gh/PaulZhang12/47/head -> origin/gh/PaulZhang12/47/head 2025-12-04T08:27:07.0307393Z * [new branch] gh/PaulZhang12/47/orig -> origin/gh/PaulZhang12/47/orig 2025-12-04T08:27:07.0307769Z * [new branch] gh/PaulZhang12/48/base -> origin/gh/PaulZhang12/48/base 2025-12-04T08:27:07.0308217Z * [new branch] gh/PaulZhang12/48/head -> origin/gh/PaulZhang12/48/head 2025-12-04T08:27:07.0309009Z * [new branch] gh/PaulZhang12/48/orig -> origin/gh/PaulZhang12/48/orig 2025-12-04T08:27:07.0312496Z * [new branch] gh/SamGinzburg/11/base -> origin/gh/SamGinzburg/11/base 2025-12-04T08:27:07.0312888Z * [new branch] gh/SamGinzburg/11/head -> origin/gh/SamGinzburg/11/head 2025-12-04T08:27:07.0313278Z * [new branch] gh/SherlockNoMad/1/base -> origin/gh/SherlockNoMad/1/base 2025-12-04T08:27:07.0313668Z * [new branch] gh/SherlockNoMad/1/head -> origin/gh/SherlockNoMad/1/head 2025-12-04T08:27:07.0314055Z * [new branch] gh/SherlockNoMad/10/base -> origin/gh/SherlockNoMad/10/base 2025-12-04T08:27:07.0314450Z * [new branch] gh/SherlockNoMad/10/head -> origin/gh/SherlockNoMad/10/head 2025-12-04T08:27:07.0314851Z * [new branch] gh/SherlockNoMad/10/orig -> origin/gh/SherlockNoMad/10/orig 2025-12-04T08:27:07.0315984Z * [new branch] gh/SherlockNoMad/11/base -> origin/gh/SherlockNoMad/11/base 2025-12-04T08:27:07.0316383Z * [new branch] gh/SherlockNoMad/11/head -> origin/gh/SherlockNoMad/11/head 2025-12-04T08:27:07.0317164Z * [new branch] gh/SherlockNoMad/11/orig -> origin/gh/SherlockNoMad/11/orig 2025-12-04T08:27:07.0318154Z * [new branch] gh/SherlockNoMad/12/base -> origin/gh/SherlockNoMad/12/base 2025-12-04T08:27:07.0318913Z * [new branch] gh/SherlockNoMad/12/head -> origin/gh/SherlockNoMad/12/head 2025-12-04T08:27:07.0319614Z * [new branch] gh/SherlockNoMad/12/orig -> origin/gh/SherlockNoMad/12/orig 2025-12-04T08:27:07.0321038Z * [new branch] gh/SherlockNoMad/15/base -> origin/gh/SherlockNoMad/15/base 2025-12-04T08:27:07.0321485Z * [new branch] gh/SherlockNoMad/15/head -> origin/gh/SherlockNoMad/15/head 2025-12-04T08:27:07.0322145Z * [new branch] gh/SherlockNoMad/15/orig -> origin/gh/SherlockNoMad/15/orig 2025-12-04T08:27:07.0323281Z * [new branch] gh/SherlockNoMad/17/base -> origin/gh/SherlockNoMad/17/base 2025-12-04T08:27:07.0323815Z * [new branch] gh/SherlockNoMad/17/head -> origin/gh/SherlockNoMad/17/head 2025-12-04T08:27:07.0324487Z * [new branch] gh/SherlockNoMad/17/orig -> origin/gh/SherlockNoMad/17/orig 2025-12-04T08:27:07.0325756Z * [new branch] gh/SherlockNoMad/18/base -> origin/gh/SherlockNoMad/18/base 2025-12-04T08:27:07.0326136Z * [new branch] gh/SherlockNoMad/18/head -> origin/gh/SherlockNoMad/18/head 2025-12-04T08:27:07.0327359Z * [new branch] gh/SherlockNoMad/18/orig -> origin/gh/SherlockNoMad/18/orig 2025-12-04T08:27:07.0327744Z * [new branch] gh/SherlockNoMad/19/base -> origin/gh/SherlockNoMad/19/base 2025-12-04T08:27:07.0328923Z * [new branch] gh/SherlockNoMad/19/head -> origin/gh/SherlockNoMad/19/head 2025-12-04T08:27:07.0329306Z * [new branch] gh/SherlockNoMad/19/orig -> origin/gh/SherlockNoMad/19/orig 2025-12-04T08:27:07.0330517Z * [new branch] gh/SherlockNoMad/2/base -> origin/gh/SherlockNoMad/2/base 2025-12-04T08:27:07.0331084Z * [new branch] gh/SherlockNoMad/2/head -> origin/gh/SherlockNoMad/2/head 2025-12-04T08:27:07.0332861Z * [new branch] gh/SherlockNoMad/20/base -> origin/gh/SherlockNoMad/20/base 2025-12-04T08:27:07.0333243Z * [new branch] gh/SherlockNoMad/20/head -> origin/gh/SherlockNoMad/20/head 2025-12-04T08:27:07.0333748Z * [new branch] gh/SherlockNoMad/20/orig -> origin/gh/SherlockNoMad/20/orig 2025-12-04T08:27:07.0334131Z * [new branch] gh/SherlockNoMad/21/base -> origin/gh/SherlockNoMad/21/base 2025-12-04T08:27:07.0335484Z * [new branch] gh/SherlockNoMad/21/head -> origin/gh/SherlockNoMad/21/head 2025-12-04T08:27:07.0335993Z * [new branch] gh/SherlockNoMad/21/orig -> origin/gh/SherlockNoMad/21/orig 2025-12-04T08:27:07.0336733Z * [new branch] gh/SherlockNoMad/3/base -> origin/gh/SherlockNoMad/3/base 2025-12-04T08:27:07.0337282Z * [new branch] gh/SherlockNoMad/3/head -> origin/gh/SherlockNoMad/3/head 2025-12-04T08:27:07.0338083Z * [new branch] gh/SherlockNoMad/4/base -> origin/gh/SherlockNoMad/4/base 2025-12-04T08:27:07.0338543Z * [new branch] gh/SherlockNoMad/4/head -> origin/gh/SherlockNoMad/4/head 2025-12-04T08:27:07.0339742Z * [new branch] gh/SherlockNoMad/5/base -> origin/gh/SherlockNoMad/5/base 2025-12-04T08:27:07.0340173Z * [new branch] gh/SherlockNoMad/5/head -> origin/gh/SherlockNoMad/5/head 2025-12-04T08:27:07.0341936Z * [new branch] gh/Sidharth123-cpu/24/base -> origin/gh/Sidharth123-cpu/24/base 2025-12-04T08:27:07.0342425Z * [new branch] gh/Sidharth123-cpu/25/base -> origin/gh/Sidharth123-cpu/25/base 2025-12-04T08:27:07.0345709Z * [new branch] gh/Sidharth123-cpu/26/base -> origin/gh/Sidharth123-cpu/26/base 2025-12-04T08:27:07.0346188Z * [new branch] gh/Sidharth123-cpu/27/base -> origin/gh/Sidharth123-cpu/27/base 2025-12-04T08:27:07.0346607Z * [new branch] gh/StrongerXi/1/base -> origin/gh/StrongerXi/1/base 2025-12-04T08:27:07.0347218Z * [new branch] gh/StrongerXi/1/head -> origin/gh/StrongerXi/1/head 2025-12-04T08:27:07.0347631Z * [new branch] gh/StrongerXi/71/base -> origin/gh/StrongerXi/71/base 2025-12-04T08:27:07.0348263Z * [new branch] gh/StrongerXi/71/head -> origin/gh/StrongerXi/71/head 2025-12-04T08:27:07.0348658Z * [new branch] gh/StrongerXi/72/base -> origin/gh/StrongerXi/72/base 2025-12-04T08:27:07.0349019Z * [new branch] gh/StrongerXi/72/head -> origin/gh/StrongerXi/72/head 2025-12-04T08:27:07.0350547Z * [new branch] gh/StrongerXi/73/base -> origin/gh/StrongerXi/73/base 2025-12-04T08:27:07.0350916Z * [new branch] gh/StrongerXi/73/head -> origin/gh/StrongerXi/73/head 2025-12-04T08:27:07.0356154Z * [new branch] gh/StrongerXi/73/orig -> origin/gh/StrongerXi/73/orig 2025-12-04T08:27:07.0356645Z * [new branch] gh/XilunWu/160/base -> origin/gh/XilunWu/160/base 2025-12-04T08:27:07.0357027Z * [new branch] gh/XilunWu/160/head -> origin/gh/XilunWu/160/head 2025-12-04T08:27:07.0357392Z * [new branch] gh/XilunWu/160/orig -> origin/gh/XilunWu/160/orig 2025-12-04T08:27:07.0357759Z * [new branch] gh/XilunWu/163/base -> origin/gh/XilunWu/163/base 2025-12-04T08:27:07.0358365Z * [new branch] gh/XilunWu/163/head -> origin/gh/XilunWu/163/head 2025-12-04T08:27:07.0358768Z * [new branch] gh/XilunWu/163/orig -> origin/gh/XilunWu/163/orig 2025-12-04T08:27:07.0359151Z * [new branch] gh/XilunWu/168/base -> origin/gh/XilunWu/168/base 2025-12-04T08:27:07.0359509Z * [new branch] gh/XilunWu/168/head -> origin/gh/XilunWu/168/head 2025-12-04T08:27:07.0359877Z * [new branch] gh/XilunWu/168/orig -> origin/gh/XilunWu/168/orig 2025-12-04T08:27:07.0366447Z * [new branch] gh/XilunWu/169/base -> origin/gh/XilunWu/169/base 2025-12-04T08:27:07.0371916Z * [new branch] gh/XilunWu/169/head -> origin/gh/XilunWu/169/head 2025-12-04T08:27:07.0372468Z * [new branch] gh/XilunWu/169/orig -> origin/gh/XilunWu/169/orig 2025-12-04T08:27:07.0373117Z * [new branch] gh/XilunWu/170/base -> origin/gh/XilunWu/170/base 2025-12-04T08:27:07.0373467Z * [new branch] gh/XilunWu/170/head -> origin/gh/XilunWu/170/head 2025-12-04T08:27:07.0373816Z * [new branch] gh/XilunWu/170/orig -> origin/gh/XilunWu/170/orig 2025-12-04T08:27:07.0374142Z * [new branch] gh/XilunWu/171/base -> origin/gh/XilunWu/171/base 2025-12-04T08:27:07.0374477Z * [new branch] gh/XilunWu/171/head -> origin/gh/XilunWu/171/head 2025-12-04T08:27:07.0374808Z * [new branch] gh/XilunWu/171/orig -> origin/gh/XilunWu/171/orig 2025-12-04T08:27:07.0375146Z * [new branch] gh/XilunWu/173/base -> origin/gh/XilunWu/173/base 2025-12-04T08:27:07.0375468Z * [new branch] gh/XilunWu/173/head -> origin/gh/XilunWu/173/head 2025-12-04T08:27:07.0375799Z * [new branch] gh/XilunWu/173/orig -> origin/gh/XilunWu/173/orig 2025-12-04T08:27:07.0376132Z * [new branch] gh/XilunWu/175/base -> origin/gh/XilunWu/175/base 2025-12-04T08:27:07.0376467Z * [new branch] gh/XilunWu/175/head -> origin/gh/XilunWu/175/head 2025-12-04T08:27:07.0376800Z * [new branch] gh/XilunWu/175/orig -> origin/gh/XilunWu/175/orig 2025-12-04T08:27:07.0377162Z * [new branch] gh/XilunWu/176/base -> origin/gh/XilunWu/176/base 2025-12-04T08:27:07.0377512Z * [new branch] gh/XilunWu/176/head -> origin/gh/XilunWu/176/head 2025-12-04T08:27:07.0377858Z * [new branch] gh/XilunWu/176/orig -> origin/gh/XilunWu/176/orig 2025-12-04T08:27:07.0378275Z * [new branch] gh/XuehaiPan/14/base -> origin/gh/XuehaiPan/14/base 2025-12-04T08:27:07.0378650Z * [new branch] gh/XuehaiPan/14/head -> origin/gh/XuehaiPan/14/head 2025-12-04T08:27:07.0379018Z * [new branch] gh/XuehaiPan/14/orig -> origin/gh/XuehaiPan/14/orig 2025-12-04T08:27:07.0379380Z * [new branch] gh/XuehaiPan/179/base -> origin/gh/XuehaiPan/179/base 2025-12-04T08:27:07.0379751Z * [new branch] gh/XuehaiPan/179/head -> origin/gh/XuehaiPan/179/head 2025-12-04T08:27:07.0380127Z * [new branch] gh/XuehaiPan/179/orig -> origin/gh/XuehaiPan/179/orig 2025-12-04T08:27:07.0380491Z * [new branch] gh/XuehaiPan/249/base -> origin/gh/XuehaiPan/249/base 2025-12-04T08:27:07.0380851Z * [new branch] gh/XuehaiPan/249/head -> origin/gh/XuehaiPan/249/head 2025-12-04T08:27:07.0381403Z * [new branch] gh/XuehaiPan/249/orig -> origin/gh/XuehaiPan/249/orig 2025-12-04T08:27:07.0381781Z * [new branch] gh/XuehaiPan/253/base -> origin/gh/XuehaiPan/253/base 2025-12-04T08:27:07.0382148Z * [new branch] gh/XuehaiPan/253/head -> origin/gh/XuehaiPan/253/head 2025-12-04T08:27:07.0382511Z * [new branch] gh/XuehaiPan/253/orig -> origin/gh/XuehaiPan/253/orig 2025-12-04T08:27:07.0382894Z * [new branch] gh/XuehaiPan/254/base -> origin/gh/XuehaiPan/254/base 2025-12-04T08:27:07.0389197Z * [new branch] gh/XuehaiPan/254/head -> origin/gh/XuehaiPan/254/head 2025-12-04T08:27:07.0391266Z * [new branch] gh/XuehaiPan/254/orig -> origin/gh/XuehaiPan/254/orig 2025-12-04T08:27:07.0391690Z * [new branch] gh/XuehaiPan/255/base -> origin/gh/XuehaiPan/255/base 2025-12-04T08:27:07.0392053Z * [new branch] gh/XuehaiPan/255/head -> origin/gh/XuehaiPan/255/head 2025-12-04T08:27:07.0392432Z * [new branch] gh/XuehaiPan/255/orig -> origin/gh/XuehaiPan/255/orig 2025-12-04T08:27:07.0392948Z * [new branch] gh/XuehaiPan/271/base -> origin/gh/XuehaiPan/271/base 2025-12-04T08:27:07.0395302Z * [new branch] gh/XuehaiPan/271/head -> origin/gh/XuehaiPan/271/head 2025-12-04T08:27:07.0395901Z * [new branch] gh/XuehaiPan/271/orig -> origin/gh/XuehaiPan/271/orig 2025-12-04T08:27:07.0396288Z * [new branch] gh/XuehaiPan/343/base -> origin/gh/XuehaiPan/343/base 2025-12-04T08:27:07.0396664Z * [new branch] gh/XuehaiPan/343/head -> origin/gh/XuehaiPan/343/head 2025-12-04T08:27:07.0397027Z * [new branch] gh/XuehaiPan/343/orig -> origin/gh/XuehaiPan/343/orig 2025-12-04T08:27:07.0397380Z * [new branch] gh/XuehaiPan/347/base -> origin/gh/XuehaiPan/347/base 2025-12-04T08:27:07.0397749Z * [new branch] gh/XuehaiPan/347/head -> origin/gh/XuehaiPan/347/head 2025-12-04T08:27:07.0398404Z * [new branch] gh/XuehaiPan/347/orig -> origin/gh/XuehaiPan/347/orig 2025-12-04T08:27:07.0398795Z * [new branch] gh/XuehaiPan/348/base -> origin/gh/XuehaiPan/348/base 2025-12-04T08:27:07.0399162Z * [new branch] gh/XuehaiPan/348/head -> origin/gh/XuehaiPan/348/head 2025-12-04T08:27:07.0399542Z * [new branch] gh/XuehaiPan/348/orig -> origin/gh/XuehaiPan/348/orig 2025-12-04T08:27:07.0399904Z * [new branch] gh/XuehaiPan/350/base -> origin/gh/XuehaiPan/350/base 2025-12-04T08:27:07.0400264Z * [new branch] gh/XuehaiPan/350/head -> origin/gh/XuehaiPan/350/head 2025-12-04T08:27:07.0400613Z * [new branch] gh/XuehaiPan/350/orig -> origin/gh/XuehaiPan/350/orig 2025-12-04T08:27:07.0400975Z * [new branch] gh/XuehaiPan/365/base -> origin/gh/XuehaiPan/365/base 2025-12-04T08:27:07.0401333Z * [new branch] gh/XuehaiPan/365/head -> origin/gh/XuehaiPan/365/head 2025-12-04T08:27:07.0401757Z * [new branch] gh/XuehaiPan/365/orig -> origin/gh/XuehaiPan/365/orig 2025-12-04T08:27:07.0404305Z * [new branch] gh/XuehaiPan/366/base -> origin/gh/XuehaiPan/366/base 2025-12-04T08:27:07.0404941Z * [new branch] gh/XuehaiPan/366/head -> origin/gh/XuehaiPan/366/head 2025-12-04T08:27:07.0405473Z * [new branch] gh/XuehaiPan/370/base -> origin/gh/XuehaiPan/370/base 2025-12-04T08:27:07.0406451Z * [new branch] gh/XuehaiPan/370/head -> origin/gh/XuehaiPan/370/head 2025-12-04T08:27:07.0406907Z * [new branch] gh/XuehaiPan/370/orig -> origin/gh/XuehaiPan/370/orig 2025-12-04T08:27:07.0407297Z * [new branch] gh/XuehaiPan/390/base -> origin/gh/XuehaiPan/390/base 2025-12-04T08:27:07.0407673Z * [new branch] gh/XuehaiPan/390/head -> origin/gh/XuehaiPan/390/head 2025-12-04T08:27:07.0408309Z * [new branch] gh/XuehaiPan/390/orig -> origin/gh/XuehaiPan/390/orig 2025-12-04T08:27:07.0408937Z * [new branch] gh/XuehaiPan/391/base -> origin/gh/XuehaiPan/391/base 2025-12-04T08:27:07.0409566Z * [new branch] gh/XuehaiPan/391/head -> origin/gh/XuehaiPan/391/head 2025-12-04T08:27:07.0410259Z * [new branch] gh/XuehaiPan/391/orig -> origin/gh/XuehaiPan/391/orig 2025-12-04T08:27:07.0412917Z * [new branch] gh/XuehaiPan/392/base -> origin/gh/XuehaiPan/392/base 2025-12-04T08:27:07.0413356Z * [new branch] gh/XuehaiPan/392/head -> origin/gh/XuehaiPan/392/head 2025-12-04T08:27:07.0413727Z * [new branch] gh/XuehaiPan/392/orig -> origin/gh/XuehaiPan/392/orig 2025-12-04T08:27:07.0420163Z * [new branch] gh/XuehaiPan/394/base -> origin/gh/XuehaiPan/394/base 2025-12-04T08:27:07.0420609Z * [new branch] gh/XuehaiPan/394/head -> origin/gh/XuehaiPan/394/head 2025-12-04T08:27:07.0421251Z * [new branch] gh/XuehaiPan/394/orig -> origin/gh/XuehaiPan/394/orig 2025-12-04T08:27:07.0421620Z * [new branch] gh/XuehaiPan/397/base -> origin/gh/XuehaiPan/397/base 2025-12-04T08:27:07.0422226Z * [new branch] gh/XuehaiPan/397/head -> origin/gh/XuehaiPan/397/head 2025-12-04T08:27:07.0422585Z * [new branch] gh/XuehaiPan/397/orig -> origin/gh/XuehaiPan/397/orig 2025-12-04T08:27:07.0422963Z * [new branch] gh/XuehaiPan/398/base -> origin/gh/XuehaiPan/398/base 2025-12-04T08:27:07.0423311Z * [new branch] gh/XuehaiPan/398/head -> origin/gh/XuehaiPan/398/head 2025-12-04T08:27:07.0423648Z * [new branch] gh/XuehaiPan/398/orig -> origin/gh/XuehaiPan/398/orig 2025-12-04T08:27:07.0423995Z * [new branch] gh/XuehaiPan/399/base -> origin/gh/XuehaiPan/399/base 2025-12-04T08:27:07.0424360Z * [new branch] gh/XuehaiPan/399/head -> origin/gh/XuehaiPan/399/head 2025-12-04T08:27:07.0425742Z * [new branch] gh/XuehaiPan/399/orig -> origin/gh/XuehaiPan/399/orig 2025-12-04T08:27:07.0426109Z * [new branch] gh/XuehaiPan/400/base -> origin/gh/XuehaiPan/400/base 2025-12-04T08:27:07.0426452Z * [new branch] gh/XuehaiPan/400/head -> origin/gh/XuehaiPan/400/head 2025-12-04T08:27:07.0426794Z * [new branch] gh/XuehaiPan/400/orig -> origin/gh/XuehaiPan/400/orig 2025-12-04T08:27:07.0427155Z * [new branch] gh/ZhiweiYan-96/39/base -> origin/gh/ZhiweiYan-96/39/base 2025-12-04T08:27:07.0427511Z * [new branch] gh/ZhiweiYan-96/39/head -> origin/gh/ZhiweiYan-96/39/head 2025-12-04T08:27:07.0427860Z * [new branch] gh/ZhiweiYan-96/39/orig -> origin/gh/ZhiweiYan-96/39/orig 2025-12-04T08:27:07.0429914Z * [new branch] gh/ZhiweiYan-96/44/base -> origin/gh/ZhiweiYan-96/44/base 2025-12-04T08:27:07.0430345Z * [new branch] gh/ZhiweiYan-96/44/head -> origin/gh/ZhiweiYan-96/44/head 2025-12-04T08:27:07.0430702Z * [new branch] gh/ZhiweiYan-96/45/base -> origin/gh/ZhiweiYan-96/45/base 2025-12-04T08:27:07.0431049Z * [new branch] gh/ZhiweiYan-96/45/head -> origin/gh/ZhiweiYan-96/45/head 2025-12-04T08:27:07.0431400Z * [new branch] gh/ZhiweiYan-96/49/base -> origin/gh/ZhiweiYan-96/49/base 2025-12-04T08:27:07.0431761Z * [new branch] gh/ZhiweiYan-96/49/head -> origin/gh/ZhiweiYan-96/49/head 2025-12-04T08:27:07.0432100Z * [new branch] gh/ZhiweiYan-96/62/base -> origin/gh/ZhiweiYan-96/62/base 2025-12-04T08:27:07.0432455Z * [new branch] gh/ZhiweiYan-96/62/head -> origin/gh/ZhiweiYan-96/62/head 2025-12-04T08:27:07.0432999Z * [new branch] gh/ZhiweiYan-96/66/base -> origin/gh/ZhiweiYan-96/66/base 2025-12-04T08:27:07.0433738Z * [new branch] gh/ZhiweiYan-96/66/head -> origin/gh/ZhiweiYan-96/66/head 2025-12-04T08:27:07.0434737Z * [new branch] gh/ZhiweiYan-96/67/base -> origin/gh/ZhiweiYan-96/67/base 2025-12-04T08:27:07.0435608Z * [new branch] gh/ZhiweiYan-96/67/head -> origin/gh/ZhiweiYan-96/67/head 2025-12-04T08:27:07.0436305Z * [new branch] gh/ZhiweiYan-96/68/base -> origin/gh/ZhiweiYan-96/68/base 2025-12-04T08:27:07.0436930Z * [new branch] gh/ZhiweiYan-96/68/head -> origin/gh/ZhiweiYan-96/68/head 2025-12-04T08:27:07.0437365Z * [new branch] gh/ZhiweiYan-96/68/orig -> origin/gh/ZhiweiYan-96/68/orig 2025-12-04T08:27:07.0439046Z * [new branch] gh/aakhundov/1/base -> origin/gh/aakhundov/1/base 2025-12-04T08:27:07.0439682Z * [new branch] gh/aakhundov/1/head -> origin/gh/aakhundov/1/head 2025-12-04T08:27:07.0440294Z * [new branch] gh/aakhundov/2/base -> origin/gh/aakhundov/2/base 2025-12-04T08:27:07.0441186Z * [new branch] gh/aakhundov/2/head -> origin/gh/aakhundov/2/head 2025-12-04T08:27:07.0444757Z * [new branch] gh/aditew01/openblas -> origin/gh/aditew01/openblas 2025-12-04T08:27:07.0445150Z * [new branch] gh/aditew01/sbgemm -> origin/gh/aditew01/sbgemm 2025-12-04T08:27:07.0445580Z * [new branch] gh/aditew01/vecbf16 -> origin/gh/aditew01/vecbf16 2025-12-04T08:27:07.0445941Z * [new branch] gh/albanD/4/base -> origin/gh/albanD/4/base 2025-12-04T08:27:07.0446286Z * [new branch] gh/albanD/4/head -> origin/gh/albanD/4/head 2025-12-04T08:27:07.0446648Z * [new branch] gh/albanD/4/orig -> origin/gh/albanD/4/orig 2025-12-04T08:27:07.0455223Z * [new branch] gh/alexbrauckmann/paddedtensor_faketensor_init -> origin/gh/alexbrauckmann/paddedtensor_faketensor_init 2025-12-04T08:27:07.0456505Z * [new branch] gh/alexsamardzic/12/base -> origin/gh/alexsamardzic/12/base 2025-12-04T08:27:07.0457386Z * [new branch] gh/alexsamardzic/12/head -> origin/gh/alexsamardzic/12/head 2025-12-04T08:27:07.0462048Z * [new branch] gh/alexsamardzic/12/orig -> origin/gh/alexsamardzic/12/orig 2025-12-04T08:27:07.0463977Z * [new branch] gh/alexsamardzic/14/base -> origin/gh/alexsamardzic/14/base 2025-12-04T08:27:07.0464512Z * [new branch] gh/alexsamardzic/14/head -> origin/gh/alexsamardzic/14/head 2025-12-04T08:27:07.0469978Z * [new branch] gh/alexsamardzic/14/orig -> origin/gh/alexsamardzic/14/orig 2025-12-04T08:27:07.0474657Z * [new branch] gh/alexsamardzic/15/base -> origin/gh/alexsamardzic/15/base 2025-12-04T08:27:07.0476482Z * [new branch] gh/alexsamardzic/15/head -> origin/gh/alexsamardzic/15/head 2025-12-04T08:27:07.0477073Z * [new branch] gh/alexsamardzic/15/orig -> origin/gh/alexsamardzic/15/orig 2025-12-04T08:27:07.0477772Z * [new branch] gh/amjames/18/base -> origin/gh/amjames/18/base 2025-12-04T08:27:07.0478325Z * [new branch] gh/amjames/18/head -> origin/gh/amjames/18/head 2025-12-04T08:27:07.0478728Z * [new branch] gh/amjames/18/orig -> origin/gh/amjames/18/orig 2025-12-04T08:27:07.0485180Z * [new branch] gh/andrewor14/35/base -> origin/gh/andrewor14/35/base 2025-12-04T08:27:07.0488727Z * [new branch] gh/andrewor14/35/head -> origin/gh/andrewor14/35/head 2025-12-04T08:27:07.0491830Z * [new branch] gh/andrewor14/35/orig -> origin/gh/andrewor14/35/orig 2025-12-04T08:27:07.0492454Z * [new branch] gh/andrewor14/50/base -> origin/gh/andrewor14/50/base 2025-12-04T08:27:07.0493014Z * [new branch] gh/andrewor14/50/head -> origin/gh/andrewor14/50/head 2025-12-04T08:27:07.0493953Z * [new branch] gh/andrewor14/50/orig -> origin/gh/andrewor14/50/orig 2025-12-04T08:27:07.0494457Z * [new branch] gh/andyanwang/30/base -> origin/gh/andyanwang/30/base 2025-12-04T08:27:07.0494871Z * [new branch] gh/andyanwang/30/orig -> origin/gh/andyanwang/30/orig 2025-12-04T08:27:07.0495227Z * [new branch] gh/andyanwang/31/base -> origin/gh/andyanwang/31/base 2025-12-04T08:27:07.0495587Z * [new branch] gh/andyanwang/31/orig -> origin/gh/andyanwang/31/orig 2025-12-04T08:27:07.0495932Z * [new branch] gh/andyanwang/39/base -> origin/gh/andyanwang/39/base 2025-12-04T08:27:07.0496270Z * [new branch] gh/andyanwang/39/head -> origin/gh/andyanwang/39/head 2025-12-04T08:27:07.0496616Z * [new branch] gh/andyanwang/39/orig -> origin/gh/andyanwang/39/orig 2025-12-04T08:27:07.0496962Z * [new branch] gh/andyanwang/42/base -> origin/gh/andyanwang/42/base 2025-12-04T08:27:07.0497309Z * [new branch] gh/andyanwang/42/head -> origin/gh/andyanwang/42/head 2025-12-04T08:27:07.0497652Z * [new branch] gh/andyanwang/42/orig -> origin/gh/andyanwang/42/orig 2025-12-04T08:27:07.0497995Z * [new branch] gh/andyanwang/45/base -> origin/gh/andyanwang/45/base 2025-12-04T08:27:07.0498509Z * [new branch] gh/andyanwang/45/head -> origin/gh/andyanwang/45/head 2025-12-04T08:27:07.0498851Z * [new branch] gh/andyanwang/45/orig -> origin/gh/andyanwang/45/orig 2025-12-04T08:27:07.0499204Z * [new branch] gh/angelayi/107/base -> origin/gh/angelayi/107/base 2025-12-04T08:27:07.0499554Z * [new branch] gh/angelayi/107/head -> origin/gh/angelayi/107/head 2025-12-04T08:27:07.0499924Z * [new branch] gh/angelayi/114/base -> origin/gh/angelayi/114/base 2025-12-04T08:27:07.0500290Z * [new branch] gh/angelayi/114/head -> origin/gh/angelayi/114/head 2025-12-04T08:27:07.0500631Z * [new branch] gh/angelayi/114/orig -> origin/gh/angelayi/114/orig 2025-12-04T08:27:07.0500973Z * [new branch] gh/angelayi/116/base -> origin/gh/angelayi/116/base 2025-12-04T08:27:07.0501311Z * [new branch] gh/angelayi/116/head -> origin/gh/angelayi/116/head 2025-12-04T08:27:07.0501645Z * [new branch] gh/angelayi/116/orig -> origin/gh/angelayi/116/orig 2025-12-04T08:27:07.0501983Z * [new branch] gh/angelayi/122/base -> origin/gh/angelayi/122/base 2025-12-04T08:27:07.0502322Z * [new branch] gh/angelayi/122/head -> origin/gh/angelayi/122/head 2025-12-04T08:27:07.0502658Z * [new branch] gh/angelayi/122/orig -> origin/gh/angelayi/122/orig 2025-12-04T08:27:07.0502984Z * [new branch] gh/angelayi/124/base -> origin/gh/angelayi/124/base 2025-12-04T08:27:07.0503316Z * [new branch] gh/angelayi/124/head -> origin/gh/angelayi/124/head 2025-12-04T08:27:07.0503694Z * [new branch] gh/angelayi/124/orig -> origin/gh/angelayi/124/orig 2025-12-04T08:27:07.0504036Z * [new branch] gh/angelayi/128/base -> origin/gh/angelayi/128/base 2025-12-04T08:27:07.0504370Z * [new branch] gh/angelayi/128/head -> origin/gh/angelayi/128/head 2025-12-04T08:27:07.0504709Z * [new branch] gh/angelayi/128/orig -> origin/gh/angelayi/128/orig 2025-12-04T08:27:07.0505044Z * [new branch] gh/angelayi/131/base -> origin/gh/angelayi/131/base 2025-12-04T08:27:07.0505367Z * [new branch] gh/angelayi/131/head -> origin/gh/angelayi/131/head 2025-12-04T08:27:07.0505701Z * [new branch] gh/angelayi/131/orig -> origin/gh/angelayi/131/orig 2025-12-04T08:27:07.0506035Z * [new branch] gh/angelayi/132/base -> origin/gh/angelayi/132/base 2025-12-04T08:27:07.0506370Z * [new branch] gh/angelayi/132/head -> origin/gh/angelayi/132/head 2025-12-04T08:27:07.0506698Z * [new branch] gh/angelayi/132/orig -> origin/gh/angelayi/132/orig 2025-12-04T08:27:07.0507062Z * [new branch] gh/angelayi/133/base -> origin/gh/angelayi/133/base 2025-12-04T08:27:07.0507433Z * [new branch] gh/angelayi/133/head -> origin/gh/angelayi/133/head 2025-12-04T08:27:07.0507794Z * [new branch] gh/angelayi/133/orig -> origin/gh/angelayi/133/orig 2025-12-04T08:27:07.0508198Z * [new branch] gh/angelayi/134/base -> origin/gh/angelayi/134/base 2025-12-04T08:27:07.0508568Z * [new branch] gh/angelayi/134/head -> origin/gh/angelayi/134/head 2025-12-04T08:27:07.0508942Z * [new branch] gh/angelayi/134/orig -> origin/gh/angelayi/134/orig 2025-12-04T08:27:07.0509296Z * [new branch] gh/angelayi/135/base -> origin/gh/angelayi/135/base 2025-12-04T08:27:07.0509652Z * [new branch] gh/angelayi/135/head -> origin/gh/angelayi/135/head 2025-12-04T08:27:07.0510014Z * [new branch] gh/angelayi/135/orig -> origin/gh/angelayi/135/orig 2025-12-04T08:27:07.0510366Z * [new branch] gh/angelayi/136/base -> origin/gh/angelayi/136/base 2025-12-04T08:27:07.0510764Z * [new branch] gh/angelayi/136/head -> origin/gh/angelayi/136/head 2025-12-04T08:27:07.0511132Z * [new branch] gh/angelayi/136/orig -> origin/gh/angelayi/136/orig 2025-12-04T08:27:07.0511496Z * [new branch] gh/angelayi/137/base -> origin/gh/angelayi/137/base 2025-12-04T08:27:07.0511870Z * [new branch] gh/angelayi/137/head -> origin/gh/angelayi/137/head 2025-12-04T08:27:07.0512240Z * [new branch] gh/angelayi/137/orig -> origin/gh/angelayi/137/orig 2025-12-04T08:27:07.0512606Z * [new branch] gh/angelayi/138/base -> origin/gh/angelayi/138/base 2025-12-04T08:27:07.0512977Z * [new branch] gh/angelayi/138/head -> origin/gh/angelayi/138/head 2025-12-04T08:27:07.0513350Z * [new branch] gh/angelayi/138/orig -> origin/gh/angelayi/138/orig 2025-12-04T08:27:07.0513703Z * [new branch] gh/angelayi/139/base -> origin/gh/angelayi/139/base 2025-12-04T08:27:07.0514067Z * [new branch] gh/angelayi/139/head -> origin/gh/angelayi/139/head 2025-12-04T08:27:07.0514434Z * [new branch] gh/angelayi/139/orig -> origin/gh/angelayi/139/orig 2025-12-04T08:27:07.0514800Z * [new branch] gh/angelayi/140/base -> origin/gh/angelayi/140/base 2025-12-04T08:27:07.0515171Z * [new branch] gh/angelayi/140/head -> origin/gh/angelayi/140/head 2025-12-04T08:27:07.0515536Z * [new branch] gh/angelayi/140/orig -> origin/gh/angelayi/140/orig 2025-12-04T08:27:07.0515910Z * [new branch] gh/angelayi/141/base -> origin/gh/angelayi/141/base 2025-12-04T08:27:07.0516562Z * [new branch] gh/angelayi/141/head -> origin/gh/angelayi/141/head 2025-12-04T08:27:07.0516945Z * [new branch] gh/angelayi/141/orig -> origin/gh/angelayi/141/orig 2025-12-04T08:27:07.0517319Z * [new branch] gh/angelayi/142/base -> origin/gh/angelayi/142/base 2025-12-04T08:27:07.0517697Z * [new branch] gh/angelayi/142/head -> origin/gh/angelayi/142/head 2025-12-04T08:27:07.0518280Z * [new branch] gh/angelayi/142/orig -> origin/gh/angelayi/142/orig 2025-12-04T08:27:07.0518689Z * [new branch] gh/angelayi/143/base -> origin/gh/angelayi/143/base 2025-12-04T08:27:07.0519063Z * [new branch] gh/angelayi/143/head -> origin/gh/angelayi/143/head 2025-12-04T08:27:07.0519432Z * [new branch] gh/angelayi/143/orig -> origin/gh/angelayi/143/orig 2025-12-04T08:27:07.0519790Z * [new branch] gh/angelayi/144/base -> origin/gh/angelayi/144/base 2025-12-04T08:27:07.0520163Z * [new branch] gh/angelayi/144/head -> origin/gh/angelayi/144/head 2025-12-04T08:27:07.0520512Z * [new branch] gh/angelayi/144/orig -> origin/gh/angelayi/144/orig 2025-12-04T08:27:07.0521119Z * [new branch] gh/anijain2305/753/base -> origin/gh/anijain2305/753/base 2025-12-04T08:27:07.0521498Z * [new branch] gh/anijain2305/753/head -> origin/gh/anijain2305/753/head 2025-12-04T08:27:07.0521881Z * [new branch] gh/anijain2305/753/orig -> origin/gh/anijain2305/753/orig 2025-12-04T08:27:07.0522265Z * [new branch] gh/anijain2305/810/base -> origin/gh/anijain2305/810/base 2025-12-04T08:27:07.0522625Z * [new branch] gh/anijain2305/810/head -> origin/gh/anijain2305/810/head 2025-12-04T08:27:07.0522991Z * [new branch] gh/anijain2305/810/orig -> origin/gh/anijain2305/810/orig 2025-12-04T08:27:07.0523528Z * [new branch] gh/anijain2305/854/base -> origin/gh/anijain2305/854/base 2025-12-04T08:27:07.0529720Z * [new branch] gh/anijain2305/854/head -> origin/gh/anijain2305/854/head 2025-12-04T08:27:07.0532094Z * [new branch] gh/anijain2305/854/orig -> origin/gh/anijain2305/854/orig 2025-12-04T08:27:07.0537253Z * [new branch] gh/anijain2305/864/base -> origin/gh/anijain2305/864/base 2025-12-04T08:27:07.0542899Z * [new branch] gh/anijain2305/864/head -> origin/gh/anijain2305/864/head 2025-12-04T08:27:07.0548606Z * [new branch] gh/anijain2305/864/orig -> origin/gh/anijain2305/864/orig 2025-12-04T08:27:07.0550978Z * [new branch] gh/anijain2305/870/base -> origin/gh/anijain2305/870/base 2025-12-04T08:27:07.0551380Z * [new branch] gh/anijain2305/870/head -> origin/gh/anijain2305/870/head 2025-12-04T08:27:07.0551874Z * [new branch] gh/anijain2305/870/orig -> origin/gh/anijain2305/870/orig 2025-12-04T08:27:07.0555459Z * [new branch] gh/anijain2305/873/base -> origin/gh/anijain2305/873/base 2025-12-04T08:27:07.0555911Z * [new branch] gh/anijain2305/873/head -> origin/gh/anijain2305/873/head 2025-12-04T08:27:07.0556273Z * [new branch] gh/anijain2305/873/orig -> origin/gh/anijain2305/873/orig 2025-12-04T08:27:07.0556666Z * [new branch] gh/anijain2305/894/base -> origin/gh/anijain2305/894/base 2025-12-04T08:27:07.0557035Z * [new branch] gh/anijain2305/894/head -> origin/gh/anijain2305/894/head 2025-12-04T08:27:07.0557404Z * [new branch] gh/anijain2305/894/orig -> origin/gh/anijain2305/894/orig 2025-12-04T08:27:07.0557772Z * [new branch] gh/anijain2305/895/base -> origin/gh/anijain2305/895/base 2025-12-04T08:27:07.0558343Z * [new branch] gh/anijain2305/895/head -> origin/gh/anijain2305/895/head 2025-12-04T08:27:07.0558716Z * [new branch] gh/anijain2305/895/orig -> origin/gh/anijain2305/895/orig 2025-12-04T08:27:07.0559287Z * [new branch] gh/anijain2305/910/base -> origin/gh/anijain2305/910/base 2025-12-04T08:27:07.0559675Z * [new branch] gh/anijain2305/910/head -> origin/gh/anijain2305/910/head 2025-12-04T08:27:07.0560050Z * [new branch] gh/anijain2305/910/orig -> origin/gh/anijain2305/910/orig 2025-12-04T08:27:07.0560413Z * [new branch] gh/anijain2305/919/base -> origin/gh/anijain2305/919/base 2025-12-04T08:27:07.0560783Z * [new branch] gh/anijain2305/919/head -> origin/gh/anijain2305/919/head 2025-12-04T08:27:07.0561150Z * [new branch] gh/anijain2305/919/orig -> origin/gh/anijain2305/919/orig 2025-12-04T08:27:07.0561516Z * [new branch] gh/anijain2305/922/base -> origin/gh/anijain2305/922/base 2025-12-04T08:27:07.0561868Z * [new branch] gh/anijain2305/922/head -> origin/gh/anijain2305/922/head 2025-12-04T08:27:07.0562230Z * [new branch] gh/anijain2305/922/orig -> origin/gh/anijain2305/922/orig 2025-12-04T08:27:07.0562596Z * [new branch] gh/anijain2305/932/base -> origin/gh/anijain2305/932/base 2025-12-04T08:27:07.0562951Z * [new branch] gh/anijain2305/932/head -> origin/gh/anijain2305/932/head 2025-12-04T08:27:07.0563302Z * [new branch] gh/anijain2305/932/orig -> origin/gh/anijain2305/932/orig 2025-12-04T08:27:07.0563655Z * [new branch] gh/anijain2305/940/base -> origin/gh/anijain2305/940/base 2025-12-04T08:27:07.0564012Z * [new branch] gh/anijain2305/940/head -> origin/gh/anijain2305/940/head 2025-12-04T08:27:07.0564371Z * [new branch] gh/anijain2305/940/orig -> origin/gh/anijain2305/940/orig 2025-12-04T08:27:07.0564737Z * [new branch] gh/anijain2305/941/base -> origin/gh/anijain2305/941/base 2025-12-04T08:27:07.0565106Z * [new branch] gh/anijain2305/941/head -> origin/gh/anijain2305/941/head 2025-12-04T08:27:07.0565476Z * [new branch] gh/anijain2305/941/orig -> origin/gh/anijain2305/941/orig 2025-12-04T08:27:07.0565836Z * [new branch] gh/anijain2305/942/base -> origin/gh/anijain2305/942/base 2025-12-04T08:27:07.0566274Z * [new branch] gh/anijain2305/942/head -> origin/gh/anijain2305/942/head 2025-12-04T08:27:07.0566635Z * [new branch] gh/anijain2305/942/orig -> origin/gh/anijain2305/942/orig 2025-12-04T08:27:07.0566990Z * [new branch] gh/anijain2305/943/base -> origin/gh/anijain2305/943/base 2025-12-04T08:27:07.0567352Z * [new branch] gh/anijain2305/943/head -> origin/gh/anijain2305/943/head 2025-12-04T08:27:07.0567712Z * [new branch] gh/anijain2305/943/orig -> origin/gh/anijain2305/943/orig 2025-12-04T08:27:07.0568102Z * [new branch] gh/anijain2305/944/base -> origin/gh/anijain2305/944/base 2025-12-04T08:27:07.0568464Z * [new branch] gh/anijain2305/944/head -> origin/gh/anijain2305/944/head 2025-12-04T08:27:07.0568830Z * [new branch] gh/anijain2305/944/orig -> origin/gh/anijain2305/944/orig 2025-12-04T08:27:07.0569195Z * [new branch] gh/anijain2305/945/base -> origin/gh/anijain2305/945/base 2025-12-04T08:27:07.0569565Z * [new branch] gh/anijain2305/945/head -> origin/gh/anijain2305/945/head 2025-12-04T08:27:07.0569929Z * [new branch] gh/anijain2305/945/orig -> origin/gh/anijain2305/945/orig 2025-12-04T08:27:07.0570300Z * [new branch] gh/anijain2305/946/base -> origin/gh/anijain2305/946/base 2025-12-04T08:27:07.0570666Z * [new branch] gh/anijain2305/946/head -> origin/gh/anijain2305/946/head 2025-12-04T08:27:07.0571045Z * [new branch] gh/anijain2305/946/orig -> origin/gh/anijain2305/946/orig 2025-12-04T08:27:07.0571426Z * [new branch] gh/anijain2305/947/base -> origin/gh/anijain2305/947/base 2025-12-04T08:27:07.0571836Z * [new branch] gh/anijain2305/947/head -> origin/gh/anijain2305/947/head 2025-12-04T08:27:07.0572210Z * [new branch] gh/anijain2305/947/orig -> origin/gh/anijain2305/947/orig 2025-12-04T08:27:07.0572581Z * [new branch] gh/anijain2305/948/base -> origin/gh/anijain2305/948/base 2025-12-04T08:27:07.0572938Z * [new branch] gh/anijain2305/948/head -> origin/gh/anijain2305/948/head 2025-12-04T08:27:07.0573302Z * [new branch] gh/anijain2305/948/orig -> origin/gh/anijain2305/948/orig 2025-12-04T08:27:07.0573665Z * [new branch] gh/anijain2305/949/base -> origin/gh/anijain2305/949/base 2025-12-04T08:27:07.0574031Z * [new branch] gh/anijain2305/949/head -> origin/gh/anijain2305/949/head 2025-12-04T08:27:07.0574580Z * [new branch] gh/anijain2305/949/orig -> origin/gh/anijain2305/949/orig 2025-12-04T08:27:07.0575052Z * [new branch] gh/anijain2305/950/base -> origin/gh/anijain2305/950/base 2025-12-04T08:27:07.0580921Z * [new branch] gh/anijain2305/950/head -> origin/gh/anijain2305/950/head 2025-12-04T08:27:07.0581379Z * [new branch] gh/anijain2305/950/orig -> origin/gh/anijain2305/950/orig 2025-12-04T08:27:07.0581798Z * [new branch] gh/anijain2305/951/base -> origin/gh/anijain2305/951/base 2025-12-04T08:27:07.0582174Z * [new branch] gh/anijain2305/951/head -> origin/gh/anijain2305/951/head 2025-12-04T08:27:07.0582547Z * [new branch] gh/anijain2305/951/orig -> origin/gh/anijain2305/951/orig 2025-12-04T08:27:07.0582919Z * [new branch] gh/anijain2305/952/base -> origin/gh/anijain2305/952/base 2025-12-04T08:27:07.0583284Z * [new branch] gh/anijain2305/952/head -> origin/gh/anijain2305/952/head 2025-12-04T08:27:07.0583661Z * [new branch] gh/anijain2305/952/orig -> origin/gh/anijain2305/952/orig 2025-12-04T08:27:07.0584037Z * [new branch] gh/anijain2305/953/base -> origin/gh/anijain2305/953/base 2025-12-04T08:27:07.0584397Z * [new branch] gh/anijain2305/953/head -> origin/gh/anijain2305/953/head 2025-12-04T08:27:07.0584936Z * [new branch] gh/anijain2305/953/orig -> origin/gh/anijain2305/953/orig 2025-12-04T08:27:07.0585305Z * [new branch] gh/anijain2305/954/base -> origin/gh/anijain2305/954/base 2025-12-04T08:27:07.0585674Z * [new branch] gh/anijain2305/954/head -> origin/gh/anijain2305/954/head 2025-12-04T08:27:07.0586036Z * [new branch] gh/anijain2305/954/orig -> origin/gh/anijain2305/954/orig 2025-12-04T08:27:07.0586404Z * [new branch] gh/anijain2305/955/base -> origin/gh/anijain2305/955/base 2025-12-04T08:27:07.0586785Z * [new branch] gh/anijain2305/955/head -> origin/gh/anijain2305/955/head 2025-12-04T08:27:07.0587177Z * [new branch] gh/anijain2305/955/orig -> origin/gh/anijain2305/955/orig 2025-12-04T08:27:07.0587563Z * [new branch] gh/anijain2305/956/base -> origin/gh/anijain2305/956/base 2025-12-04T08:27:07.0587953Z * [new branch] gh/anijain2305/956/head -> origin/gh/anijain2305/956/head 2025-12-04T08:27:07.0588350Z * [new branch] gh/anijain2305/956/orig -> origin/gh/anijain2305/956/orig 2025-12-04T08:27:07.0588720Z * [new branch] gh/anijain2305/957/base -> origin/gh/anijain2305/957/base 2025-12-04T08:27:07.0589111Z * [new branch] gh/anijain2305/957/head -> origin/gh/anijain2305/957/head 2025-12-04T08:27:07.0589507Z * [new branch] gh/anijain2305/957/orig -> origin/gh/anijain2305/957/orig 2025-12-04T08:27:07.0589896Z * [new branch] gh/anijain2305/958/base -> origin/gh/anijain2305/958/base 2025-12-04T08:27:07.0590286Z * [new branch] gh/anijain2305/958/head -> origin/gh/anijain2305/958/head 2025-12-04T08:27:07.0590939Z * [new branch] gh/anijain2305/958/orig -> origin/gh/anijain2305/958/orig 2025-12-04T08:27:07.0591349Z * [new branch] gh/anijain2305/959/base -> origin/gh/anijain2305/959/base 2025-12-04T08:27:07.0591876Z * [new branch] gh/anijain2305/959/head -> origin/gh/anijain2305/959/head 2025-12-04T08:27:07.0593430Z * [new branch] gh/anijain2305/959/orig -> origin/gh/anijain2305/959/orig 2025-12-04T08:27:07.0593896Z * [new branch] gh/anijain2305/960/base -> origin/gh/anijain2305/960/base 2025-12-04T08:27:07.0594361Z * [new branch] gh/anijain2305/960/head -> origin/gh/anijain2305/960/head 2025-12-04T08:27:07.0595049Z * [new branch] gh/anijain2305/960/orig -> origin/gh/anijain2305/960/orig 2025-12-04T08:27:07.0598558Z * [new branch] gh/anijain2305/961/base -> origin/gh/anijain2305/961/base 2025-12-04T08:27:07.0599042Z * [new branch] gh/anijain2305/961/head -> origin/gh/anijain2305/961/head 2025-12-04T08:27:07.0599443Z * [new branch] gh/anijain2305/961/orig -> origin/gh/anijain2305/961/orig 2025-12-04T08:27:07.0599822Z * [new branch] gh/anijain2305/962/base -> origin/gh/anijain2305/962/base 2025-12-04T08:27:07.0600414Z * [new branch] gh/anijain2305/962/head -> origin/gh/anijain2305/962/head 2025-12-04T08:27:07.0600959Z * [new branch] gh/anijain2305/962/orig -> origin/gh/anijain2305/962/orig 2025-12-04T08:27:07.0601603Z * [new branch] gh/anijain2305/963/base -> origin/gh/anijain2305/963/base 2025-12-04T08:27:07.0603179Z * [new branch] gh/anijain2305/963/head -> origin/gh/anijain2305/963/head 2025-12-04T08:27:07.0603628Z * [new branch] gh/anijain2305/963/orig -> origin/gh/anijain2305/963/orig 2025-12-04T08:27:07.0604489Z * [new branch] gh/anijain2305/964/base -> origin/gh/anijain2305/964/base 2025-12-04T08:27:07.0605195Z * [new branch] gh/anijain2305/964/head -> origin/gh/anijain2305/964/head 2025-12-04T08:27:07.0605913Z * [new branch] gh/anijain2305/964/orig -> origin/gh/anijain2305/964/orig 2025-12-04T08:27:07.0606840Z * [new branch] gh/anijain2305/965/base -> origin/gh/anijain2305/965/base 2025-12-04T08:27:07.0607516Z * [new branch] gh/anijain2305/965/head -> origin/gh/anijain2305/965/head 2025-12-04T08:27:07.0608327Z * [new branch] gh/anijain2305/965/orig -> origin/gh/anijain2305/965/orig 2025-12-04T08:27:07.0609181Z * [new branch] gh/anijain2305/966/base -> origin/gh/anijain2305/966/base 2025-12-04T08:27:07.0609851Z * [new branch] gh/anijain2305/966/head -> origin/gh/anijain2305/966/head 2025-12-04T08:27:07.0610561Z * [new branch] gh/anijain2305/966/orig -> origin/gh/anijain2305/966/orig 2025-12-04T08:27:07.0611826Z * [new branch] gh/anijain2305/967/base -> origin/gh/anijain2305/967/base 2025-12-04T08:27:07.0612240Z * [new branch] gh/anijain2305/967/head -> origin/gh/anijain2305/967/head 2025-12-04T08:27:07.0613094Z * [new branch] gh/anijain2305/967/orig -> origin/gh/anijain2305/967/orig 2025-12-04T08:27:07.0615796Z * [new branch] gh/anijain2305/968/base -> origin/gh/anijain2305/968/base 2025-12-04T08:27:07.0616238Z * [new branch] gh/anijain2305/968/head -> origin/gh/anijain2305/968/head 2025-12-04T08:27:07.0616620Z * [new branch] gh/anijain2305/968/orig -> origin/gh/anijain2305/968/orig 2025-12-04T08:27:07.0617000Z * [new branch] gh/anijain2305/969/base -> origin/gh/anijain2305/969/base 2025-12-04T08:27:07.0617386Z * [new branch] gh/anijain2305/969/head -> origin/gh/anijain2305/969/head 2025-12-04T08:27:07.0618803Z * [new branch] gh/anijain2305/969/orig -> origin/gh/anijain2305/969/orig 2025-12-04T08:27:07.0619370Z * [new branch] gh/anijain2305/970/base -> origin/gh/anijain2305/970/base 2025-12-04T08:27:07.0619758Z * [new branch] gh/anijain2305/970/head -> origin/gh/anijain2305/970/head 2025-12-04T08:27:07.0620169Z * [new branch] gh/anijain2305/970/orig -> origin/gh/anijain2305/970/orig 2025-12-04T08:27:07.0623892Z * [new branch] gh/anjali411/216/base -> origin/gh/anjali411/216/base 2025-12-04T08:27:07.0629445Z * [new branch] gh/anjali411/216/head -> origin/gh/anjali411/216/head 2025-12-04T08:27:07.0629890Z * [new branch] gh/anjali411/216/orig -> origin/gh/anjali411/216/orig 2025-12-04T08:27:07.0630277Z * [new branch] gh/anshul-si/1/base -> origin/gh/anshul-si/1/base 2025-12-04T08:27:07.0630638Z * [new branch] gh/anshul-si/1/head -> origin/gh/anshul-si/1/head 2025-12-04T08:27:07.0631006Z * [new branch] gh/anshul-si/2/base -> origin/gh/anshul-si/2/base 2025-12-04T08:27:07.0631378Z * [new branch] gh/anshul-si/2/head -> origin/gh/anshul-si/2/head 2025-12-04T08:27:07.0631726Z * [new branch] gh/anshul-si/3/base -> origin/gh/anshul-si/3/base 2025-12-04T08:27:07.0632072Z * [new branch] gh/anshul-si/3/head -> origin/gh/anshul-si/3/head 2025-12-04T08:27:07.0632425Z * [new branch] gh/anshul-si/4/base -> origin/gh/anshul-si/4/base 2025-12-04T08:27:07.0632832Z * [new branch] gh/anshul-si/4/head -> origin/gh/anshul-si/4/head 2025-12-04T08:27:07.0633175Z * [new branch] gh/anshul-si/5/base -> origin/gh/anshul-si/5/base 2025-12-04T08:27:07.0633521Z * [new branch] gh/anshul-si/5/head -> origin/gh/anshul-si/5/head 2025-12-04T08:27:07.0633931Z * [new branch] gh/anshul-si/53/base -> origin/gh/anshul-si/53/base 2025-12-04T08:27:07.0634684Z * [new branch] gh/anshul-si/53/head -> origin/gh/anshul-si/53/head 2025-12-04T08:27:07.0635868Z * [new branch] gh/anshul-si/58/base -> origin/gh/anshul-si/58/base 2025-12-04T08:27:07.0636427Z * [new branch] gh/anshul-si/58/head -> origin/gh/anshul-si/58/head 2025-12-04T08:27:07.0637441Z * [new branch] gh/anshul-si/66/base -> origin/gh/anshul-si/66/base 2025-12-04T08:27:07.0638262Z * [new branch] gh/anshul-si/66/head -> origin/gh/anshul-si/66/head 2025-12-04T08:27:07.0638899Z * [new branch] gh/anshul-si/66/orig -> origin/gh/anshul-si/66/orig 2025-12-04T08:27:07.0643160Z * [new branch] gh/anshul-si/67/base -> origin/gh/anshul-si/67/base 2025-12-04T08:27:07.0645318Z * [new branch] gh/anshul-si/67/head -> origin/gh/anshul-si/67/head 2025-12-04T08:27:07.0645708Z * [new branch] gh/anshul-si/67/orig -> origin/gh/anshul-si/67/orig 2025-12-04T08:27:07.0646079Z * [new branch] gh/anshul-si/68/base -> origin/gh/anshul-si/68/base 2025-12-04T08:27:07.0646424Z * [new branch] gh/anshul-si/68/head -> origin/gh/anshul-si/68/head 2025-12-04T08:27:07.0646768Z * [new branch] gh/anshul-si/68/orig -> origin/gh/anshul-si/68/orig 2025-12-04T08:27:07.0647119Z * [new branch] gh/anshul-si/69/base -> origin/gh/anshul-si/69/base 2025-12-04T08:27:07.0647463Z * [new branch] gh/anshul-si/69/head -> origin/gh/anshul-si/69/head 2025-12-04T08:27:07.0649536Z * [new branch] gh/anshul-si/69/orig -> origin/gh/anshul-si/69/orig 2025-12-04T08:27:07.0649969Z * [new branch] gh/anshul-si/70/base -> origin/gh/anshul-si/70/base 2025-12-04T08:27:07.0654994Z * [new branch] gh/anshul-si/70/head -> origin/gh/anshul-si/70/head 2025-12-04T08:27:07.0659792Z * [new branch] gh/anshul-si/70/orig -> origin/gh/anshul-si/70/orig 2025-12-04T08:27:07.0664248Z * [new branch] gh/anshul-si/71/base -> origin/gh/anshul-si/71/base 2025-12-04T08:27:07.0664680Z * [new branch] gh/anshul-si/71/head -> origin/gh/anshul-si/71/head 2025-12-04T08:27:07.0665012Z * [new branch] gh/anshul-si/71/orig -> origin/gh/anshul-si/71/orig 2025-12-04T08:27:07.0665373Z * [new branch] gh/anshul-si/72/base -> origin/gh/anshul-si/72/base 2025-12-04T08:27:07.0665706Z * [new branch] gh/anshul-si/72/head -> origin/gh/anshul-si/72/head 2025-12-04T08:27:07.0666031Z * [new branch] gh/anshul-si/72/orig -> origin/gh/anshul-si/72/orig 2025-12-04T08:27:07.0666367Z * [new branch] gh/anshul-si/73/base -> origin/gh/anshul-si/73/base 2025-12-04T08:27:07.0666701Z * [new branch] gh/anshul-si/73/head -> origin/gh/anshul-si/73/head 2025-12-04T08:27:07.0667038Z * [new branch] gh/anshul-si/73/orig -> origin/gh/anshul-si/73/orig 2025-12-04T08:27:07.0667386Z * [new branch] gh/aorenste/132/base -> origin/gh/aorenste/132/base 2025-12-04T08:27:07.0667732Z * [new branch] gh/aorenste/132/head -> origin/gh/aorenste/132/head 2025-12-04T08:27:07.0668078Z * [new branch] gh/aorenste/134/base -> origin/gh/aorenste/134/base 2025-12-04T08:27:07.0668417Z * [new branch] gh/aorenste/134/head -> origin/gh/aorenste/134/head 2025-12-04T08:27:07.0668746Z * [new branch] gh/aorenste/134/orig -> origin/gh/aorenste/134/orig 2025-12-04T08:27:07.0669085Z * [new branch] gh/aorenste/139/base -> origin/gh/aorenste/139/base 2025-12-04T08:27:07.0669416Z * [new branch] gh/aorenste/139/head -> origin/gh/aorenste/139/head 2025-12-04T08:27:07.0669751Z * [new branch] gh/aorenste/139/orig -> origin/gh/aorenste/139/orig 2025-12-04T08:27:07.0670080Z * [new branch] gh/aorenste/141/base -> origin/gh/aorenste/141/base 2025-12-04T08:27:07.0670415Z * [new branch] gh/aorenste/141/head -> origin/gh/aorenste/141/head 2025-12-04T08:27:07.0670752Z * [new branch] gh/aorenste/145/base -> origin/gh/aorenste/145/base 2025-12-04T08:27:07.0671151Z * [new branch] gh/aorenste/145/head -> origin/gh/aorenste/145/head 2025-12-04T08:27:07.0671504Z * [new branch] gh/aorenste/145/orig -> origin/gh/aorenste/145/orig 2025-12-04T08:27:07.0671873Z * [new branch] gh/aorenste/146/base -> origin/gh/aorenste/146/base 2025-12-04T08:27:07.0672211Z * [new branch] gh/aorenste/146/head -> origin/gh/aorenste/146/head 2025-12-04T08:27:07.0672542Z * [new branch] gh/aorenste/146/orig -> origin/gh/aorenste/146/orig 2025-12-04T08:27:07.0672882Z * [new branch] gh/aorenste/147/base -> origin/gh/aorenste/147/base 2025-12-04T08:27:07.0673224Z * [new branch] gh/aorenste/147/head -> origin/gh/aorenste/147/head 2025-12-04T08:27:07.0674072Z * [new branch] gh/aorenste/147/orig -> origin/gh/aorenste/147/orig 2025-12-04T08:27:07.0674433Z * [new branch] gh/aorenste/148/base -> origin/gh/aorenste/148/base 2025-12-04T08:27:07.0674803Z * [new branch] gh/aorenste/148/head -> origin/gh/aorenste/148/head 2025-12-04T08:27:07.0675173Z * [new branch] gh/aorenste/148/orig -> origin/gh/aorenste/148/orig 2025-12-04T08:27:07.0675540Z * [new branch] gh/aorenste/149/base -> origin/gh/aorenste/149/base 2025-12-04T08:27:07.0675902Z * [new branch] gh/aorenste/149/head -> origin/gh/aorenste/149/head 2025-12-04T08:27:07.0676286Z * [new branch] gh/aorenste/149/orig -> origin/gh/aorenste/149/orig 2025-12-04T08:27:07.0677623Z * [new branch] gh/aorenste/150/base -> origin/gh/aorenste/150/base 2025-12-04T08:27:07.0678626Z * [new branch] gh/aorenste/150/head -> origin/gh/aorenste/150/head 2025-12-04T08:27:07.0679095Z * [new branch] gh/aorenste/150/orig -> origin/gh/aorenste/150/orig 2025-12-04T08:27:07.0679902Z * [new branch] gh/aorenste/151/base -> origin/gh/aorenste/151/base 2025-12-04T08:27:07.0680616Z * [new branch] gh/aorenste/151/head -> origin/gh/aorenste/151/head 2025-12-04T08:27:07.0681386Z * [new branch] gh/aorenste/151/orig -> origin/gh/aorenste/151/orig 2025-12-04T08:27:07.0682588Z * [new branch] gh/aorenste/152/base -> origin/gh/aorenste/152/base 2025-12-04T08:27:07.0683155Z * [new branch] gh/aorenste/152/head -> origin/gh/aorenste/152/head 2025-12-04T08:27:07.0683844Z * [new branch] gh/aorenste/152/orig -> origin/gh/aorenste/152/orig 2025-12-04T08:27:07.0684960Z * [new branch] gh/aorenste/153/base -> origin/gh/aorenste/153/base 2025-12-04T08:27:07.0685349Z * [new branch] gh/aorenste/153/head -> origin/gh/aorenste/153/head 2025-12-04T08:27:07.0686377Z * [new branch] gh/aorenste/153/orig -> origin/gh/aorenste/153/orig 2025-12-04T08:27:07.0687006Z * [new branch] gh/aorenste/154/base -> origin/gh/aorenste/154/base 2025-12-04T08:27:07.0687827Z * [new branch] gh/aorenste/154/head -> origin/gh/aorenste/154/head 2025-12-04T08:27:07.0688384Z * [new branch] gh/aorenste/154/orig -> origin/gh/aorenste/154/orig 2025-12-04T08:27:07.0689484Z * [new branch] gh/aorenste/155/base -> origin/gh/aorenste/155/base 2025-12-04T08:27:07.0689911Z * [new branch] gh/aorenste/155/head -> origin/gh/aorenste/155/head 2025-12-04T08:27:07.0692940Z * [new branch] gh/aorenste/155/orig -> origin/gh/aorenste/155/orig 2025-12-04T08:27:07.0693303Z * [new branch] gh/aorenste/156/base -> origin/gh/aorenste/156/base 2025-12-04T08:27:07.0693659Z * [new branch] gh/aorenste/156/head -> origin/gh/aorenste/156/head 2025-12-04T08:27:07.0694008Z * [new branch] gh/aorenste/156/orig -> origin/gh/aorenste/156/orig 2025-12-04T08:27:07.0694435Z * [new branch] gh/aorenste/157/base -> origin/gh/aorenste/157/base 2025-12-04T08:27:07.0694789Z * [new branch] gh/aorenste/157/head -> origin/gh/aorenste/157/head 2025-12-04T08:27:07.0695146Z * [new branch] gh/aorenste/157/orig -> origin/gh/aorenste/157/orig 2025-12-04T08:27:07.0699681Z * [new branch] gh/aorenste/158/base -> origin/gh/aorenste/158/base 2025-12-04T08:27:07.0700127Z * [new branch] gh/aorenste/158/head -> origin/gh/aorenste/158/head 2025-12-04T08:27:07.0700534Z * [new branch] gh/aorenste/158/orig -> origin/gh/aorenste/158/orig 2025-12-04T08:27:07.0700938Z * [new branch] gh/aorenste/159/base -> origin/gh/aorenste/159/base 2025-12-04T08:27:07.0701314Z * [new branch] gh/aorenste/159/head -> origin/gh/aorenste/159/head 2025-12-04T08:27:07.0701684Z * [new branch] gh/aorenste/159/orig -> origin/gh/aorenste/159/orig 2025-12-04T08:27:07.0702104Z * [new branch] gh/avikchaudhuri/1/base -> origin/gh/avikchaudhuri/1/base 2025-12-04T08:27:07.0704121Z * [new branch] gh/avikchaudhuri/1/head -> origin/gh/avikchaudhuri/1/head 2025-12-04T08:27:07.0704528Z * [new branch] gh/avikchaudhuri/2/base -> origin/gh/avikchaudhuri/2/base 2025-12-04T08:27:07.0704917Z * [new branch] gh/avikchaudhuri/2/head -> origin/gh/avikchaudhuri/2/head 2025-12-04T08:27:07.0705301Z * [new branch] gh/avikchaudhuri/2/orig -> origin/gh/avikchaudhuri/2/orig 2025-12-04T08:27:07.0705683Z * [new branch] gh/bdhirsh/666/base -> origin/gh/bdhirsh/666/base 2025-12-04T08:27:07.0706194Z * [new branch] gh/bdhirsh/666/head -> origin/gh/bdhirsh/666/head 2025-12-04T08:27:07.0708730Z * [new branch] gh/bdhirsh/666/orig -> origin/gh/bdhirsh/666/orig 2025-12-04T08:27:07.0709107Z * [new branch] gh/bdhirsh/668/base -> origin/gh/bdhirsh/668/base 2025-12-04T08:27:07.0709606Z * [new branch] gh/bdhirsh/668/head -> origin/gh/bdhirsh/668/head 2025-12-04T08:27:07.0709955Z * [new branch] gh/bdhirsh/668/orig -> origin/gh/bdhirsh/668/orig 2025-12-04T08:27:07.0710300Z * [new branch] gh/bdhirsh/669/base -> origin/gh/bdhirsh/669/base 2025-12-04T08:27:07.0710655Z * [new branch] gh/bdhirsh/669/head -> origin/gh/bdhirsh/669/head 2025-12-04T08:27:07.0713382Z * [new branch] gh/bdhirsh/669/orig -> origin/gh/bdhirsh/669/orig 2025-12-04T08:27:07.0713743Z * [new branch] gh/bdhirsh/670/base -> origin/gh/bdhirsh/670/base 2025-12-04T08:27:07.0714118Z * [new branch] gh/bdhirsh/670/head -> origin/gh/bdhirsh/670/head 2025-12-04T08:27:07.0714477Z * [new branch] gh/bdhirsh/670/orig -> origin/gh/bdhirsh/670/orig 2025-12-04T08:27:07.0714837Z * [new branch] gh/bdhirsh/672/base -> origin/gh/bdhirsh/672/base 2025-12-04T08:27:07.0715183Z * [new branch] gh/bdhirsh/672/head -> origin/gh/bdhirsh/672/head 2025-12-04T08:27:07.0715539Z * [new branch] gh/bdhirsh/672/orig -> origin/gh/bdhirsh/672/orig 2025-12-04T08:27:07.0715900Z * [new branch] gh/bdhirsh/675/base -> origin/gh/bdhirsh/675/base 2025-12-04T08:27:07.0716268Z * [new branch] gh/bdhirsh/675/head -> origin/gh/bdhirsh/675/head 2025-12-04T08:27:07.0717098Z * [new branch] gh/bdhirsh/675/orig -> origin/gh/bdhirsh/675/orig 2025-12-04T08:27:07.0718540Z * [new branch] gh/bdhirsh/676/base -> origin/gh/bdhirsh/676/base 2025-12-04T08:27:07.0718935Z * [new branch] gh/bdhirsh/676/head -> origin/gh/bdhirsh/676/head 2025-12-04T08:27:07.0719760Z * [new branch] gh/bdhirsh/676/orig -> origin/gh/bdhirsh/676/orig 2025-12-04T08:27:07.0721047Z * [new branch] gh/bdhirsh/677/base -> origin/gh/bdhirsh/677/base 2025-12-04T08:27:07.0725145Z * [new branch] gh/bdhirsh/677/head -> origin/gh/bdhirsh/677/head 2025-12-04T08:27:07.0725335Z * [new branch] gh/bdhirsh/677/orig -> origin/gh/bdhirsh/677/orig 2025-12-04T08:27:07.0725492Z * [new branch] gh/bdhirsh/678/base -> origin/gh/bdhirsh/678/base 2025-12-04T08:27:07.0725633Z * [new branch] gh/bdhirsh/678/head -> origin/gh/bdhirsh/678/head 2025-12-04T08:27:07.0725784Z * [new branch] gh/bdhirsh/678/orig -> origin/gh/bdhirsh/678/orig 2025-12-04T08:27:07.0727653Z * [new branch] gh/bdhirsh/679/base -> origin/gh/bdhirsh/679/base 2025-12-04T08:27:07.0728010Z * [new branch] gh/bdhirsh/679/head -> origin/gh/bdhirsh/679/head 2025-12-04T08:27:07.0728272Z * [new branch] gh/bdhirsh/679/orig -> origin/gh/bdhirsh/679/orig 2025-12-04T08:27:07.0731254Z * [new branch] gh/bdhirsh/680/base -> origin/gh/bdhirsh/680/base 2025-12-04T08:27:07.0731610Z * [new branch] gh/bdhirsh/680/head -> origin/gh/bdhirsh/680/head 2025-12-04T08:27:07.0731889Z * [new branch] gh/bdhirsh/680/orig -> origin/gh/bdhirsh/680/orig 2025-12-04T08:27:07.0732043Z * [new branch] gh/bdhirsh/681/base -> origin/gh/bdhirsh/681/base 2025-12-04T08:27:07.0732209Z * [new branch] gh/bdhirsh/681/head -> origin/gh/bdhirsh/681/head 2025-12-04T08:27:07.0733471Z * [new branch] gh/bdhirsh/681/orig -> origin/gh/bdhirsh/681/orig 2025-12-04T08:27:07.0737064Z * [new branch] gh/benjaminglass1/101/base -> origin/gh/benjaminglass1/101/base 2025-12-04T08:27:07.0737668Z * [new branch] gh/benjaminglass1/101/head -> origin/gh/benjaminglass1/101/head 2025-12-04T08:27:07.0738008Z * [new branch] gh/benjaminglass1/101/orig -> origin/gh/benjaminglass1/101/orig 2025-12-04T08:27:07.0738678Z * [new branch] gh/benjaminglass1/102/base -> origin/gh/benjaminglass1/102/base 2025-12-04T08:27:07.0742183Z * [new branch] gh/benjaminglass1/102/head -> origin/gh/benjaminglass1/102/head 2025-12-04T08:27:07.0742397Z * [new branch] gh/benjaminglass1/102/orig -> origin/gh/benjaminglass1/102/orig 2025-12-04T08:27:07.0742591Z * [new branch] gh/benjaminglass1/106/base -> origin/gh/benjaminglass1/106/base 2025-12-04T08:27:07.0742758Z * [new branch] gh/benjaminglass1/106/head -> origin/gh/benjaminglass1/106/head 2025-12-04T08:27:07.0742933Z * [new branch] gh/benjaminglass1/106/orig -> origin/gh/benjaminglass1/106/orig 2025-12-04T08:27:07.0743129Z * [new branch] gh/benjaminglass1/107/base -> origin/gh/benjaminglass1/107/base 2025-12-04T08:27:07.0744606Z * [new branch] gh/benjaminglass1/107/head -> origin/gh/benjaminglass1/107/head 2025-12-04T08:27:07.0744997Z * [new branch] gh/benjaminglass1/107/orig -> origin/gh/benjaminglass1/107/orig 2025-12-04T08:27:07.0745168Z * [new branch] gh/benjaminglass1/108/base -> origin/gh/benjaminglass1/108/base 2025-12-04T08:27:07.0745358Z * [new branch] gh/benjaminglass1/108/head -> origin/gh/benjaminglass1/108/head 2025-12-04T08:27:07.0745531Z * [new branch] gh/benjaminglass1/108/orig -> origin/gh/benjaminglass1/108/orig 2025-12-04T08:27:07.0745708Z * [new branch] gh/benjaminglass1/109/base -> origin/gh/benjaminglass1/109/base 2025-12-04T08:27:07.0750952Z * [new branch] gh/benjaminglass1/109/head -> origin/gh/benjaminglass1/109/head 2025-12-04T08:27:07.0751176Z * [new branch] gh/benjaminglass1/109/orig -> origin/gh/benjaminglass1/109/orig 2025-12-04T08:27:07.0751359Z * [new branch] gh/benjaminglass1/97/base -> origin/gh/benjaminglass1/97/base 2025-12-04T08:27:07.0751723Z * [new branch] gh/benjaminglass1/97/head -> origin/gh/benjaminglass1/97/head 2025-12-04T08:27:07.0751890Z * [new branch] gh/benjaminglass1/97/orig -> origin/gh/benjaminglass1/97/orig 2025-12-04T08:27:07.0752044Z * [new branch] gh/bobrenjc93/570/base -> origin/gh/bobrenjc93/570/base 2025-12-04T08:27:07.0753144Z * [new branch] gh/bobrenjc93/570/head -> origin/gh/bobrenjc93/570/head 2025-12-04T08:27:07.0753577Z * [new branch] gh/bobrenjc93/570/orig -> origin/gh/bobrenjc93/570/orig 2025-12-04T08:27:07.0753784Z * [new branch] gh/bobrenjc93/604/base -> origin/gh/bobrenjc93/604/base 2025-12-04T08:27:07.0753957Z * [new branch] gh/bobrenjc93/604/head -> origin/gh/bobrenjc93/604/head 2025-12-04T08:27:07.0754139Z * [new branch] gh/bobrenjc93/604/orig -> origin/gh/bobrenjc93/604/orig 2025-12-04T08:27:07.0754302Z * [new branch] gh/bobrenjc93/638/base -> origin/gh/bobrenjc93/638/base 2025-12-04T08:27:07.0754456Z * [new branch] gh/bobrenjc93/638/head -> origin/gh/bobrenjc93/638/head 2025-12-04T08:27:07.0755508Z * [new branch] gh/bobrenjc93/638/orig -> origin/gh/bobrenjc93/638/orig 2025-12-04T08:27:07.0756066Z * [new branch] gh/bobrenjc93/653/base -> origin/gh/bobrenjc93/653/base 2025-12-04T08:27:07.0757054Z * [new branch] gh/bobrenjc93/653/head -> origin/gh/bobrenjc93/653/head 2025-12-04T08:27:07.0757346Z * [new branch] gh/bobrenjc93/653/orig -> origin/gh/bobrenjc93/653/orig 2025-12-04T08:27:07.0761916Z * [new branch] gh/bobrenjc93/654/base -> origin/gh/bobrenjc93/654/base 2025-12-04T08:27:07.0766288Z * [new branch] gh/bobrenjc93/654/head -> origin/gh/bobrenjc93/654/head 2025-12-04T08:27:07.0767996Z * [new branch] gh/bobrenjc93/654/orig -> origin/gh/bobrenjc93/654/orig 2025-12-04T08:27:07.0768171Z * [new branch] gh/bobrenjc93/657/base -> origin/gh/bobrenjc93/657/base 2025-12-04T08:27:07.0768339Z * [new branch] gh/bobrenjc93/657/head -> origin/gh/bobrenjc93/657/head 2025-12-04T08:27:07.0768490Z * [new branch] gh/bobrenjc93/657/orig -> origin/gh/bobrenjc93/657/orig 2025-12-04T08:27:07.0768657Z * [new branch] gh/bobrenjc93/672/base -> origin/gh/bobrenjc93/672/base 2025-12-04T08:27:07.0768807Z * [new branch] gh/bobrenjc93/672/head -> origin/gh/bobrenjc93/672/head 2025-12-04T08:27:07.0768963Z * [new branch] gh/bobrenjc93/672/orig -> origin/gh/bobrenjc93/672/orig 2025-12-04T08:27:07.0769110Z * [new branch] gh/bobrenjc93/679/base -> origin/gh/bobrenjc93/679/base 2025-12-04T08:27:07.0769261Z * [new branch] gh/bobrenjc93/679/head -> origin/gh/bobrenjc93/679/head 2025-12-04T08:27:07.0769414Z * [new branch] gh/bobrenjc93/679/orig -> origin/gh/bobrenjc93/679/orig 2025-12-04T08:27:07.0774925Z * [new branch] gh/bobrenjc93/680/base -> origin/gh/bobrenjc93/680/base 2025-12-04T08:27:07.0775121Z * [new branch] gh/bobrenjc93/680/head -> origin/gh/bobrenjc93/680/head 2025-12-04T08:27:07.0775290Z * [new branch] gh/bobrenjc93/680/orig -> origin/gh/bobrenjc93/680/orig 2025-12-04T08:27:07.0775449Z * [new branch] gh/bobrenjc93/681/base -> origin/gh/bobrenjc93/681/base 2025-12-04T08:27:07.0775613Z * [new branch] gh/bobrenjc93/681/head -> origin/gh/bobrenjc93/681/head 2025-12-04T08:27:07.0775772Z * [new branch] gh/bobrenjc93/681/orig -> origin/gh/bobrenjc93/681/orig 2025-12-04T08:27:07.0775958Z * [new branch] gh/bobrenjc93/682/base -> origin/gh/bobrenjc93/682/base 2025-12-04T08:27:07.0776131Z * [new branch] gh/bobrenjc93/682/head -> origin/gh/bobrenjc93/682/head 2025-12-04T08:27:07.0776285Z * [new branch] gh/bobrenjc93/682/orig -> origin/gh/bobrenjc93/682/orig 2025-12-04T08:27:07.0777081Z * [new branch] gh/bobrenjc93/683/base -> origin/gh/bobrenjc93/683/base 2025-12-04T08:27:07.0777246Z * [new branch] gh/bobrenjc93/683/head -> origin/gh/bobrenjc93/683/head 2025-12-04T08:27:07.0777408Z * [new branch] gh/bobrenjc93/683/orig -> origin/gh/bobrenjc93/683/orig 2025-12-04T08:27:07.0777582Z * [new branch] gh/bobrenjc93/684/base -> origin/gh/bobrenjc93/684/base 2025-12-04T08:27:07.0777735Z * [new branch] gh/bobrenjc93/684/head -> origin/gh/bobrenjc93/684/head 2025-12-04T08:27:07.0777896Z * [new branch] gh/bobrenjc93/684/orig -> origin/gh/bobrenjc93/684/orig 2025-12-04T08:27:07.0778061Z * [new branch] gh/bobrenjc93/685/base -> origin/gh/bobrenjc93/685/base 2025-12-04T08:27:07.0778238Z * [new branch] gh/bobrenjc93/685/head -> origin/gh/bobrenjc93/685/head 2025-12-04T08:27:07.0785208Z * [new branch] gh/bobrenjc93/685/orig -> origin/gh/bobrenjc93/685/orig 2025-12-04T08:27:07.0785403Z * [new branch] gh/bobrenjc93/686/base -> origin/gh/bobrenjc93/686/base 2025-12-04T08:27:07.0785574Z * [new branch] gh/bobrenjc93/686/head -> origin/gh/bobrenjc93/686/head 2025-12-04T08:27:07.0785733Z * [new branch] gh/bobrenjc93/686/orig -> origin/gh/bobrenjc93/686/orig 2025-12-04T08:27:07.0785894Z * [new branch] gh/bobrenjc93/687/base -> origin/gh/bobrenjc93/687/base 2025-12-04T08:27:07.0786589Z * [new branch] gh/bobrenjc93/687/head -> origin/gh/bobrenjc93/687/head 2025-12-04T08:27:07.0786979Z * [new branch] gh/bobrenjc93/687/orig -> origin/gh/bobrenjc93/687/orig 2025-12-04T08:27:07.0787144Z * [new branch] gh/bobrenjc93/688/base -> origin/gh/bobrenjc93/688/base 2025-12-04T08:27:07.0787289Z * [new branch] gh/bobrenjc93/688/head -> origin/gh/bobrenjc93/688/head 2025-12-04T08:27:07.0787439Z * [new branch] gh/bobrenjc93/688/orig -> origin/gh/bobrenjc93/688/orig 2025-12-04T08:27:07.0787616Z * [new branch] gh/bobrenjc93/689/base -> origin/gh/bobrenjc93/689/base 2025-12-04T08:27:07.0788564Z * [new branch] gh/bobrenjc93/689/head -> origin/gh/bobrenjc93/689/head 2025-12-04T08:27:07.0789048Z * [new branch] gh/bobrenjc93/689/orig -> origin/gh/bobrenjc93/689/orig 2025-12-04T08:27:07.0790138Z * [new branch] gh/bobrenjc93/690/base -> origin/gh/bobrenjc93/690/base 2025-12-04T08:27:07.0790564Z * [new branch] gh/bobrenjc93/690/head -> origin/gh/bobrenjc93/690/head 2025-12-04T08:27:07.0794507Z * [new branch] gh/bobrenjc93/690/orig -> origin/gh/bobrenjc93/690/orig 2025-12-04T08:27:07.0794712Z * [new branch] gh/bobrenjc93/691/base -> origin/gh/bobrenjc93/691/base 2025-12-04T08:27:07.0794873Z * [new branch] gh/bobrenjc93/691/head -> origin/gh/bobrenjc93/691/head 2025-12-04T08:27:07.0795035Z * [new branch] gh/bobrenjc93/691/orig -> origin/gh/bobrenjc93/691/orig 2025-12-04T08:27:07.0796824Z * [new branch] gh/bobrenjc93/692/base -> origin/gh/bobrenjc93/692/base 2025-12-04T08:27:07.0797266Z * [new branch] gh/bobrenjc93/692/head -> origin/gh/bobrenjc93/692/head 2025-12-04T08:27:07.0797847Z * [new branch] gh/bobrenjc93/692/orig -> origin/gh/bobrenjc93/692/orig 2025-12-04T08:27:07.0799051Z * [new branch] gh/bobrenjc93/693/base -> origin/gh/bobrenjc93/693/base 2025-12-04T08:27:07.0799525Z * [new branch] gh/bobrenjc93/693/head -> origin/gh/bobrenjc93/693/head 2025-12-04T08:27:07.0800524Z * [new branch] gh/bobrenjc93/693/orig -> origin/gh/bobrenjc93/693/orig 2025-12-04T08:27:07.0801416Z * [new branch] gh/bobrenjc93/694/base -> origin/gh/bobrenjc93/694/base 2025-12-04T08:27:07.0801932Z * [new branch] gh/bobrenjc93/694/head -> origin/gh/bobrenjc93/694/head 2025-12-04T08:27:07.0802816Z * [new branch] gh/bobrenjc93/694/orig -> origin/gh/bobrenjc93/694/orig 2025-12-04T08:27:07.0803675Z * [new branch] gh/bobrenjc93/695/base -> origin/gh/bobrenjc93/695/base 2025-12-04T08:27:07.0804431Z * [new branch] gh/bobrenjc93/695/head -> origin/gh/bobrenjc93/695/head 2025-12-04T08:27:07.0804951Z * [new branch] gh/bobrenjc93/695/orig -> origin/gh/bobrenjc93/695/orig 2025-12-04T08:27:07.0806416Z * [new branch] gh/c00w/23/base -> origin/gh/c00w/23/base 2025-12-04T08:27:07.0806749Z * [new branch] gh/c00w/23/head -> origin/gh/c00w/23/head 2025-12-04T08:27:07.0808056Z * [new branch] gh/c00w/53/base -> origin/gh/c00w/53/base 2025-12-04T08:27:07.0808413Z * [new branch] gh/c00w/53/head -> origin/gh/c00w/53/head 2025-12-04T08:27:07.0809396Z * [new branch] gh/c00w/53/orig -> origin/gh/c00w/53/orig 2025-12-04T08:27:07.0810232Z * [new branch] gh/c00w/54/base -> origin/gh/c00w/54/base 2025-12-04T08:27:07.0810743Z * [new branch] gh/c00w/54/head -> origin/gh/c00w/54/head 2025-12-04T08:27:07.0811706Z * [new branch] gh/c00w/54/orig -> origin/gh/c00w/54/orig 2025-12-04T08:27:07.0812631Z * [new branch] gh/c00w/56/base -> origin/gh/c00w/56/base 2025-12-04T08:27:07.0814051Z * [new branch] gh/c00w/56/head -> origin/gh/c00w/56/head 2025-12-04T08:27:07.0814408Z * [new branch] gh/c00w/56/orig -> origin/gh/c00w/56/orig 2025-12-04T08:27:07.0814709Z * [new branch] gh/c00w/57/base -> origin/gh/c00w/57/base 2025-12-04T08:27:07.0816169Z * [new branch] gh/c00w/57/head -> origin/gh/c00w/57/head 2025-12-04T08:27:07.0816446Z * [new branch] gh/c00w/57/orig -> origin/gh/c00w/57/orig 2025-12-04T08:27:07.0817930Z * [new branch] gh/c00w/58/base -> origin/gh/c00w/58/base 2025-12-04T08:27:07.0818248Z * [new branch] gh/c00w/58/head -> origin/gh/c00w/58/head 2025-12-04T08:27:07.0818393Z * [new branch] gh/c00w/58/orig -> origin/gh/c00w/58/orig 2025-12-04T08:27:07.0819598Z * [new branch] gh/clee2000/1/base -> origin/gh/clee2000/1/base 2025-12-04T08:27:07.0820115Z * [new branch] gh/clee2000/1/head -> origin/gh/clee2000/1/head 2025-12-04T08:27:07.0821094Z * [new branch] gh/clee2000/1/orig -> origin/gh/clee2000/1/orig 2025-12-04T08:27:07.0825966Z * [new branch] gh/coconutruben/1/base -> origin/gh/coconutruben/1/base 2025-12-04T08:27:07.0826447Z * [new branch] gh/coconutruben/1/head -> origin/gh/coconutruben/1/head 2025-12-04T08:27:07.0828739Z * [new branch] gh/coconutruben/55/base -> origin/gh/coconutruben/55/base 2025-12-04T08:27:07.0828938Z * [new branch] gh/coconutruben/55/head -> origin/gh/coconutruben/55/head 2025-12-04T08:27:07.0829146Z * [new branch] gh/coconutruben/55/orig -> origin/gh/coconutruben/55/orig 2025-12-04T08:27:07.0830510Z * [new branch] gh/coconutruben/57/base -> origin/gh/coconutruben/57/base 2025-12-04T08:27:07.0831067Z * [new branch] gh/coconutruben/57/head -> origin/gh/coconutruben/57/head 2025-12-04T08:27:07.0832009Z * [new branch] gh/coconutruben/57/orig -> origin/gh/coconutruben/57/orig 2025-12-04T08:27:07.0832979Z * [new branch] gh/coconutruben/70/base -> origin/gh/coconutruben/70/base 2025-12-04T08:27:07.0833580Z * [new branch] gh/coconutruben/70/head -> origin/gh/coconutruben/70/head 2025-12-04T08:27:07.0834511Z * [new branch] gh/coconutruben/70/orig -> origin/gh/coconutruben/70/orig 2025-12-04T08:27:07.0835281Z * [new branch] gh/coconutruben/71/base -> origin/gh/coconutruben/71/base 2025-12-04T08:27:07.0836253Z * [new branch] gh/coconutruben/71/head -> origin/gh/coconutruben/71/head 2025-12-04T08:27:07.0836628Z * [new branch] gh/coconutruben/71/orig -> origin/gh/coconutruben/71/orig 2025-12-04T08:27:07.0837838Z * [new branch] gh/coconutruben/72/base -> origin/gh/coconutruben/72/base 2025-12-04T08:27:07.0838612Z * [new branch] gh/coconutruben/72/head -> origin/gh/coconutruben/72/head 2025-12-04T08:27:07.0839233Z * [new branch] gh/coconutruben/72/orig -> origin/gh/coconutruben/72/orig 2025-12-04T08:27:07.0840320Z * [new branch] gh/coconutruben/73/base -> origin/gh/coconutruben/73/base 2025-12-04T08:27:07.0840752Z * [new branch] gh/coconutruben/73/head -> origin/gh/coconutruben/73/head 2025-12-04T08:27:07.0844305Z * [new branch] gh/coconutruben/73/orig -> origin/gh/coconutruben/73/orig 2025-12-04T08:27:07.0844493Z * [new branch] gh/coconutruben/74/base -> origin/gh/coconutruben/74/base 2025-12-04T08:27:07.0844652Z * [new branch] gh/coconutruben/74/head -> origin/gh/coconutruben/74/head 2025-12-04T08:27:07.0844807Z * [new branch] gh/coconutruben/74/orig -> origin/gh/coconutruben/74/orig 2025-12-04T08:27:07.0845002Z * [new branch] gh/coconutruben/79/base -> origin/gh/coconutruben/79/base 2025-12-04T08:27:07.0846059Z * [new branch] gh/coconutruben/79/head -> origin/gh/coconutruben/79/head 2025-12-04T08:27:07.0847627Z * [new branch] gh/coconutruben/79/orig -> origin/gh/coconutruben/79/orig 2025-12-04T08:27:07.0847792Z * [new branch] gh/coconutruben/80/base -> origin/gh/coconutruben/80/base 2025-12-04T08:27:07.0847942Z * [new branch] gh/coconutruben/80/head -> origin/gh/coconutruben/80/head 2025-12-04T08:27:07.0852390Z * [new branch] gh/coconutruben/80/orig -> origin/gh/coconutruben/80/orig 2025-12-04T08:27:07.0852549Z * [new branch] gh/coconutruben/82/base -> origin/gh/coconutruben/82/base 2025-12-04T08:27:07.0852692Z * [new branch] gh/coconutruben/82/head -> origin/gh/coconutruben/82/head 2025-12-04T08:27:07.0852841Z * [new branch] gh/coconutruben/82/orig -> origin/gh/coconutruben/82/orig 2025-12-04T08:27:07.0852983Z * [new branch] gh/coconutruben/83/base -> origin/gh/coconutruben/83/base 2025-12-04T08:27:07.0857211Z * [new branch] gh/coconutruben/83/head -> origin/gh/coconutruben/83/head 2025-12-04T08:27:07.0857375Z * [new branch] gh/coconutruben/83/orig -> origin/gh/coconutruben/83/orig 2025-12-04T08:27:07.0857518Z * [new branch] gh/coconutruben/84/base -> origin/gh/coconutruben/84/base 2025-12-04T08:27:07.0857697Z * [new branch] gh/coconutruben/84/head -> origin/gh/coconutruben/84/head 2025-12-04T08:27:07.0857856Z * [new branch] gh/coconutruben/84/orig -> origin/gh/coconutruben/84/orig 2025-12-04T08:27:07.0857999Z * [new branch] gh/coconutruben/85/base -> origin/gh/coconutruben/85/base 2025-12-04T08:27:07.0858149Z * [new branch] gh/coconutruben/85/head -> origin/gh/coconutruben/85/head 2025-12-04T08:27:07.0858287Z * [new branch] gh/coconutruben/85/orig -> origin/gh/coconutruben/85/orig 2025-12-04T08:27:07.0858438Z * [new branch] gh/coconutruben/86/base -> origin/gh/coconutruben/86/base 2025-12-04T08:27:07.0861934Z * [new branch] gh/coconutruben/86/head -> origin/gh/coconutruben/86/head 2025-12-04T08:27:07.0862082Z * [new branch] gh/coconutruben/86/orig -> origin/gh/coconutruben/86/orig 2025-12-04T08:27:07.0862246Z * [new branch] gh/colinchan15/1/base -> origin/gh/colinchan15/1/base 2025-12-04T08:27:07.0862446Z * [new branch] gh/colinchan15/1/head -> origin/gh/colinchan15/1/head 2025-12-04T08:27:07.0862592Z * [new branch] gh/colinchan15/2/base -> origin/gh/colinchan15/2/base 2025-12-04T08:27:07.0862730Z * [new branch] gh/colinchan15/2/head -> origin/gh/colinchan15/2/head 2025-12-04T08:27:07.0862864Z * [new branch] gh/colinchan15/3/base -> origin/gh/colinchan15/3/base 2025-12-04T08:27:07.0866867Z * [new branch] gh/colinchan15/3/head -> origin/gh/colinchan15/3/head 2025-12-04T08:27:07.0867010Z * [new branch] gh/colinchan15/6/base -> origin/gh/colinchan15/6/base 2025-12-04T08:27:07.0867163Z * [new branch] gh/colinchan15/6/head -> origin/gh/colinchan15/6/head 2025-12-04T08:27:07.0867298Z * [new branch] gh/d4l3k/1/base -> origin/gh/d4l3k/1/base 2025-12-04T08:27:07.0867436Z * [new branch] gh/d4l3k/1/head -> origin/gh/d4l3k/1/head 2025-12-04T08:27:07.0867557Z * [new branch] gh/d4l3k/2/base -> origin/gh/d4l3k/2/base 2025-12-04T08:27:07.0867701Z * [new branch] gh/d4l3k/2/head -> origin/gh/d4l3k/2/head 2025-12-04T08:27:07.0871135Z * [new branch] gh/d4l3k/2/orig -> origin/gh/d4l3k/2/orig 2025-12-04T08:27:07.0871430Z * [new branch] gh/d4l3k/3/base -> origin/gh/d4l3k/3/base 2025-12-04T08:27:07.0871580Z * [new branch] gh/d4l3k/3/head -> origin/gh/d4l3k/3/head 2025-12-04T08:27:07.0871721Z * [new branch] gh/d4l3k/3/orig -> origin/gh/d4l3k/3/orig 2025-12-04T08:27:07.0871915Z * [new branch] gh/d4l3k/4/base -> origin/gh/d4l3k/4/base 2025-12-04T08:27:07.0872062Z * [new branch] gh/d4l3k/4/head -> origin/gh/d4l3k/4/head 2025-12-04T08:27:07.0876719Z * [new branch] gh/d4l3k/4/orig -> origin/gh/d4l3k/4/orig 2025-12-04T08:27:07.0877333Z * [new branch] gh/d4l3k/5/base -> origin/gh/d4l3k/5/base 2025-12-04T08:27:07.0877521Z * [new branch] gh/d4l3k/5/orig -> origin/gh/d4l3k/5/orig 2025-12-04T08:27:07.0877705Z * [new branch] gh/davidberard98/392/base -> origin/gh/davidberard98/392/base 2025-12-04T08:27:07.0877898Z * [new branch] gh/davidberard98/392/head -> origin/gh/davidberard98/392/head 2025-12-04T08:27:07.0878158Z * [new branch] gh/davidberard98/392/orig -> origin/gh/davidberard98/392/orig 2025-12-04T08:27:07.0878343Z * [new branch] gh/davidberard98/399/base -> origin/gh/davidberard98/399/base 2025-12-04T08:27:07.0878569Z * [new branch] gh/davidberard98/399/head -> origin/gh/davidberard98/399/head 2025-12-04T08:27:07.0878880Z * [new branch] gh/davidberard98/399/orig -> origin/gh/davidberard98/399/orig 2025-12-04T08:27:07.0883940Z * [new branch] gh/desertfire/605/base -> origin/gh/desertfire/605/base 2025-12-04T08:27:07.0884163Z * [new branch] gh/desertfire/605/head -> origin/gh/desertfire/605/head 2025-12-04T08:27:07.0884328Z * [new branch] gh/desertfire/605/orig -> origin/gh/desertfire/605/orig 2025-12-04T08:27:07.0884499Z * [new branch] gh/desertfire/606/base -> origin/gh/desertfire/606/base 2025-12-04T08:27:07.0884658Z * [new branch] gh/desertfire/606/head -> origin/gh/desertfire/606/head 2025-12-04T08:27:07.0884827Z * [new branch] gh/desertfire/606/orig -> origin/gh/desertfire/606/orig 2025-12-04T08:27:07.0885077Z * [new branch] gh/desertfire/607/base -> origin/gh/desertfire/607/base 2025-12-04T08:27:07.0885533Z * [new branch] gh/desertfire/607/head -> origin/gh/desertfire/607/head 2025-12-04T08:27:07.0885867Z * [new branch] gh/desertfire/607/orig -> origin/gh/desertfire/607/orig 2025-12-04T08:27:07.0887060Z * [new branch] gh/desertfire/608/base -> origin/gh/desertfire/608/base 2025-12-04T08:27:07.0887370Z * [new branch] gh/desertfire/608/head -> origin/gh/desertfire/608/head 2025-12-04T08:27:07.0888519Z * [new branch] gh/desertfire/608/orig -> origin/gh/desertfire/608/orig 2025-12-04T08:27:07.0889350Z * [new branch] gh/desertfire/609/base -> origin/gh/desertfire/609/base 2025-12-04T08:27:07.0889997Z * [new branch] gh/desertfire/609/head -> origin/gh/desertfire/609/head 2025-12-04T08:27:07.0891309Z * [new branch] gh/desertfire/609/orig -> origin/gh/desertfire/609/orig 2025-12-04T08:27:07.0892208Z * [new branch] gh/desertfire/610/base -> origin/gh/desertfire/610/base 2025-12-04T08:27:07.0892728Z * [new branch] gh/desertfire/610/head -> origin/gh/desertfire/610/head 2025-12-04T08:27:07.0893684Z * [new branch] gh/desertfire/610/orig -> origin/gh/desertfire/610/orig 2025-12-04T08:27:07.0894483Z * [new branch] gh/desertfire/611/base -> origin/gh/desertfire/611/base 2025-12-04T08:27:07.0894957Z * [new branch] gh/desertfire/611/head -> origin/gh/desertfire/611/head 2025-12-04T08:27:07.0896015Z * [new branch] gh/desertfire/611/orig -> origin/gh/desertfire/611/orig 2025-12-04T08:27:07.0896979Z * [new branch] gh/desertfire/612/base -> origin/gh/desertfire/612/base 2025-12-04T08:27:07.0897391Z * [new branch] gh/desertfire/612/head -> origin/gh/desertfire/612/head 2025-12-04T08:27:07.0898417Z * [new branch] gh/desertfire/612/orig -> origin/gh/desertfire/612/orig 2025-12-04T08:27:07.0899621Z * [new branch] gh/desertfire/613/base -> origin/gh/desertfire/613/base 2025-12-04T08:27:07.0899925Z * [new branch] gh/desertfire/613/head -> origin/gh/desertfire/613/head 2025-12-04T08:27:07.0900928Z * [new branch] gh/desertfire/613/orig -> origin/gh/desertfire/613/orig 2025-12-04T08:27:07.0901962Z * [new branch] gh/desertfire/614/base -> origin/gh/desertfire/614/base 2025-12-04T08:27:07.0902836Z * [new branch] gh/desertfire/614/head -> origin/gh/desertfire/614/head 2025-12-04T08:27:07.0903311Z * [new branch] gh/desertfire/614/orig -> origin/gh/desertfire/614/orig 2025-12-04T08:27:07.0904470Z * [new branch] gh/desertfire/615/base -> origin/gh/desertfire/615/base 2025-12-04T08:27:07.0905405Z * [new branch] gh/desertfire/615/head -> origin/gh/desertfire/615/head 2025-12-04T08:27:07.0905910Z * [new branch] gh/desertfire/615/orig -> origin/gh/desertfire/615/orig 2025-12-04T08:27:07.0906941Z * [new branch] gh/desertfire/616/base -> origin/gh/desertfire/616/base 2025-12-04T08:27:07.0907512Z * [new branch] gh/desertfire/616/head -> origin/gh/desertfire/616/head 2025-12-04T08:27:07.0908097Z * [new branch] gh/desertfire/616/orig -> origin/gh/desertfire/616/orig 2025-12-04T08:27:07.0909136Z * [new branch] gh/desertfire/617/base -> origin/gh/desertfire/617/base 2025-12-04T08:27:07.0909645Z * [new branch] gh/desertfire/617/head -> origin/gh/desertfire/617/head 2025-12-04T08:27:07.0910309Z * [new branch] gh/desertfire/617/orig -> origin/gh/desertfire/617/orig 2025-12-04T08:27:07.0912340Z * [new branch] gh/dharakk/1/base -> origin/gh/dharakk/1/base 2025-12-04T08:27:07.0912683Z * [new branch] gh/dharakk/1/head -> origin/gh/dharakk/1/head 2025-12-04T08:27:07.0913525Z * [new branch] gh/drisspg/170/base -> origin/gh/drisspg/170/base 2025-12-04T08:27:07.0914940Z * [new branch] gh/drisspg/170/head -> origin/gh/drisspg/170/head 2025-12-04T08:27:07.0915292Z * [new branch] gh/drisspg/170/orig -> origin/gh/drisspg/170/orig 2025-12-04T08:27:07.0915449Z * [new branch] gh/drisspg/182/base -> origin/gh/drisspg/182/base 2025-12-04T08:27:07.0916495Z * [new branch] gh/drisspg/182/head -> origin/gh/drisspg/182/head 2025-12-04T08:27:07.0917062Z * [new branch] gh/drisspg/183/base -> origin/gh/drisspg/183/base 2025-12-04T08:27:07.0917937Z * [new branch] gh/drisspg/183/head -> origin/gh/drisspg/183/head 2025-12-04T08:27:07.0918964Z * [new branch] gh/drisspg/184/base -> origin/gh/drisspg/184/base 2025-12-04T08:27:07.0919216Z * [new branch] gh/drisspg/184/head -> origin/gh/drisspg/184/head 2025-12-04T08:27:07.0920492Z * [new branch] gh/drisspg/185/base -> origin/gh/drisspg/185/base 2025-12-04T08:27:07.0921175Z * [new branch] gh/drisspg/185/head -> origin/gh/drisspg/185/head 2025-12-04T08:27:07.0922288Z * [new branch] gh/drisspg/194/base -> origin/gh/drisspg/194/base 2025-12-04T08:27:07.0922544Z * [new branch] gh/drisspg/194/head -> origin/gh/drisspg/194/head 2025-12-04T08:27:07.0925520Z * [new branch] gh/drisspg/194/orig -> origin/gh/drisspg/194/orig 2025-12-04T08:27:07.0925701Z * [new branch] gh/drisspg/200/base -> origin/gh/drisspg/200/base 2025-12-04T08:27:07.0925859Z * [new branch] gh/drisspg/200/head -> origin/gh/drisspg/200/head 2025-12-04T08:27:07.0926438Z * [new branch] gh/drisspg/200/orig -> origin/gh/drisspg/200/orig 2025-12-04T08:27:07.0930377Z * [new branch] gh/drisspg/218/base -> origin/gh/drisspg/218/base 2025-12-04T08:27:07.0930709Z * [new branch] gh/drisspg/218/head -> origin/gh/drisspg/218/head 2025-12-04T08:27:07.0930858Z * [new branch] gh/drisspg/218/orig -> origin/gh/drisspg/218/orig 2025-12-04T08:27:07.0930994Z * [new branch] gh/drisspg/219/base -> origin/gh/drisspg/219/base 2025-12-04T08:27:07.0931143Z * [new branch] gh/drisspg/219/head -> origin/gh/drisspg/219/head 2025-12-04T08:27:07.0931277Z * [new branch] gh/drisspg/219/orig -> origin/gh/drisspg/219/orig 2025-12-04T08:27:07.0934187Z * [new branch] gh/drisspg/220/base -> origin/gh/drisspg/220/base 2025-12-04T08:27:07.0934486Z * [new branch] gh/drisspg/220/head -> origin/gh/drisspg/220/head 2025-12-04T08:27:07.0934648Z * [new branch] gh/drisspg/220/orig -> origin/gh/drisspg/220/orig 2025-12-04T08:27:07.0934817Z * [new branch] gh/drisspg/221/base -> origin/gh/drisspg/221/base 2025-12-04T08:27:07.0934952Z * [new branch] gh/drisspg/221/head -> origin/gh/drisspg/221/head 2025-12-04T08:27:07.0935097Z * [new branch] gh/drisspg/221/orig -> origin/gh/drisspg/221/orig 2025-12-04T08:27:07.0938038Z * [new branch] gh/drisspg/222/base -> origin/gh/drisspg/222/base 2025-12-04T08:27:07.0938507Z * [new branch] gh/drisspg/222/head -> origin/gh/drisspg/222/head 2025-12-04T08:27:07.0938689Z * [new branch] gh/drisspg/222/orig -> origin/gh/drisspg/222/orig 2025-12-04T08:27:07.0938845Z * [new branch] gh/drisspg/223/base -> origin/gh/drisspg/223/base 2025-12-04T08:27:07.0939148Z * [new branch] gh/drisspg/223/head -> origin/gh/drisspg/223/head 2025-12-04T08:27:07.0939305Z * [new branch] gh/drisspg/223/orig -> origin/gh/drisspg/223/orig 2025-12-04T08:27:07.0943047Z * [new branch] gh/drisspg/224/base -> origin/gh/drisspg/224/base 2025-12-04T08:27:07.0943343Z * [new branch] gh/drisspg/224/head -> origin/gh/drisspg/224/head 2025-12-04T08:27:07.0943503Z * [new branch] gh/drisspg/224/orig -> origin/gh/drisspg/224/orig 2025-12-04T08:27:07.0943892Z * [new branch] gh/drisspg/225/base -> origin/gh/drisspg/225/base 2025-12-04T08:27:07.0944044Z * [new branch] gh/drisspg/225/head -> origin/gh/drisspg/225/head 2025-12-04T08:27:07.0944199Z * [new branch] gh/drisspg/225/orig -> origin/gh/drisspg/225/orig 2025-12-04T08:27:07.0947834Z * [new branch] gh/drisspg/226/base -> origin/gh/drisspg/226/base 2025-12-04T08:27:07.0948000Z * [new branch] gh/drisspg/226/head -> origin/gh/drisspg/226/head 2025-12-04T08:27:07.0948152Z * [new branch] gh/drisspg/226/orig -> origin/gh/drisspg/226/orig 2025-12-04T08:27:07.0948313Z * [new branch] gh/drisspg/227/base -> origin/gh/drisspg/227/base 2025-12-04T08:27:07.0954084Z * [new branch] gh/drisspg/227/head -> origin/gh/drisspg/227/head 2025-12-04T08:27:07.0955968Z * [new branch] gh/drisspg/227/orig -> origin/gh/drisspg/227/orig 2025-12-04T08:27:07.0956244Z * [new branch] gh/drisspg/228/base -> origin/gh/drisspg/228/base 2025-12-04T08:27:07.0956409Z * [new branch] gh/drisspg/228/head -> origin/gh/drisspg/228/head 2025-12-04T08:27:07.0956551Z * [new branch] gh/drisspg/228/orig -> origin/gh/drisspg/228/orig 2025-12-04T08:27:07.0956701Z * [new branch] gh/drisspg/229/base -> origin/gh/drisspg/229/base 2025-12-04T08:27:07.0956842Z * [new branch] gh/drisspg/229/head -> origin/gh/drisspg/229/head 2025-12-04T08:27:07.0956983Z * [new branch] gh/drisspg/229/orig -> origin/gh/drisspg/229/orig 2025-12-04T08:27:07.0957206Z * [new branch] gh/drisspg/230/base -> origin/gh/drisspg/230/base 2025-12-04T08:27:07.0957360Z * [new branch] gh/drisspg/230/head -> origin/gh/drisspg/230/head 2025-12-04T08:27:07.0957515Z * [new branch] gh/drisspg/230/orig -> origin/gh/drisspg/230/orig 2025-12-04T08:27:07.0957668Z * [new branch] gh/dsjohns2/1/base -> origin/gh/dsjohns2/1/base 2025-12-04T08:27:07.0957818Z * [new branch] gh/dsjohns2/1/head -> origin/gh/dsjohns2/1/head 2025-12-04T08:27:07.0958963Z * [new branch] gh/dzmitry-huba/1/base -> origin/gh/dzmitry-huba/1/base 2025-12-04T08:27:07.0959321Z * [new branch] gh/dzmitry-huba/1/head -> origin/gh/dzmitry-huba/1/head 2025-12-04T08:27:07.0965336Z * [new branch] gh/dzmitry-huba/12/base -> origin/gh/dzmitry-huba/12/base 2025-12-04T08:27:07.0965582Z * [new branch] gh/dzmitry-huba/12/head -> origin/gh/dzmitry-huba/12/head 2025-12-04T08:27:07.0965775Z * [new branch] gh/dzmitry-huba/12/orig -> origin/gh/dzmitry-huba/12/orig 2025-12-04T08:27:07.0965927Z * [new branch] gh/dzmitry-huba/13/base -> origin/gh/dzmitry-huba/13/base 2025-12-04T08:27:07.0966109Z * [new branch] gh/dzmitry-huba/13/head -> origin/gh/dzmitry-huba/13/head 2025-12-04T08:27:07.0970315Z * [new branch] gh/dzmitry-huba/13/orig -> origin/gh/dzmitry-huba/13/orig 2025-12-04T08:27:07.0970601Z * [new branch] gh/dzmitry-huba/14/base -> origin/gh/dzmitry-huba/14/base 2025-12-04T08:27:07.0970767Z * [new branch] gh/dzmitry-huba/14/head -> origin/gh/dzmitry-huba/14/head 2025-12-04T08:27:07.0970916Z * [new branch] gh/dzmitry-huba/14/orig -> origin/gh/dzmitry-huba/14/orig 2025-12-04T08:27:07.0971060Z * [new branch] gh/dzmitry-huba/15/base -> origin/gh/dzmitry-huba/15/base 2025-12-04T08:27:07.0977057Z * [new branch] gh/dzmitry-huba/15/head -> origin/gh/dzmitry-huba/15/head 2025-12-04T08:27:07.0980672Z * [new branch] gh/dzmitry-huba/15/orig -> origin/gh/dzmitry-huba/15/orig 2025-12-04T08:27:07.0981039Z * [new branch] gh/dzmitry-huba/16/base -> origin/gh/dzmitry-huba/16/base 2025-12-04T08:27:07.0981222Z * [new branch] gh/dzmitry-huba/16/head -> origin/gh/dzmitry-huba/16/head 2025-12-04T08:27:07.0981385Z * [new branch] gh/dzmitry-huba/16/orig -> origin/gh/dzmitry-huba/16/orig 2025-12-04T08:27:07.0981546Z * [new branch] gh/dzmitry-huba/17/base -> origin/gh/dzmitry-huba/17/base 2025-12-04T08:27:07.0981728Z * [new branch] gh/dzmitry-huba/17/head -> origin/gh/dzmitry-huba/17/head 2025-12-04T08:27:07.0981888Z * [new branch] gh/dzmitry-huba/17/orig -> origin/gh/dzmitry-huba/17/orig 2025-12-04T08:27:07.0982053Z * [new branch] gh/dzmitry-huba/2/base -> origin/gh/dzmitry-huba/2/base 2025-12-04T08:27:07.0982221Z * [new branch] gh/dzmitry-huba/2/head -> origin/gh/dzmitry-huba/2/head 2025-12-04T08:27:07.0982365Z * [new branch] gh/dzmitry-huba/3/base -> origin/gh/dzmitry-huba/3/base 2025-12-04T08:27:07.0982519Z * [new branch] gh/dzmitry-huba/3/head -> origin/gh/dzmitry-huba/3/head 2025-12-04T08:27:07.0982671Z * [new branch] gh/eellison/808/base -> origin/gh/eellison/808/base 2025-12-04T08:27:07.0982830Z * [new branch] gh/eellison/808/head -> origin/gh/eellison/808/head 2025-12-04T08:27:07.0982976Z * [new branch] gh/eellison/808/orig -> origin/gh/eellison/808/orig 2025-12-04T08:27:07.0983120Z * [new branch] gh/eellison/822/base -> origin/gh/eellison/822/base 2025-12-04T08:27:07.0983268Z * [new branch] gh/eellison/822/head -> origin/gh/eellison/822/head 2025-12-04T08:27:07.0983476Z * [new branch] gh/eellison/822/orig -> origin/gh/eellison/822/orig 2025-12-04T08:27:07.0986886Z * [new branch] gh/eellison/823/base -> origin/gh/eellison/823/base 2025-12-04T08:27:07.0987522Z * [new branch] gh/eellison/823/head -> origin/gh/eellison/823/head 2025-12-04T08:27:07.0987895Z * [new branch] gh/eellison/823/orig -> origin/gh/eellison/823/orig 2025-12-04T08:27:07.0988049Z * [new branch] gh/eellison/862/base -> origin/gh/eellison/862/base 2025-12-04T08:27:07.0988202Z * [new branch] gh/eellison/862/head -> origin/gh/eellison/862/head 2025-12-04T08:27:07.0988351Z * [new branch] gh/eellison/862/orig -> origin/gh/eellison/862/orig 2025-12-04T08:27:07.0988505Z * [new branch] gh/eellison/863/base -> origin/gh/eellison/863/base 2025-12-04T08:27:07.0988912Z * [new branch] gh/eellison/863/head -> origin/gh/eellison/863/head 2025-12-04T08:27:07.0989275Z * [new branch] gh/eellison/863/orig -> origin/gh/eellison/863/orig 2025-12-04T08:27:07.0989459Z * [new branch] gh/eellison/864/base -> origin/gh/eellison/864/base 2025-12-04T08:27:07.0989613Z * [new branch] gh/eellison/864/head -> origin/gh/eellison/864/head 2025-12-04T08:27:07.0989793Z * [new branch] gh/eellison/864/orig -> origin/gh/eellison/864/orig 2025-12-04T08:27:07.0989941Z * [new branch] gh/eellison/865/base -> origin/gh/eellison/865/base 2025-12-04T08:27:07.0996637Z * [new branch] gh/eellison/865/head -> origin/gh/eellison/865/head 2025-12-04T08:27:07.0996796Z * [new branch] gh/eellison/865/orig -> origin/gh/eellison/865/orig 2025-12-04T08:27:07.0997060Z * [new branch] gh/eellison/866/base -> origin/gh/eellison/866/base 2025-12-04T08:27:07.0997219Z * [new branch] gh/eellison/866/head -> origin/gh/eellison/866/head 2025-12-04T08:27:07.0997375Z * [new branch] gh/eellison/866/orig -> origin/gh/eellison/866/orig 2025-12-04T08:27:07.0997528Z * [new branch] gh/eellison/867/base -> origin/gh/eellison/867/base 2025-12-04T08:27:07.0997849Z * [new branch] gh/eellison/867/head -> origin/gh/eellison/867/head 2025-12-04T08:27:07.0998063Z * [new branch] gh/eellison/867/orig -> origin/gh/eellison/867/orig 2025-12-04T08:27:07.0998243Z * [new branch] gh/eellison/868/base -> origin/gh/eellison/868/base 2025-12-04T08:27:07.0998390Z * [new branch] gh/eellison/868/head -> origin/gh/eellison/868/head 2025-12-04T08:27:07.0998542Z * [new branch] gh/eellison/868/orig -> origin/gh/eellison/868/orig 2025-12-04T08:27:07.1001783Z * [new branch] gh/eellison/869/base -> origin/gh/eellison/869/base 2025-12-04T08:27:07.1002241Z * [new branch] gh/eellison/869/head -> origin/gh/eellison/869/head 2025-12-04T08:27:07.1002427Z * [new branch] gh/eellison/869/orig -> origin/gh/eellison/869/orig 2025-12-04T08:27:07.1002575Z * [new branch] gh/eellison/870/base -> origin/gh/eellison/870/base 2025-12-04T08:27:07.1002737Z * [new branch] gh/eellison/870/head -> origin/gh/eellison/870/head 2025-12-04T08:27:07.1002878Z * [new branch] gh/eellison/870/orig -> origin/gh/eellison/870/orig 2025-12-04T08:27:07.1006043Z * [new branch] gh/eellison/871/base -> origin/gh/eellison/871/base 2025-12-04T08:27:07.1006318Z * [new branch] gh/eellison/871/head -> origin/gh/eellison/871/head 2025-12-04T08:27:07.1006468Z * [new branch] gh/eellison/871/orig -> origin/gh/eellison/871/orig 2025-12-04T08:27:07.1006616Z * [new branch] gh/eellison/872/base -> origin/gh/eellison/872/base 2025-12-04T08:27:07.1009706Z * [new branch] gh/eellison/872/head -> origin/gh/eellison/872/head 2025-12-04T08:27:07.1009865Z * [new branch] gh/eellison/872/orig -> origin/gh/eellison/872/orig 2025-12-04T08:27:07.1010191Z * [new branch] gh/eellison/873/base -> origin/gh/eellison/873/base 2025-12-04T08:27:07.1010350Z * [new branch] gh/eellison/873/head -> origin/gh/eellison/873/head 2025-12-04T08:27:07.1010500Z * [new branch] gh/eellison/873/orig -> origin/gh/eellison/873/orig 2025-12-04T08:27:07.1014052Z * [new branch] gh/eellison/874/base -> origin/gh/eellison/874/base 2025-12-04T08:27:07.1014201Z * [new branch] gh/eellison/874/head -> origin/gh/eellison/874/head 2025-12-04T08:27:07.1014451Z * [new branch] gh/eellison/874/orig -> origin/gh/eellison/874/orig 2025-12-04T08:27:07.1014594Z * [new branch] gh/eellison/875/base -> origin/gh/eellison/875/base 2025-12-04T08:27:07.1018220Z * [new branch] gh/eellison/875/head -> origin/gh/eellison/875/head 2025-12-04T08:27:07.1018445Z * [new branch] gh/eellison/875/orig -> origin/gh/eellison/875/orig 2025-12-04T08:27:07.1018591Z * [new branch] gh/eellison/876/base -> origin/gh/eellison/876/base 2025-12-04T08:27:07.1018751Z * [new branch] gh/eellison/876/head -> origin/gh/eellison/876/head 2025-12-04T08:27:07.1018893Z * [new branch] gh/eellison/876/orig -> origin/gh/eellison/876/orig 2025-12-04T08:27:07.1019039Z * [new branch] gh/eellison/877/base -> origin/gh/eellison/877/base 2025-12-04T08:27:07.1022254Z * [new branch] gh/eellison/877/head -> origin/gh/eellison/877/head 2025-12-04T08:27:07.1022556Z * [new branch] gh/eellison/877/orig -> origin/gh/eellison/877/orig 2025-12-04T08:27:07.1025430Z * [new branch] gh/eellison/878/base -> origin/gh/eellison/878/base 2025-12-04T08:27:07.1025638Z * [new branch] gh/eellison/878/head -> origin/gh/eellison/878/head 2025-12-04T08:27:07.1025817Z * [new branch] gh/eellison/878/orig -> origin/gh/eellison/878/orig 2025-12-04T08:27:07.1026164Z * [new branch] gh/eellison/879/base -> origin/gh/eellison/879/base 2025-12-04T08:27:07.1026309Z * [new branch] gh/eellison/879/head -> origin/gh/eellison/879/head 2025-12-04T08:27:07.1026461Z * [new branch] gh/eellison/879/orig -> origin/gh/eellison/879/orig 2025-12-04T08:27:07.1026604Z * [new branch] gh/eellison/880/base -> origin/gh/eellison/880/base 2025-12-04T08:27:07.1029268Z * [new branch] gh/eellison/880/head -> origin/gh/eellison/880/head 2025-12-04T08:27:07.1032848Z * [new branch] gh/eellison/880/orig -> origin/gh/eellison/880/orig 2025-12-04T08:27:07.1033112Z * [new branch] gh/eellison/881/base -> origin/gh/eellison/881/base 2025-12-04T08:27:07.1033793Z * [new branch] gh/eellison/881/head -> origin/gh/eellison/881/head 2025-12-04T08:27:07.1033976Z * [new branch] gh/eellison/881/orig -> origin/gh/eellison/881/orig 2025-12-04T08:27:07.1034163Z * [new branch] gh/eellison/882/base -> origin/gh/eellison/882/base 2025-12-04T08:27:07.1034313Z * [new branch] gh/eellison/882/head -> origin/gh/eellison/882/head 2025-12-04T08:27:07.1034461Z * [new branch] gh/eellison/882/orig -> origin/gh/eellison/882/orig 2025-12-04T08:27:07.1035043Z * [new branch] gh/eellison/883/base -> origin/gh/eellison/883/base 2025-12-04T08:27:07.1035203Z * [new branch] gh/eellison/883/head -> origin/gh/eellison/883/head 2025-12-04T08:27:07.1035361Z * [new branch] gh/eellison/883/orig -> origin/gh/eellison/883/orig 2025-12-04T08:27:07.1035717Z * [new branch] gh/eellison/884/base -> origin/gh/eellison/884/base 2025-12-04T08:27:07.1035873Z * [new branch] gh/eellison/884/head -> origin/gh/eellison/884/head 2025-12-04T08:27:07.1036022Z * [new branch] gh/eellison/884/orig -> origin/gh/eellison/884/orig 2025-12-04T08:27:07.1036172Z * [new branch] gh/etaf/147/base -> origin/gh/etaf/147/base 2025-12-04T08:27:07.1036323Z * [new branch] gh/etaf/147/head -> origin/gh/etaf/147/head 2025-12-04T08:27:07.1037511Z * [new branch] gh/etaf/154/base -> origin/gh/etaf/154/base 2025-12-04T08:27:07.1038089Z * [new branch] gh/etaf/154/head -> origin/gh/etaf/154/head 2025-12-04T08:27:07.1038677Z * [new branch] gh/etaf/154/orig -> origin/gh/etaf/154/orig 2025-12-04T08:27:07.1044178Z * [new branch] gh/etaf/156/base -> origin/gh/etaf/156/base 2025-12-04T08:27:07.1046759Z * [new branch] gh/etaf/156/head -> origin/gh/etaf/156/head 2025-12-04T08:27:07.1046978Z * [new branch] gh/etaf/156/orig -> origin/gh/etaf/156/orig 2025-12-04T08:27:07.1047250Z * [new branch] gh/etaf/157/base -> origin/gh/etaf/157/base 2025-12-04T08:27:07.1047405Z * [new branch] gh/etaf/157/head -> origin/gh/etaf/157/head 2025-12-04T08:27:07.1047616Z * [new branch] gh/etaf/157/orig -> origin/gh/etaf/157/orig 2025-12-04T08:27:07.1047767Z * [new branch] gh/etaf/158/base -> origin/gh/etaf/158/base 2025-12-04T08:27:07.1047979Z * [new branch] gh/etaf/158/head -> origin/gh/etaf/158/head 2025-12-04T08:27:07.1048235Z * [new branch] gh/etaf/158/orig -> origin/gh/etaf/158/orig 2025-12-04T08:27:07.1048362Z * [new branch] gh/etaf/159/base -> origin/gh/etaf/159/base 2025-12-04T08:27:07.1048500Z * [new branch] gh/etaf/159/head -> origin/gh/etaf/159/head 2025-12-04T08:27:07.1048975Z * [new branch] gh/etaf/159/orig -> origin/gh/etaf/159/orig 2025-12-04T08:27:07.1057080Z * [new branch] gh/etaf/160/base -> origin/gh/etaf/160/base 2025-12-04T08:27:07.1058863Z * [new branch] gh/etaf/160/head -> origin/gh/etaf/160/head 2025-12-04T08:27:07.1059014Z * [new branch] gh/etaf/160/orig -> origin/gh/etaf/160/orig 2025-12-04T08:27:07.1059151Z * [new branch] gh/etaf/161/base -> origin/gh/etaf/161/base 2025-12-04T08:27:07.1059279Z * [new branch] gh/etaf/161/head -> origin/gh/etaf/161/head 2025-12-04T08:27:07.1059407Z * [new branch] gh/etaf/161/orig -> origin/gh/etaf/161/orig 2025-12-04T08:27:07.1059531Z * [new branch] gh/etaf/166/base -> origin/gh/etaf/166/base 2025-12-04T08:27:07.1059677Z * [new branch] gh/etaf/166/head -> origin/gh/etaf/166/head 2025-12-04T08:27:07.1059817Z * [new branch] gh/etaf/166/orig -> origin/gh/etaf/166/orig 2025-12-04T08:27:07.1059942Z * [new branch] gh/etaf/167/base -> origin/gh/etaf/167/base 2025-12-04T08:27:07.1060077Z * [new branch] gh/etaf/167/head -> origin/gh/etaf/167/head 2025-12-04T08:27:07.1060213Z * [new branch] gh/etaf/167/orig -> origin/gh/etaf/167/orig 2025-12-04T08:27:07.1060349Z * [new branch] gh/etaf/168/base -> origin/gh/etaf/168/base 2025-12-04T08:27:07.1060488Z * [new branch] gh/etaf/168/head -> origin/gh/etaf/168/head 2025-12-04T08:27:07.1060646Z * [new branch] gh/etaf/168/orig -> origin/gh/etaf/168/orig 2025-12-04T08:27:07.1063173Z * [new branch] gh/etaf/172/base -> origin/gh/etaf/172/base 2025-12-04T08:27:07.1063450Z * [new branch] gh/etaf/172/head -> origin/gh/etaf/172/head 2025-12-04T08:27:07.1063658Z * [new branch] gh/etaf/172/orig -> origin/gh/etaf/172/orig 2025-12-04T08:27:07.1063795Z * [new branch] gh/etaf/173/base -> origin/gh/etaf/173/base 2025-12-04T08:27:07.1063990Z * [new branch] gh/etaf/173/head -> origin/gh/etaf/173/head 2025-12-04T08:27:07.1069759Z * [new branch] gh/etaf/173/orig -> origin/gh/etaf/173/orig 2025-12-04T08:27:07.1072472Z * [new branch] gh/etaf/174/base -> origin/gh/etaf/174/base 2025-12-04T08:27:07.1072635Z * [new branch] gh/etaf/174/head -> origin/gh/etaf/174/head 2025-12-04T08:27:07.1072775Z * [new branch] gh/etaf/175/base -> origin/gh/etaf/175/base 2025-12-04T08:27:07.1072928Z * [new branch] gh/etaf/175/head -> origin/gh/etaf/175/head 2025-12-04T08:27:07.1073061Z * [new branch] gh/etaf/175/orig -> origin/gh/etaf/175/orig 2025-12-04T08:27:07.1073231Z * [new branch] gh/etaf/176/base -> origin/gh/etaf/176/base 2025-12-04T08:27:07.1073367Z * [new branch] gh/etaf/176/head -> origin/gh/etaf/176/head 2025-12-04T08:27:07.1073521Z * [new branch] gh/etaf/176/orig -> origin/gh/etaf/176/orig 2025-12-04T08:27:07.1073659Z * [new branch] gh/etaf/177/base -> origin/gh/etaf/177/base 2025-12-04T08:27:07.1073786Z * [new branch] gh/etaf/177/head -> origin/gh/etaf/177/head 2025-12-04T08:27:07.1073921Z * [new branch] gh/etaf/177/orig -> origin/gh/etaf/177/orig 2025-12-04T08:27:07.1074053Z * [new branch] gh/etaf/178/base -> origin/gh/etaf/178/base 2025-12-04T08:27:07.1074182Z * [new branch] gh/etaf/178/head -> origin/gh/etaf/178/head 2025-12-04T08:27:07.1074319Z * [new branch] gh/etaf/178/orig -> origin/gh/etaf/178/orig 2025-12-04T08:27:07.1076744Z * [new branch] gh/etaf/179/base -> origin/gh/etaf/179/base 2025-12-04T08:27:07.1077010Z * [new branch] gh/etaf/179/head -> origin/gh/etaf/179/head 2025-12-04T08:27:07.1077316Z * [new branch] gh/etaf/179/orig -> origin/gh/etaf/179/orig 2025-12-04T08:27:07.1077458Z * [new branch] gh/etaf/180/base -> origin/gh/etaf/180/base 2025-12-04T08:27:07.1077600Z * [new branch] gh/etaf/180/head -> origin/gh/etaf/180/head 2025-12-04T08:27:07.1077729Z * [new branch] gh/etaf/180/orig -> origin/gh/etaf/180/orig 2025-12-04T08:27:07.1078612Z * [new branch] gh/exclamaforte/1/base -> origin/gh/exclamaforte/1/base 2025-12-04T08:27:07.1088888Z * [new branch] gh/exclamaforte/1/head -> origin/gh/exclamaforte/1/head 2025-12-04T08:27:07.1090982Z * [new branch] gh/exclamaforte/2/base -> origin/gh/exclamaforte/2/base 2025-12-04T08:27:07.1091175Z * [new branch] gh/exclamaforte/2/head -> origin/gh/exclamaforte/2/head 2025-12-04T08:27:07.1091363Z * [new branch] gh/exclamaforte/3/base -> origin/gh/exclamaforte/3/base 2025-12-04T08:27:07.1091523Z * [new branch] gh/exclamaforte/3/head -> origin/gh/exclamaforte/3/head 2025-12-04T08:27:07.1091691Z * [new branch] gh/exclamaforte/4/base -> origin/gh/exclamaforte/4/base 2025-12-04T08:27:07.1091856Z * [new branch] gh/exclamaforte/4/head -> origin/gh/exclamaforte/4/head 2025-12-04T08:27:07.1092011Z * [new branch] gh/ezyang/2374/base -> origin/gh/ezyang/2374/base 2025-12-04T08:27:07.1092167Z * [new branch] gh/ezyang/2374/head -> origin/gh/ezyang/2374/head 2025-12-04T08:27:07.1092311Z * [new branch] gh/ezyang/2374/orig -> origin/gh/ezyang/2374/orig 2025-12-04T08:27:07.1092596Z * [new branch] gh/ezyang/2973/base -> origin/gh/ezyang/2973/base 2025-12-04T08:27:07.1092751Z * [new branch] gh/ezyang/2973/head -> origin/gh/ezyang/2973/head 2025-12-04T08:27:07.1092893Z * [new branch] gh/ezyang/2973/orig -> origin/gh/ezyang/2973/orig 2025-12-04T08:27:07.1093048Z * [new branch] gh/ezyang/2974/base -> origin/gh/ezyang/2974/base 2025-12-04T08:27:07.1093190Z * [new branch] gh/ezyang/2974/head -> origin/gh/ezyang/2974/head 2025-12-04T08:27:07.1093328Z * [new branch] gh/ezyang/2974/orig -> origin/gh/ezyang/2974/orig 2025-12-04T08:27:07.1096627Z * [new branch] gh/ezyang/3131/base -> origin/gh/ezyang/3131/base 2025-12-04T08:27:07.1096775Z * [new branch] gh/ezyang/3131/head -> origin/gh/ezyang/3131/head 2025-12-04T08:27:07.1097178Z * [new branch] gh/ezyang/3131/orig -> origin/gh/ezyang/3131/orig 2025-12-04T08:27:07.1097356Z * [new branch] gh/ezyang/3139/base -> origin/gh/ezyang/3139/base 2025-12-04T08:27:07.1097497Z * [new branch] gh/ezyang/3139/head -> origin/gh/ezyang/3139/head 2025-12-04T08:27:07.1097646Z * [new branch] gh/ezyang/3139/orig -> origin/gh/ezyang/3139/orig 2025-12-04T08:27:07.1101986Z * [new branch] gh/ezyang/3140/base -> origin/gh/ezyang/3140/base 2025-12-04T08:27:07.1102138Z * [new branch] gh/ezyang/3140/head -> origin/gh/ezyang/3140/head 2025-12-04T08:27:07.1102270Z * [new branch] gh/ezyang/3140/orig -> origin/gh/ezyang/3140/orig 2025-12-04T08:27:07.1102402Z * [new branch] gh/ezyang/3143/base -> origin/gh/ezyang/3143/base 2025-12-04T08:27:07.1102540Z * [new branch] gh/ezyang/3143/head -> origin/gh/ezyang/3143/head 2025-12-04T08:27:07.1102670Z * [new branch] gh/ezyang/3143/orig -> origin/gh/ezyang/3143/orig 2025-12-04T08:27:07.1102813Z * [new branch] gh/ezyang/3144/base -> origin/gh/ezyang/3144/base 2025-12-04T08:27:07.1102944Z * [new branch] gh/ezyang/3144/head -> origin/gh/ezyang/3144/head 2025-12-04T08:27:07.1103206Z * [new branch] gh/ezyang/3144/orig -> origin/gh/ezyang/3144/orig 2025-12-04T08:27:07.1103349Z * [new branch] gh/ezyang/3167/base -> origin/gh/ezyang/3167/base 2025-12-04T08:27:07.1107594Z * [new branch] gh/ezyang/3167/head -> origin/gh/ezyang/3167/head 2025-12-04T08:27:07.1109239Z * [new branch] gh/ezyang/3167/orig -> origin/gh/ezyang/3167/orig 2025-12-04T08:27:07.1109405Z * [new branch] gh/ezyang/3173/base -> origin/gh/ezyang/3173/base 2025-12-04T08:27:07.1109562Z * [new branch] gh/ezyang/3173/head -> origin/gh/ezyang/3173/head 2025-12-04T08:27:07.1109716Z * [new branch] gh/ezyang/3173/orig -> origin/gh/ezyang/3173/orig 2025-12-04T08:27:07.1109895Z * [new branch] gh/ezyang/3175/base -> origin/gh/ezyang/3175/base 2025-12-04T08:27:07.1110042Z * [new branch] gh/ezyang/3175/head -> origin/gh/ezyang/3175/head 2025-12-04T08:27:07.1110207Z * [new branch] gh/ezyang/3175/orig -> origin/gh/ezyang/3175/orig 2025-12-04T08:27:07.1110356Z * [new branch] gh/ezyang/3182/base -> origin/gh/ezyang/3182/base 2025-12-04T08:27:07.1110504Z * [new branch] gh/ezyang/3182/head -> origin/gh/ezyang/3182/head 2025-12-04T08:27:07.1115696Z * [new branch] gh/ezyang/3182/orig -> origin/gh/ezyang/3182/orig 2025-12-04T08:27:07.1116046Z * [new branch] gh/ezyang/3185/base -> origin/gh/ezyang/3185/base 2025-12-04T08:27:07.1116255Z * [new branch] gh/ezyang/3185/head -> origin/gh/ezyang/3185/head 2025-12-04T08:27:07.1116486Z * [new branch] gh/ezyang/3185/orig -> origin/gh/ezyang/3185/orig 2025-12-04T08:27:07.1116907Z * [new branch] gh/ezyang/3189/base -> origin/gh/ezyang/3189/base 2025-12-04T08:27:07.1117211Z * [new branch] gh/ezyang/3189/head -> origin/gh/ezyang/3189/head 2025-12-04T08:27:07.1117885Z * [new branch] gh/ezyang/3189/orig -> origin/gh/ezyang/3189/orig 2025-12-04T08:27:07.1118235Z * [new branch] gh/ezyang/3191/base -> origin/gh/ezyang/3191/base 2025-12-04T08:27:07.1118400Z * [new branch] gh/ezyang/3191/head -> origin/gh/ezyang/3191/head 2025-12-04T08:27:07.1118550Z * [new branch] gh/ezyang/3191/orig -> origin/gh/ezyang/3191/orig 2025-12-04T08:27:07.1124698Z * [new branch] gh/ezyang/3192/base -> origin/gh/ezyang/3192/base 2025-12-04T08:27:07.1129635Z * [new branch] gh/ezyang/3192/head -> origin/gh/ezyang/3192/head 2025-12-04T08:27:07.1131873Z * [new branch] gh/ezyang/3192/orig -> origin/gh/ezyang/3192/orig 2025-12-04T08:27:07.1132063Z * [new branch] gh/ezyang/3193/base -> origin/gh/ezyang/3193/base 2025-12-04T08:27:07.1132199Z * [new branch] gh/ezyang/3193/head -> origin/gh/ezyang/3193/head 2025-12-04T08:27:07.1132349Z * [new branch] gh/ezyang/3193/orig -> origin/gh/ezyang/3193/orig 2025-12-04T08:27:07.1132479Z * [new branch] gh/ezyang/3194/base -> origin/gh/ezyang/3194/base 2025-12-04T08:27:07.1132616Z * [new branch] gh/ezyang/3194/head -> origin/gh/ezyang/3194/head 2025-12-04T08:27:07.1132754Z * [new branch] gh/ezyang/3194/orig -> origin/gh/ezyang/3194/orig 2025-12-04T08:27:07.1132887Z * [new branch] gh/ezyang/3195/base -> origin/gh/ezyang/3195/base 2025-12-04T08:27:07.1133020Z * [new branch] gh/ezyang/3195/head -> origin/gh/ezyang/3195/head 2025-12-04T08:27:07.1133166Z * [new branch] gh/ezyang/3195/orig -> origin/gh/ezyang/3195/orig 2025-12-04T08:27:07.1133297Z * [new branch] gh/ezyang/3196/base -> origin/gh/ezyang/3196/base 2025-12-04T08:27:07.1133437Z * [new branch] gh/ezyang/3196/head -> origin/gh/ezyang/3196/head 2025-12-04T08:27:07.1133765Z * [new branch] gh/ezyang/3196/orig -> origin/gh/ezyang/3196/orig 2025-12-04T08:27:07.1138016Z * [new branch] gh/ezyang/3197/base -> origin/gh/ezyang/3197/base 2025-12-04T08:27:07.1140294Z * [new branch] gh/ezyang/3197/head -> origin/gh/ezyang/3197/head 2025-12-04T08:27:07.1145611Z * [new branch] gh/ezyang/3197/orig -> origin/gh/ezyang/3197/orig 2025-12-04T08:27:07.1149962Z * [new branch] gh/ezyang/3198/base -> origin/gh/ezyang/3198/base 2025-12-04T08:27:07.1152828Z * [new branch] gh/ezyang/3198/head -> origin/gh/ezyang/3198/head 2025-12-04T08:27:07.1156305Z * [new branch] gh/ezyang/3198/orig -> origin/gh/ezyang/3198/orig 2025-12-04T08:27:07.1156509Z * [new branch] gh/ezyang/3199/base -> origin/gh/ezyang/3199/base 2025-12-04T08:27:07.1156674Z * [new branch] gh/ezyang/3199/head -> origin/gh/ezyang/3199/head 2025-12-04T08:27:07.1156818Z * [new branch] gh/ezyang/3199/orig -> origin/gh/ezyang/3199/orig 2025-12-04T08:27:07.1156971Z * [new branch] gh/ezyang/3200/base -> origin/gh/ezyang/3200/base 2025-12-04T08:27:07.1157110Z * [new branch] gh/ezyang/3200/head -> origin/gh/ezyang/3200/head 2025-12-04T08:27:07.1157250Z * [new branch] gh/ezyang/3200/orig -> origin/gh/ezyang/3200/orig 2025-12-04T08:27:07.1157399Z * [new branch] gh/ezyang/3201/base -> origin/gh/ezyang/3201/base 2025-12-04T08:27:07.1157536Z * [new branch] gh/ezyang/3201/head -> origin/gh/ezyang/3201/head 2025-12-04T08:27:07.1157874Z * [new branch] gh/ezyang/3201/orig -> origin/gh/ezyang/3201/orig 2025-12-04T08:27:07.1158183Z * [new branch] gh/ezyang/3202/base -> origin/gh/ezyang/3202/base 2025-12-04T08:27:07.1158338Z * [new branch] gh/ezyang/3202/head -> origin/gh/ezyang/3202/head 2025-12-04T08:27:07.1158485Z * [new branch] gh/ezyang/3202/orig -> origin/gh/ezyang/3202/orig 2025-12-04T08:27:07.1158622Z * [new branch] gh/ezyang/3203/base -> origin/gh/ezyang/3203/base 2025-12-04T08:27:07.1158768Z * [new branch] gh/ezyang/3203/head -> origin/gh/ezyang/3203/head 2025-12-04T08:27:07.1158906Z * [new branch] gh/ezyang/3203/orig -> origin/gh/ezyang/3203/orig 2025-12-04T08:27:07.1159062Z * [new branch] gh/ezyang/3204/base -> origin/gh/ezyang/3204/base 2025-12-04T08:27:07.1159208Z * [new branch] gh/ezyang/3204/head -> origin/gh/ezyang/3204/head 2025-12-04T08:27:07.1159350Z * [new branch] gh/ezyang/3204/orig -> origin/gh/ezyang/3204/orig 2025-12-04T08:27:07.1159495Z * [new branch] gh/ezyang/3205/base -> origin/gh/ezyang/3205/base 2025-12-04T08:27:07.1159637Z * [new branch] gh/ezyang/3205/head -> origin/gh/ezyang/3205/head 2025-12-04T08:27:07.1159776Z * [new branch] gh/ezyang/3205/orig -> origin/gh/ezyang/3205/orig 2025-12-04T08:27:07.1159923Z * [new branch] gh/ezyang/3206/base -> origin/gh/ezyang/3206/base 2025-12-04T08:27:07.1160061Z * [new branch] gh/ezyang/3206/head -> origin/gh/ezyang/3206/head 2025-12-04T08:27:07.1160207Z * [new branch] gh/ezyang/3206/orig -> origin/gh/ezyang/3206/orig 2025-12-04T08:27:07.1160343Z * [new branch] gh/ezyang/3207/base -> origin/gh/ezyang/3207/base 2025-12-04T08:27:07.1160481Z * [new branch] gh/ezyang/3207/head -> origin/gh/ezyang/3207/head 2025-12-04T08:27:07.1160628Z * [new branch] gh/ezyang/3207/orig -> origin/gh/ezyang/3207/orig 2025-12-04T08:27:07.1160764Z * [new branch] gh/ezyang/3208/base -> origin/gh/ezyang/3208/base 2025-12-04T08:27:07.1160970Z * [new branch] gh/ezyang/3208/head -> origin/gh/ezyang/3208/head 2025-12-04T08:27:07.1161117Z * [new branch] gh/ezyang/3208/orig -> origin/gh/ezyang/3208/orig 2025-12-04T08:27:07.1161254Z * [new branch] gh/ezyang/3209/base -> origin/gh/ezyang/3209/base 2025-12-04T08:27:07.1161400Z * [new branch] gh/ezyang/3209/head -> origin/gh/ezyang/3209/head 2025-12-04T08:27:07.1161538Z * [new branch] gh/ezyang/3209/orig -> origin/gh/ezyang/3209/orig 2025-12-04T08:27:07.1161699Z * [new branch] gh/fadara01/3/base -> origin/gh/fadara01/3/base 2025-12-04T08:27:07.1161988Z * [new branch] gh/fadara01/3/head -> origin/gh/fadara01/3/head 2025-12-04T08:27:07.1165874Z * [new branch] gh/fadara01/3/orig -> origin/gh/fadara01/3/orig 2025-12-04T08:27:07.1171415Z * [new branch] gh/fadara01/5/base -> origin/gh/fadara01/5/base 2025-12-04T08:27:07.1174243Z * [new branch] gh/fadara01/5/head -> origin/gh/fadara01/5/head 2025-12-04T08:27:07.1174533Z * [new branch] gh/fadara01/5/orig -> origin/gh/fadara01/5/orig 2025-12-04T08:27:07.1179872Z * [new branch] gh/fadara01/6/base -> origin/gh/fadara01/6/base 2025-12-04T08:27:07.1181733Z * [new branch] gh/fadara01/6/head -> origin/gh/fadara01/6/head 2025-12-04T08:27:07.1181890Z * [new branch] gh/fadara01/6/orig -> origin/gh/fadara01/6/orig 2025-12-04T08:27:07.1182025Z * [new branch] gh/fadara01/7/base -> origin/gh/fadara01/7/base 2025-12-04T08:27:07.1182308Z * [new branch] gh/fadara01/7/head -> origin/gh/fadara01/7/head 2025-12-04T08:27:07.1182449Z * [new branch] gh/fadara01/7/orig -> origin/gh/fadara01/7/orig 2025-12-04T08:27:07.1182591Z * [new branch] gh/fadara01/8/base -> origin/gh/fadara01/8/base 2025-12-04T08:27:07.1182730Z * [new branch] gh/fadara01/8/head -> origin/gh/fadara01/8/head 2025-12-04T08:27:07.1182862Z * [new branch] gh/fadara01/8/orig -> origin/gh/fadara01/8/orig 2025-12-04T08:27:07.1183002Z * [new branch] gh/fadara01/9/base -> origin/gh/fadara01/9/base 2025-12-04T08:27:07.1183131Z * [new branch] gh/fadara01/9/head -> origin/gh/fadara01/9/head 2025-12-04T08:27:07.1183270Z * [new branch] gh/fadara01/9/orig -> origin/gh/fadara01/9/orig 2025-12-04T08:27:07.1183403Z * [new branch] gh/fduwjj/182/base -> origin/gh/fduwjj/182/base 2025-12-04T08:27:07.1183536Z * [new branch] gh/fduwjj/182/head -> origin/gh/fduwjj/182/head 2025-12-04T08:27:07.1183677Z * [new branch] gh/fduwjj/182/orig -> origin/gh/fduwjj/182/orig 2025-12-04T08:27:07.1183817Z * [new branch] gh/fduwjj/211/base -> origin/gh/fduwjj/211/base 2025-12-04T08:27:07.1183956Z * [new branch] gh/fduwjj/211/head -> origin/gh/fduwjj/211/head 2025-12-04T08:27:07.1184087Z * [new branch] gh/fduwjj/211/orig -> origin/gh/fduwjj/211/orig 2025-12-04T08:27:07.1184221Z * [new branch] gh/fduwjj/212/base -> origin/gh/fduwjj/212/base 2025-12-04T08:27:07.1184359Z * [new branch] gh/fduwjj/212/head -> origin/gh/fduwjj/212/head 2025-12-04T08:27:07.1184490Z * [new branch] gh/fduwjj/212/orig -> origin/gh/fduwjj/212/orig 2025-12-04T08:27:07.1184631Z * [new branch] gh/fduwjj/213/base -> origin/gh/fduwjj/213/base 2025-12-04T08:27:07.1184765Z * [new branch] gh/fduwjj/213/head -> origin/gh/fduwjj/213/head 2025-12-04T08:27:07.1184897Z * [new branch] gh/fduwjj/213/orig -> origin/gh/fduwjj/213/orig 2025-12-04T08:27:07.1185077Z * [new branch] gh/fduwjj/226/base -> origin/gh/fduwjj/226/base 2025-12-04T08:27:07.1185275Z * [new branch] gh/fduwjj/226/head -> origin/gh/fduwjj/226/head 2025-12-04T08:27:07.1185514Z * [new branch] gh/fduwjj/226/orig -> origin/gh/fduwjj/226/orig 2025-12-04T08:27:07.1189724Z * [new branch] gh/fduwjj/229/base -> origin/gh/fduwjj/229/base 2025-12-04T08:27:07.1189872Z * [new branch] gh/fduwjj/229/head -> origin/gh/fduwjj/229/head 2025-12-04T08:27:07.1190088Z * [new branch] gh/fduwjj/229/orig -> origin/gh/fduwjj/229/orig 2025-12-04T08:27:07.1190271Z * [new branch] gh/fduwjj/233/base -> origin/gh/fduwjj/233/base 2025-12-04T08:27:07.1190500Z * [new branch] gh/fduwjj/233/head -> origin/gh/fduwjj/233/head 2025-12-04T08:27:07.1190642Z * [new branch] gh/fduwjj/233/orig -> origin/gh/fduwjj/233/orig 2025-12-04T08:27:07.1190857Z * [new branch] gh/fduwjj/234/base -> origin/gh/fduwjj/234/base 2025-12-04T08:27:07.1191605Z * [new branch] gh/fduwjj/234/head -> origin/gh/fduwjj/234/head 2025-12-04T08:27:07.1191806Z * [new branch] gh/fduwjj/234/orig -> origin/gh/fduwjj/234/orig 2025-12-04T08:27:07.1192064Z * [new branch] gh/fduwjj/235/base -> origin/gh/fduwjj/235/base 2025-12-04T08:27:07.1192207Z * [new branch] gh/fduwjj/235/head -> origin/gh/fduwjj/235/head 2025-12-04T08:27:07.1192349Z * [new branch] gh/fduwjj/235/orig -> origin/gh/fduwjj/235/orig 2025-12-04T08:27:07.1192785Z * [new branch] gh/fduwjj/236/base -> origin/gh/fduwjj/236/base 2025-12-04T08:27:07.1193673Z * [new branch] gh/fduwjj/236/head -> origin/gh/fduwjj/236/head 2025-12-04T08:27:07.1194026Z * [new branch] gh/fduwjj/236/orig -> origin/gh/fduwjj/236/orig 2025-12-04T08:27:07.1197076Z * [new branch] gh/fduwjj/237/base -> origin/gh/fduwjj/237/base 2025-12-04T08:27:07.1197260Z * [new branch] gh/fduwjj/237/head -> origin/gh/fduwjj/237/head 2025-12-04T08:27:07.1197399Z * [new branch] gh/fduwjj/237/orig -> origin/gh/fduwjj/237/orig 2025-12-04T08:27:07.1197542Z * [new branch] gh/fduwjj/238/base -> origin/gh/fduwjj/238/base 2025-12-04T08:27:07.1198004Z * [new branch] gh/fduwjj/238/head -> origin/gh/fduwjj/238/head 2025-12-04T08:27:07.1202294Z * [new branch] gh/fduwjj/238/orig -> origin/gh/fduwjj/238/orig 2025-12-04T08:27:07.1202477Z * [new branch] gh/fduwjj/239/base -> origin/gh/fduwjj/239/base 2025-12-04T08:27:07.1202756Z * [new branch] gh/fduwjj/239/head -> origin/gh/fduwjj/239/head 2025-12-04T08:27:07.1202886Z * [new branch] gh/fduwjj/239/orig -> origin/gh/fduwjj/239/orig 2025-12-04T08:27:07.1203022Z * [new branch] gh/fegin/332/base -> origin/gh/fegin/332/base 2025-12-04T08:27:07.1206609Z * [new branch] gh/fegin/332/head -> origin/gh/fegin/332/head 2025-12-04T08:27:07.1206737Z * [new branch] gh/fegin/332/orig -> origin/gh/fegin/332/orig 2025-12-04T08:27:07.1206916Z * [new branch] gh/fegin/333/base -> origin/gh/fegin/333/base 2025-12-04T08:27:07.1207041Z * [new branch] gh/fegin/333/head -> origin/gh/fegin/333/head 2025-12-04T08:27:07.1207172Z * [new branch] gh/fegin/333/orig -> origin/gh/fegin/333/orig 2025-12-04T08:27:07.1207305Z * [new branch] gh/fegin/334/base -> origin/gh/fegin/334/base 2025-12-04T08:27:07.1207436Z * [new branch] gh/fegin/334/head -> origin/gh/fegin/334/head 2025-12-04T08:27:07.1207780Z * [new branch] gh/fegin/334/orig -> origin/gh/fegin/334/orig 2025-12-04T08:27:07.1213811Z * [new branch] gh/fegin/335/base -> origin/gh/fegin/335/base 2025-12-04T08:27:07.1213952Z * [new branch] gh/fegin/335/head -> origin/gh/fegin/335/head 2025-12-04T08:27:07.1214092Z * [new branch] gh/fegin/335/orig -> origin/gh/fegin/335/orig 2025-12-04T08:27:07.1214220Z * [new branch] gh/fffrog/160/base -> origin/gh/fffrog/160/base 2025-12-04T08:27:07.1214352Z * [new branch] gh/fffrog/160/head -> origin/gh/fffrog/160/head 2025-12-04T08:27:07.1214475Z * [new branch] gh/fffrog/177/base -> origin/gh/fffrog/177/base 2025-12-04T08:27:07.1214597Z * [new branch] gh/fffrog/177/head -> origin/gh/fffrog/177/head 2025-12-04T08:27:07.1218051Z * [new branch] gh/fffrog/177/orig -> origin/gh/fffrog/177/orig 2025-12-04T08:27:07.1218348Z * [new branch] gh/fffrog/178/base -> origin/gh/fffrog/178/base 2025-12-04T08:27:07.1218514Z * [new branch] gh/fffrog/178/head -> origin/gh/fffrog/178/head 2025-12-04T08:27:07.1218642Z * [new branch] gh/fffrog/178/orig -> origin/gh/fffrog/178/orig 2025-12-04T08:27:07.1218777Z * [new branch] gh/fffrog/181/base -> origin/gh/fffrog/181/base 2025-12-04T08:27:07.1218908Z * [new branch] gh/fffrog/181/head -> origin/gh/fffrog/181/head 2025-12-04T08:27:07.1219036Z * [new branch] gh/fffrog/181/orig -> origin/gh/fffrog/181/orig 2025-12-04T08:27:07.1222887Z * [new branch] gh/fffrog/183/base -> origin/gh/fffrog/183/base 2025-12-04T08:27:07.1223161Z * [new branch] gh/fffrog/183/head -> origin/gh/fffrog/183/head 2025-12-04T08:27:07.1223566Z * [new branch] gh/fffrog/183/orig -> origin/gh/fffrog/183/orig 2025-12-04T08:27:07.1223771Z * [new branch] gh/fxdawnn/10/base -> origin/gh/fxdawnn/10/base 2025-12-04T08:27:07.1226770Z * [new branch] gh/fxdawnn/10/head -> origin/gh/fxdawnn/10/head 2025-12-04T08:27:07.1227020Z * [new branch] gh/fxdawnn/10/orig -> origin/gh/fxdawnn/10/orig 2025-12-04T08:27:07.1227168Z * [new branch] gh/fxdawnn/11/base -> origin/gh/fxdawnn/11/base 2025-12-04T08:27:07.1227369Z * [new branch] gh/fxdawnn/11/head -> origin/gh/fxdawnn/11/head 2025-12-04T08:27:07.1227576Z * [new branch] gh/fxdawnn/11/orig -> origin/gh/fxdawnn/11/orig 2025-12-04T08:27:07.1227815Z * [new branch] gh/fxdawnn/12/base -> origin/gh/fxdawnn/12/base 2025-12-04T08:27:07.1228023Z * [new branch] gh/fxdawnn/12/head -> origin/gh/fxdawnn/12/head 2025-12-04T08:27:07.1228238Z * [new branch] gh/fxdawnn/12/orig -> origin/gh/fxdawnn/12/orig 2025-12-04T08:27:07.1228843Z * [new branch] gh/fxdawnn/13/base -> origin/gh/fxdawnn/13/base 2025-12-04T08:27:07.1233543Z * [new branch] gh/fxdawnn/13/head -> origin/gh/fxdawnn/13/head 2025-12-04T08:27:07.1235377Z * [new branch] gh/fxdawnn/13/orig -> origin/gh/fxdawnn/13/orig 2025-12-04T08:27:07.1235542Z * [new branch] gh/fxdawnn/14/base -> origin/gh/fxdawnn/14/base 2025-12-04T08:27:07.1235676Z * [new branch] gh/fxdawnn/14/head -> origin/gh/fxdawnn/14/head 2025-12-04T08:27:07.1235808Z * [new branch] gh/fxdawnn/14/orig -> origin/gh/fxdawnn/14/orig 2025-12-04T08:27:07.1235951Z * [new branch] gh/fxdawnn/15/base -> origin/gh/fxdawnn/15/base 2025-12-04T08:27:07.1236124Z * [new branch] gh/fxdawnn/15/head -> origin/gh/fxdawnn/15/head 2025-12-04T08:27:07.1236262Z * [new branch] gh/fxdawnn/15/orig -> origin/gh/fxdawnn/15/orig 2025-12-04T08:27:07.1236406Z * [new branch] gh/fxdawnn/6/base -> origin/gh/fxdawnn/6/base 2025-12-04T08:27:07.1236734Z * [new branch] gh/fxdawnn/6/head -> origin/gh/fxdawnn/6/head 2025-12-04T08:27:07.1236874Z * [new branch] gh/fxdawnn/6/orig -> origin/gh/fxdawnn/6/orig 2025-12-04T08:27:07.1237003Z * [new branch] gh/fxdawnn/7/base -> origin/gh/fxdawnn/7/base 2025-12-04T08:27:07.1237131Z * [new branch] gh/fxdawnn/7/head -> origin/gh/fxdawnn/7/head 2025-12-04T08:27:07.1237266Z * [new branch] gh/fxdawnn/7/orig -> origin/gh/fxdawnn/7/orig 2025-12-04T08:27:07.1237401Z * [new branch] gh/fxdawnn/9/base -> origin/gh/fxdawnn/9/base 2025-12-04T08:27:07.1237539Z * [new branch] gh/fxdawnn/9/head -> origin/gh/fxdawnn/9/head 2025-12-04T08:27:07.1237668Z * [new branch] gh/fxdawnn/9/orig -> origin/gh/fxdawnn/9/orig 2025-12-04T08:27:07.1242118Z * [new branch] gh/galv/1/base -> origin/gh/galv/1/base 2025-12-04T08:27:07.1244379Z * [new branch] gh/galv/1/head -> origin/gh/galv/1/head 2025-12-04T08:27:07.1248598Z * [new branch] gh/galv/1/orig -> origin/gh/galv/1/orig 2025-12-04T08:27:07.1248765Z * [new branch] gh/galv/2/base -> origin/gh/galv/2/base 2025-12-04T08:27:07.1248899Z * [new branch] gh/galv/2/head -> origin/gh/galv/2/head 2025-12-04T08:27:07.1249023Z * [new branch] gh/galv/2/orig -> origin/gh/galv/2/orig 2025-12-04T08:27:07.1249154Z * [new branch] gh/galv/3/base -> origin/gh/galv/3/base 2025-12-04T08:27:07.1249274Z * [new branch] gh/galv/3/head -> origin/gh/galv/3/head 2025-12-04T08:27:07.1249534Z * [new branch] gh/galv/3/orig -> origin/gh/galv/3/orig 2025-12-04T08:27:07.1249699Z * [new branch] gh/guangyey/134/base -> origin/gh/guangyey/134/base 2025-12-04T08:27:07.1249848Z * [new branch] gh/guangyey/134/head -> origin/gh/guangyey/134/head 2025-12-04T08:27:07.1249984Z * [new branch] gh/guangyey/134/orig -> origin/gh/guangyey/134/orig 2025-12-04T08:27:07.1250128Z * [new branch] gh/guangyey/163/base -> origin/gh/guangyey/163/base 2025-12-04T08:27:07.1250263Z * [new branch] gh/guangyey/163/head -> origin/gh/guangyey/163/head 2025-12-04T08:27:07.1250405Z * [new branch] gh/guangyey/163/orig -> origin/gh/guangyey/163/orig 2025-12-04T08:27:07.1251936Z * [new branch] gh/guangyey/168/base -> origin/gh/guangyey/168/base 2025-12-04T08:27:07.1252158Z * [new branch] gh/guangyey/168/head -> origin/gh/guangyey/168/head 2025-12-04T08:27:07.1252368Z * [new branch] gh/guangyey/168/orig -> origin/gh/guangyey/168/orig 2025-12-04T08:27:07.1252515Z * [new branch] gh/guangyey/169/base -> origin/gh/guangyey/169/base 2025-12-04T08:27:07.1252842Z * [new branch] gh/guangyey/169/head -> origin/gh/guangyey/169/head 2025-12-04T08:27:07.1253043Z * [new branch] gh/guangyey/169/orig -> origin/gh/guangyey/169/orig 2025-12-04T08:27:07.1257089Z * [new branch] gh/guangyey/170/base -> origin/gh/guangyey/170/base 2025-12-04T08:27:07.1257348Z * [new branch] gh/guangyey/170/head -> origin/gh/guangyey/170/head 2025-12-04T08:27:07.1257498Z * [new branch] gh/guangyey/170/orig -> origin/gh/guangyey/170/orig 2025-12-04T08:27:07.1257635Z * [new branch] gh/guangyey/171/base -> origin/gh/guangyey/171/base 2025-12-04T08:27:07.1257767Z * [new branch] gh/guangyey/171/head -> origin/gh/guangyey/171/head 2025-12-04T08:27:07.1257967Z * [new branch] gh/guangyey/171/orig -> origin/gh/guangyey/171/orig 2025-12-04T08:27:07.1263126Z * [new branch] gh/guangyey/178/base -> origin/gh/guangyey/178/base 2025-12-04T08:27:07.1267751Z * [new branch] gh/guangyey/178/head -> origin/gh/guangyey/178/head 2025-12-04T08:27:07.1267930Z * [new branch] gh/guangyey/178/orig -> origin/gh/guangyey/178/orig 2025-12-04T08:27:07.1268077Z * [new branch] gh/guangyey/182/base -> origin/gh/guangyey/182/base 2025-12-04T08:27:07.1268216Z * [new branch] gh/guangyey/182/head -> origin/gh/guangyey/182/head 2025-12-04T08:27:07.1268360Z * [new branch] gh/guangyey/182/orig -> origin/gh/guangyey/182/orig 2025-12-04T08:27:07.1268515Z * [new branch] gh/guangyey/183/base -> origin/gh/guangyey/183/base 2025-12-04T08:27:07.1268683Z * [new branch] gh/guangyey/183/head -> origin/gh/guangyey/183/head 2025-12-04T08:27:07.1268822Z * [new branch] gh/guangyey/183/orig -> origin/gh/guangyey/183/orig 2025-12-04T08:27:07.1268960Z * [new branch] gh/guangyey/185/base -> origin/gh/guangyey/185/base 2025-12-04T08:27:07.1269098Z * [new branch] gh/guangyey/185/head -> origin/gh/guangyey/185/head 2025-12-04T08:27:07.1269261Z * [new branch] gh/guangyey/185/orig -> origin/gh/guangyey/185/orig 2025-12-04T08:27:07.1269391Z * [new branch] gh/guangyey/186/base -> origin/gh/guangyey/186/base 2025-12-04T08:27:07.1269526Z * [new branch] gh/guangyey/186/head -> origin/gh/guangyey/186/head 2025-12-04T08:27:07.1269658Z * [new branch] gh/guangyey/186/orig -> origin/gh/guangyey/186/orig 2025-12-04T08:27:07.1269797Z * [new branch] gh/guangyey/187/base -> origin/gh/guangyey/187/base 2025-12-04T08:27:07.1270110Z * [new branch] gh/guangyey/187/head -> origin/gh/guangyey/187/head 2025-12-04T08:27:07.1270249Z * [new branch] gh/guangyey/187/orig -> origin/gh/guangyey/187/orig 2025-12-04T08:27:07.1270455Z * [new branch] gh/guangyey/188/base -> origin/gh/guangyey/188/base 2025-12-04T08:27:07.1271476Z * [new branch] gh/guangyey/188/head -> origin/gh/guangyey/188/head 2025-12-04T08:27:07.1271801Z * [new branch] gh/guangyey/188/orig -> origin/gh/guangyey/188/orig 2025-12-04T08:27:07.1272975Z * [new branch] gh/guangyey/190/base -> origin/gh/guangyey/190/base 2025-12-04T08:27:07.1273294Z * [new branch] gh/guangyey/190/head -> origin/gh/guangyey/190/head 2025-12-04T08:27:07.1274235Z * [new branch] gh/guangyey/190/orig -> origin/gh/guangyey/190/orig 2025-12-04T08:27:07.1275084Z * [new branch] gh/guangyey/208/base -> origin/gh/guangyey/208/base 2025-12-04T08:27:07.1275402Z * [new branch] gh/guangyey/208/head -> origin/gh/guangyey/208/head 2025-12-04T08:27:07.1276977Z * [new branch] gh/guangyey/208/orig -> origin/gh/guangyey/208/orig 2025-12-04T08:27:07.1277281Z * [new branch] gh/guangyey/228/base -> origin/gh/guangyey/228/base 2025-12-04T08:27:07.1277583Z * [new branch] gh/guangyey/228/head -> origin/gh/guangyey/228/head 2025-12-04T08:27:07.1278598Z * [new branch] gh/guangyey/228/orig -> origin/gh/guangyey/228/orig 2025-12-04T08:27:07.1281833Z * [new branch] gh/guangyey/230/base -> origin/gh/guangyey/230/base 2025-12-04T08:27:07.1281979Z * [new branch] gh/guangyey/230/head -> origin/gh/guangyey/230/head 2025-12-04T08:27:07.1282150Z * [new branch] gh/guangyey/230/orig -> origin/gh/guangyey/230/orig 2025-12-04T08:27:07.1282307Z * [new branch] gh/guangyey/231/base -> origin/gh/guangyey/231/base 2025-12-04T08:27:07.1282444Z * [new branch] gh/guangyey/231/head -> origin/gh/guangyey/231/head 2025-12-04T08:27:07.1282902Z * [new branch] gh/guangyey/231/orig -> origin/gh/guangyey/231/orig 2025-12-04T08:27:07.1284496Z * [new branch] gh/guangyey/232/base -> origin/gh/guangyey/232/base 2025-12-04T08:27:07.1284800Z * [new branch] gh/guangyey/232/head -> origin/gh/guangyey/232/head 2025-12-04T08:27:07.1285138Z * [new branch] gh/guangyey/232/orig -> origin/gh/guangyey/232/orig 2025-12-04T08:27:07.1287208Z * [new branch] gh/guangyey/233/base -> origin/gh/guangyey/233/base 2025-12-04T08:27:07.1287392Z * [new branch] gh/guangyey/233/head -> origin/gh/guangyey/233/head 2025-12-04T08:27:07.1287894Z * [new branch] gh/guangyey/233/orig -> origin/gh/guangyey/233/orig 2025-12-04T08:27:07.1288847Z * [new branch] gh/guangyey/234/base -> origin/gh/guangyey/234/base 2025-12-04T08:27:07.1289213Z * [new branch] gh/guangyey/234/head -> origin/gh/guangyey/234/head 2025-12-04T08:27:07.1290183Z * [new branch] gh/guangyey/234/orig -> origin/gh/guangyey/234/orig 2025-12-04T08:27:07.1291087Z * [new branch] gh/guangyey/235/base -> origin/gh/guangyey/235/base 2025-12-04T08:27:07.1292050Z * [new branch] gh/guangyey/235/head -> origin/gh/guangyey/235/head 2025-12-04T08:27:07.1292405Z * [new branch] gh/guangyey/235/orig -> origin/gh/guangyey/235/orig 2025-12-04T08:27:07.1293517Z * [new branch] gh/guangyey/236/base -> origin/gh/guangyey/236/base 2025-12-04T08:27:07.1294003Z * [new branch] gh/guangyey/236/head -> origin/gh/guangyey/236/head 2025-12-04T08:27:07.1294784Z * [new branch] gh/guangyey/236/orig -> origin/gh/guangyey/236/orig 2025-12-04T08:27:07.1296088Z * [new branch] gh/guangyey/237/base -> origin/gh/guangyey/237/base 2025-12-04T08:27:07.1296279Z * [new branch] gh/guangyey/237/head -> origin/gh/guangyey/237/head 2025-12-04T08:27:07.1297286Z * [new branch] gh/guangyey/237/orig -> origin/gh/guangyey/237/orig 2025-12-04T08:27:07.1298194Z * [new branch] gh/guangyey/238/base -> origin/gh/guangyey/238/base 2025-12-04T08:27:07.1299483Z * [new branch] gh/guangyey/238/head -> origin/gh/guangyey/238/head 2025-12-04T08:27:07.1299823Z * [new branch] gh/guangyey/239/base -> origin/gh/guangyey/239/base 2025-12-04T08:27:07.1299979Z * [new branch] gh/guangyey/239/head -> origin/gh/guangyey/239/head 2025-12-04T08:27:07.1300998Z * [new branch] gh/guangyey/239/orig -> origin/gh/guangyey/239/orig 2025-12-04T08:27:07.1301933Z * [new branch] gh/guangyey/240/base -> origin/gh/guangyey/240/base 2025-12-04T08:27:07.1302957Z * [new branch] gh/guangyey/240/head -> origin/gh/guangyey/240/head 2025-12-04T08:27:07.1303325Z * [new branch] gh/guangyey/240/orig -> origin/gh/guangyey/240/orig 2025-12-04T08:27:07.1304562Z * [new branch] gh/guangyey/241/base -> origin/gh/guangyey/241/base 2025-12-04T08:27:07.1305562Z * [new branch] gh/guangyey/241/head -> origin/gh/guangyey/241/head 2025-12-04T08:27:07.1306023Z * [new branch] gh/guangyey/241/orig -> origin/gh/guangyey/241/orig 2025-12-04T08:27:07.1307207Z * [new branch] gh/guangyey/242/base -> origin/gh/guangyey/242/base 2025-12-04T08:27:07.1307771Z * [new branch] gh/guangyey/242/head -> origin/gh/guangyey/242/head 2025-12-04T08:27:07.1308621Z * [new branch] gh/guangyey/242/orig -> origin/gh/guangyey/242/orig 2025-12-04T08:27:07.1310412Z * [new branch] gh/guangyey/243/base -> origin/gh/guangyey/243/base 2025-12-04T08:27:07.1310690Z * [new branch] gh/guangyey/243/head -> origin/gh/guangyey/243/head 2025-12-04T08:27:07.1310839Z * [new branch] gh/guangyey/243/orig -> origin/gh/guangyey/243/orig 2025-12-04T08:27:07.1313537Z * [new branch] gh/guangyey/244/base -> origin/gh/guangyey/244/base 2025-12-04T08:27:07.1313881Z * [new branch] gh/guangyey/244/head -> origin/gh/guangyey/244/head 2025-12-04T08:27:07.1314170Z * [new branch] gh/guangyey/244/orig -> origin/gh/guangyey/244/orig 2025-12-04T08:27:07.1314345Z * [new branch] gh/guangyey/245/base -> origin/gh/guangyey/245/base 2025-12-04T08:27:07.1314632Z * [new branch] gh/guangyey/245/head -> origin/gh/guangyey/245/head 2025-12-04T08:27:07.1316342Z * [new branch] gh/guangyey/245/orig -> origin/gh/guangyey/245/orig 2025-12-04T08:27:07.1316997Z * [new branch] gh/guangyey/246/base -> origin/gh/guangyey/246/base 2025-12-04T08:27:07.1317185Z * [new branch] gh/guangyey/246/head -> origin/gh/guangyey/246/head 2025-12-04T08:27:07.1317634Z * [new branch] gh/guangyey/246/orig -> origin/gh/guangyey/246/orig 2025-12-04T08:27:07.1318841Z * [new branch] gh/guangyey/247/base -> origin/gh/guangyey/247/base 2025-12-04T08:27:07.1322636Z * [new branch] gh/guangyey/247/head -> origin/gh/guangyey/247/head 2025-12-04T08:27:07.1322814Z * [new branch] gh/guangyey/247/orig -> origin/gh/guangyey/247/orig 2025-12-04T08:27:07.1323095Z * [new branch] gh/guangyey/248/base -> origin/gh/guangyey/248/base 2025-12-04T08:27:07.1329741Z * [new branch] gh/guangyey/248/head -> origin/gh/guangyey/248/head 2025-12-04T08:27:07.1329936Z * [new branch] gh/guangyey/248/orig -> origin/gh/guangyey/248/orig 2025-12-04T08:27:07.1330312Z * [new branch] gh/guangyey/249/base -> origin/gh/guangyey/249/base 2025-12-04T08:27:07.1330472Z * [new branch] gh/guangyey/249/head -> origin/gh/guangyey/249/head 2025-12-04T08:27:07.1330651Z * [new branch] gh/guangyey/249/orig -> origin/gh/guangyey/249/orig 2025-12-04T08:27:07.1330811Z * [new branch] gh/guangyey/250/base -> origin/gh/guangyey/250/base 2025-12-04T08:27:07.1330961Z * [new branch] gh/guangyey/250/head -> origin/gh/guangyey/250/head 2025-12-04T08:27:07.1331114Z * [new branch] gh/guangyey/250/orig -> origin/gh/guangyey/250/orig 2025-12-04T08:27:07.1331268Z * [new branch] gh/guangyey/251/base -> origin/gh/guangyey/251/base 2025-12-04T08:27:07.1332282Z * [new branch] gh/guangyey/251/head -> origin/gh/guangyey/251/head 2025-12-04T08:27:07.1332462Z * [new branch] gh/guangyey/251/orig -> origin/gh/guangyey/251/orig 2025-12-04T08:27:07.1333419Z * [new branch] gh/guangyey/252/base -> origin/gh/guangyey/252/base 2025-12-04T08:27:07.1333851Z * [new branch] gh/guangyey/252/head -> origin/gh/guangyey/252/head 2025-12-04T08:27:07.1334793Z * [new branch] gh/guangyey/252/orig -> origin/gh/guangyey/252/orig 2025-12-04T08:27:07.1335769Z * [new branch] gh/guangyey/253/base -> origin/gh/guangyey/253/base 2025-12-04T08:27:07.1336188Z * [new branch] gh/guangyey/253/head -> origin/gh/guangyey/253/head 2025-12-04T08:27:07.1337198Z * [new branch] gh/guangyey/253/orig -> origin/gh/guangyey/253/orig 2025-12-04T08:27:07.1338069Z * [new branch] gh/guangyey/254/base -> origin/gh/guangyey/254/base 2025-12-04T08:27:07.1338602Z * [new branch] gh/guangyey/254/head -> origin/gh/guangyey/254/head 2025-12-04T08:27:07.1339889Z * [new branch] gh/guangyey/254/orig -> origin/gh/guangyey/254/orig 2025-12-04T08:27:07.1340539Z * [new branch] gh/guangyey/255/base -> origin/gh/guangyey/255/base 2025-12-04T08:27:07.1340946Z * [new branch] gh/guangyey/255/head -> origin/gh/guangyey/255/head 2025-12-04T08:27:07.1341636Z * [new branch] gh/guangyey/255/orig -> origin/gh/guangyey/255/orig 2025-12-04T08:27:07.1343139Z * [new branch] gh/guilhermeleobas/107/base -> origin/gh/guilhermeleobas/107/base 2025-12-04T08:27:07.1343601Z * [new branch] gh/guilhermeleobas/107/head -> origin/gh/guilhermeleobas/107/head 2025-12-04T08:27:07.1344523Z * [new branch] gh/guilhermeleobas/107/orig -> origin/gh/guilhermeleobas/107/orig 2025-12-04T08:27:07.1345324Z * [new branch] gh/guilhermeleobas/108/base -> origin/gh/guilhermeleobas/108/base 2025-12-04T08:27:07.1345937Z * [new branch] gh/guilhermeleobas/108/head -> origin/gh/guilhermeleobas/108/head 2025-12-04T08:27:07.1346839Z * [new branch] gh/guilhermeleobas/108/orig -> origin/gh/guilhermeleobas/108/orig 2025-12-04T08:27:07.1347905Z * [new branch] gh/guilhermeleobas/150/base -> origin/gh/guilhermeleobas/150/base 2025-12-04T08:27:07.1348570Z * [new branch] gh/guilhermeleobas/150/head -> origin/gh/guilhermeleobas/150/head 2025-12-04T08:27:07.1348988Z * [new branch] gh/guilhermeleobas/150/orig -> origin/gh/guilhermeleobas/150/orig 2025-12-04T08:27:07.1351266Z * [new branch] gh/guilhermeleobas/168/base -> origin/gh/guilhermeleobas/168/base 2025-12-04T08:27:07.1351836Z * [new branch] gh/guilhermeleobas/168/head -> origin/gh/guilhermeleobas/168/head 2025-12-04T08:27:07.1354757Z * [new branch] gh/guilhermeleobas/168/orig -> origin/gh/guilhermeleobas/168/orig 2025-12-04T08:27:07.1354933Z * [new branch] gh/guilhermeleobas/169/base -> origin/gh/guilhermeleobas/169/base 2025-12-04T08:27:07.1355169Z * [new branch] gh/guilhermeleobas/169/head -> origin/gh/guilhermeleobas/169/head 2025-12-04T08:27:07.1355340Z * [new branch] gh/guilhermeleobas/169/orig -> origin/gh/guilhermeleobas/169/orig 2025-12-04T08:27:07.1355530Z * [new branch] gh/guilhermeleobas/170/base -> origin/gh/guilhermeleobas/170/base 2025-12-04T08:27:07.1356655Z * [new branch] gh/guilhermeleobas/170/head -> origin/gh/guilhermeleobas/170/head 2025-12-04T08:27:07.1357583Z * [new branch] gh/guilhermeleobas/170/orig -> origin/gh/guilhermeleobas/170/orig 2025-12-04T08:27:07.1359051Z * [new branch] gh/guilhermeleobas/171/base -> origin/gh/guilhermeleobas/171/base 2025-12-04T08:27:07.1365493Z * [new branch] gh/guilhermeleobas/171/head -> origin/gh/guilhermeleobas/171/head 2025-12-04T08:27:07.1365869Z * [new branch] gh/guilhermeleobas/171/orig -> origin/gh/guilhermeleobas/171/orig 2025-12-04T08:27:07.1366144Z * [new branch] gh/guilhermeleobas/173/base -> origin/gh/guilhermeleobas/173/base 2025-12-04T08:27:07.1366348Z * [new branch] gh/guilhermeleobas/173/head -> origin/gh/guilhermeleobas/173/head 2025-12-04T08:27:07.1366623Z * [new branch] gh/guilhermeleobas/173/orig -> origin/gh/guilhermeleobas/173/orig 2025-12-04T08:27:07.1366803Z * [new branch] gh/guilhermeleobas/193/base -> origin/gh/guilhermeleobas/193/base 2025-12-04T08:27:07.1366968Z * [new branch] gh/guilhermeleobas/193/head -> origin/gh/guilhermeleobas/193/head 2025-12-04T08:27:07.1367130Z * [new branch] gh/guilhermeleobas/193/orig -> origin/gh/guilhermeleobas/193/orig 2025-12-04T08:27:07.1369159Z * [new branch] gh/guilhermeleobas/204/base -> origin/gh/guilhermeleobas/204/base 2025-12-04T08:27:07.1369540Z * [new branch] gh/guilhermeleobas/204/head -> origin/gh/guilhermeleobas/204/head 2025-12-04T08:27:07.1369836Z * [new branch] gh/guilhermeleobas/204/orig -> origin/gh/guilhermeleobas/204/orig 2025-12-04T08:27:07.1370086Z * [new branch] gh/guilhermeleobas/211/base -> origin/gh/guilhermeleobas/211/base 2025-12-04T08:27:07.1370596Z * [new branch] gh/guilhermeleobas/211/head -> origin/gh/guilhermeleobas/211/head 2025-12-04T08:27:07.1371022Z * [new branch] gh/guilhermeleobas/211/orig -> origin/gh/guilhermeleobas/211/orig 2025-12-04T08:27:07.1371339Z * [new branch] gh/guilhermeleobas/226/base -> origin/gh/guilhermeleobas/226/base 2025-12-04T08:27:07.1371661Z * [new branch] gh/guilhermeleobas/226/head -> origin/gh/guilhermeleobas/226/head 2025-12-04T08:27:07.1371966Z * [new branch] gh/guilhermeleobas/226/orig -> origin/gh/guilhermeleobas/226/orig 2025-12-04T08:27:07.1372509Z * [new branch] gh/guilhermeleobas/236/base -> origin/gh/guilhermeleobas/236/base 2025-12-04T08:27:07.1372701Z * [new branch] gh/guilhermeleobas/236/head -> origin/gh/guilhermeleobas/236/head 2025-12-04T08:27:07.1372889Z * [new branch] gh/guilhermeleobas/236/orig -> origin/gh/guilhermeleobas/236/orig 2025-12-04T08:27:07.1374176Z * [new branch] gh/guilhermeleobas/247/base -> origin/gh/guilhermeleobas/247/base 2025-12-04T08:27:07.1374395Z * [new branch] gh/guilhermeleobas/247/head -> origin/gh/guilhermeleobas/247/head 2025-12-04T08:27:07.1374983Z * [new branch] gh/guilhermeleobas/247/orig -> origin/gh/guilhermeleobas/247/orig 2025-12-04T08:27:07.1378678Z * [new branch] gh/guilhermeleobas/248/base -> origin/gh/guilhermeleobas/248/base 2025-12-04T08:27:07.1379080Z * [new branch] gh/guilhermeleobas/248/head -> origin/gh/guilhermeleobas/248/head 2025-12-04T08:27:07.1379404Z * [new branch] gh/guilhermeleobas/248/orig -> origin/gh/guilhermeleobas/248/orig 2025-12-04T08:27:07.1379671Z * [new branch] gh/guilhermeleobas/250/base -> origin/gh/guilhermeleobas/250/base 2025-12-04T08:27:07.1380130Z * [new branch] gh/guilhermeleobas/250/head -> origin/gh/guilhermeleobas/250/head 2025-12-04T08:27:07.1380330Z * [new branch] gh/guilhermeleobas/250/orig -> origin/gh/guilhermeleobas/250/orig 2025-12-04T08:27:07.1380910Z * [new branch] gh/guilhermeleobas/253/base -> origin/gh/guilhermeleobas/253/base 2025-12-04T08:27:07.1381826Z * [new branch] gh/guilhermeleobas/253/head -> origin/gh/guilhermeleobas/253/head 2025-12-04T08:27:07.1382329Z * [new branch] gh/guilhermeleobas/253/orig -> origin/gh/guilhermeleobas/253/orig 2025-12-04T08:27:07.1384577Z * [new branch] gh/guilhermeleobas/254/base -> origin/gh/guilhermeleobas/254/base 2025-12-04T08:27:07.1384967Z * [new branch] gh/guilhermeleobas/254/head -> origin/gh/guilhermeleobas/254/head 2025-12-04T08:27:07.1385288Z * [new branch] gh/guilhermeleobas/254/orig -> origin/gh/guilhermeleobas/254/orig 2025-12-04T08:27:07.1385783Z * [new branch] gh/guilhermeleobas/255/base -> origin/gh/guilhermeleobas/255/base 2025-12-04T08:27:07.1386702Z * [new branch] gh/guilhermeleobas/255/head -> origin/gh/guilhermeleobas/255/head 2025-12-04T08:27:07.1387151Z * [new branch] gh/guilhermeleobas/255/orig -> origin/gh/guilhermeleobas/255/orig 2025-12-04T08:27:07.1389853Z * [new branch] gh/guilhermeleobas/256/base -> origin/gh/guilhermeleobas/256/base 2025-12-04T08:27:07.1390078Z * [new branch] gh/guilhermeleobas/256/head -> origin/gh/guilhermeleobas/256/head 2025-12-04T08:27:07.1390276Z * [new branch] gh/guilhermeleobas/256/orig -> origin/gh/guilhermeleobas/256/orig 2025-12-04T08:27:07.1390663Z * [new branch] gh/guilhermeleobas/257/base -> origin/gh/guilhermeleobas/257/base 2025-12-04T08:27:07.1391981Z * [new branch] gh/guilhermeleobas/257/head -> origin/gh/guilhermeleobas/257/head 2025-12-04T08:27:07.1392279Z * [new branch] gh/guilhermeleobas/257/orig -> origin/gh/guilhermeleobas/257/orig 2025-12-04T08:27:07.1393475Z * [new branch] gh/guilhermeleobas/258/base -> origin/gh/guilhermeleobas/258/base 2025-12-04T08:27:07.1393919Z * [new branch] gh/guilhermeleobas/258/head -> origin/gh/guilhermeleobas/258/head 2025-12-04T08:27:07.1394566Z * [new branch] gh/guilhermeleobas/258/orig -> origin/gh/guilhermeleobas/258/orig 2025-12-04T08:27:07.1395714Z * [new branch] gh/guilhermeleobas/259/base -> origin/gh/guilhermeleobas/259/base 2025-12-04T08:27:07.1395994Z * [new branch] gh/guilhermeleobas/259/head -> origin/gh/guilhermeleobas/259/head 2025-12-04T08:27:07.1398611Z * [new branch] gh/guilhermeleobas/259/orig -> origin/gh/guilhermeleobas/259/orig 2025-12-04T08:27:07.1398825Z * [new branch] gh/guilhermeleobas/260/base -> origin/gh/guilhermeleobas/260/base 2025-12-04T08:27:07.1399005Z * [new branch] gh/guilhermeleobas/260/head -> origin/gh/guilhermeleobas/260/head 2025-12-04T08:27:07.1399561Z * [new branch] gh/guilhermeleobas/260/orig -> origin/gh/guilhermeleobas/260/orig 2025-12-04T08:27:07.1400047Z * [new branch] gh/guilhermeleobas/261/base -> origin/gh/guilhermeleobas/261/base 2025-12-04T08:27:07.1400990Z * [new branch] gh/guilhermeleobas/261/head -> origin/gh/guilhermeleobas/261/head 2025-12-04T08:27:07.1401429Z * [new branch] gh/guilhermeleobas/261/orig -> origin/gh/guilhermeleobas/261/orig 2025-12-04T08:27:07.1402527Z * [new branch] gh/guilhermeleobas/262/base -> origin/gh/guilhermeleobas/262/base 2025-12-04T08:27:07.1403002Z * [new branch] gh/guilhermeleobas/262/head -> origin/gh/guilhermeleobas/262/head 2025-12-04T08:27:07.1404048Z * [new branch] gh/guilhermeleobas/262/orig -> origin/gh/guilhermeleobas/262/orig 2025-12-04T08:27:07.1404673Z * [new branch] gh/guilhermeleobas/263/base -> origin/gh/guilhermeleobas/263/base 2025-12-04T08:27:07.1405803Z * [new branch] gh/guilhermeleobas/263/head -> origin/gh/guilhermeleobas/263/head 2025-12-04T08:27:07.1405995Z * [new branch] gh/guilhermeleobas/263/orig -> origin/gh/guilhermeleobas/263/orig 2025-12-04T08:27:07.1407189Z * [new branch] gh/guilhermeleobas/264/base -> origin/gh/guilhermeleobas/264/base 2025-12-04T08:27:07.1407538Z * [new branch] gh/guilhermeleobas/264/head -> origin/gh/guilhermeleobas/264/head 2025-12-04T08:27:07.1408472Z * [new branch] gh/guilhermeleobas/264/orig -> origin/gh/guilhermeleobas/264/orig 2025-12-04T08:27:07.1409429Z * [new branch] gh/guilhermeleobas/265/base -> origin/gh/guilhermeleobas/265/base 2025-12-04T08:27:07.1409620Z * [new branch] gh/guilhermeleobas/265/head -> origin/gh/guilhermeleobas/265/head 2025-12-04T08:27:07.1411155Z * [new branch] gh/guilhermeleobas/265/orig -> origin/gh/guilhermeleobas/265/orig 2025-12-04T08:27:07.1411522Z * [new branch] gh/guilhermeleobas/266/base -> origin/gh/guilhermeleobas/266/base 2025-12-04T08:27:07.1412092Z * [new branch] gh/guilhermeleobas/266/head -> origin/gh/guilhermeleobas/266/head 2025-12-04T08:27:07.1412690Z * [new branch] gh/guilhermeleobas/266/orig -> origin/gh/guilhermeleobas/266/orig 2025-12-04T08:27:07.1413979Z * [new branch] gh/guilhermeleobas/267/base -> origin/gh/guilhermeleobas/267/base 2025-12-04T08:27:07.1414186Z * [new branch] gh/guilhermeleobas/267/head -> origin/gh/guilhermeleobas/267/head 2025-12-04T08:27:07.1419589Z * [new branch] gh/guilhermeleobas/267/orig -> origin/gh/guilhermeleobas/267/orig 2025-12-04T08:27:07.1419798Z * [new branch] gh/hameerabbasi/1/base -> origin/gh/hameerabbasi/1/base 2025-12-04T08:27:07.1419990Z * [new branch] gh/hameerabbasi/1/head -> origin/gh/hameerabbasi/1/head 2025-12-04T08:27:07.1420148Z * [new branch] gh/hameerabbasi/2/base -> origin/gh/hameerabbasi/2/base 2025-12-04T08:27:07.1420324Z * [new branch] gh/hameerabbasi/2/head -> origin/gh/hameerabbasi/2/head 2025-12-04T08:27:07.1420475Z * [new branch] gh/hameerabbasi/2/orig -> origin/gh/hameerabbasi/2/orig 2025-12-04T08:27:07.1420935Z * [new branch] gh/hameerabbasi/3/base -> origin/gh/hameerabbasi/3/base 2025-12-04T08:27:07.1421112Z * [new branch] gh/hameerabbasi/3/head -> origin/gh/hameerabbasi/3/head 2025-12-04T08:27:07.1421562Z * [new branch] gh/hameerabbasi/3/orig -> origin/gh/hameerabbasi/3/orig 2025-12-04T08:27:07.1423376Z * [new branch] gh/hameerabbasi/4/base -> origin/gh/hameerabbasi/4/base 2025-12-04T08:27:07.1423636Z * [new branch] gh/hameerabbasi/4/head -> origin/gh/hameerabbasi/4/head 2025-12-04T08:27:07.1423879Z * [new branch] gh/hameerabbasi/4/orig -> origin/gh/hameerabbasi/4/orig 2025-12-04T08:27:07.1426192Z * [new branch] gh/huydhn/1/next -> origin/gh/huydhn/1/next 2025-12-04T08:27:07.1426537Z * [new branch] gh/huydhn/2/next -> origin/gh/huydhn/2/next 2025-12-04T08:27:07.1426835Z * [new branch] gh/huydhn/3/next -> origin/gh/huydhn/3/next 2025-12-04T08:27:07.1427176Z * [new branch] gh/huydhn/4/next -> origin/gh/huydhn/4/next 2025-12-04T08:27:07.1428759Z * [new branch] gh/huydhn/5/next -> origin/gh/huydhn/5/next 2025-12-04T08:27:07.1428961Z * [new branch] gh/huydhn/6/next -> origin/gh/huydhn/6/next 2025-12-04T08:27:07.1432720Z * [new branch] gh/int3/97/base -> origin/gh/int3/97/base 2025-12-04T08:27:07.1433071Z * [new branch] gh/int3/97/head -> origin/gh/int3/97/head 2025-12-04T08:27:07.1433366Z * [new branch] gh/isuruf/101/base -> origin/gh/isuruf/101/base 2025-12-04T08:27:07.1433861Z * [new branch] gh/isuruf/101/head -> origin/gh/isuruf/101/head 2025-12-04T08:27:07.1434174Z * [new branch] gh/isuruf/146/base -> origin/gh/isuruf/146/base 2025-12-04T08:27:07.1434793Z * [new branch] gh/isuruf/146/head -> origin/gh/isuruf/146/head 2025-12-04T08:27:07.1435434Z * [new branch] gh/isuruf/146/orig -> origin/gh/isuruf/146/orig 2025-12-04T08:27:07.1435727Z * [new branch] gh/isuruf/158/base -> origin/gh/isuruf/158/base 2025-12-04T08:27:07.1436110Z * [new branch] gh/isuruf/158/head -> origin/gh/isuruf/158/head 2025-12-04T08:27:07.1437334Z * [new branch] gh/isuruf/159/base -> origin/gh/isuruf/159/base 2025-12-04T08:27:07.1437627Z * [new branch] gh/isuruf/159/head -> origin/gh/isuruf/159/head 2025-12-04T08:27:07.1438886Z * [new branch] gh/isuruf/160/base -> origin/gh/isuruf/160/base 2025-12-04T08:27:07.1439319Z * [new branch] gh/isuruf/160/head -> origin/gh/isuruf/160/head 2025-12-04T08:27:07.1441625Z * [new branch] gh/isuruf/160/orig -> origin/gh/isuruf/160/orig 2025-12-04T08:27:07.1446002Z * [new branch] gh/isuruf/81/base -> origin/gh/isuruf/81/base 2025-12-04T08:27:07.1446578Z * [new branch] gh/isuruf/81/head -> origin/gh/isuruf/81/head 2025-12-04T08:27:07.1446755Z * [new branch] gh/isuruf/81/orig -> origin/gh/isuruf/81/orig 2025-12-04T08:27:07.1447109Z * [new branch] gh/jamesjwu/176/base -> origin/gh/jamesjwu/176/base 2025-12-04T08:27:07.1447278Z * [new branch] gh/jamesjwu/176/head -> origin/gh/jamesjwu/176/head 2025-12-04T08:27:07.1447426Z * [new branch] gh/jamesjwu/176/orig -> origin/gh/jamesjwu/176/orig 2025-12-04T08:27:07.1447579Z * [new branch] gh/jamesjwu/187/base -> origin/gh/jamesjwu/187/base 2025-12-04T08:27:07.1447740Z * [new branch] gh/jamesjwu/187/head -> origin/gh/jamesjwu/187/head 2025-12-04T08:27:07.1448123Z * [new branch] gh/jamesjwu/187/orig -> origin/gh/jamesjwu/187/orig 2025-12-04T08:27:07.1448675Z * [new branch] gh/jamesjwu/196/base -> origin/gh/jamesjwu/196/base 2025-12-04T08:27:07.1451587Z * [new branch] gh/jamesjwu/196/head -> origin/gh/jamesjwu/196/head 2025-12-04T08:27:07.1454972Z * [new branch] gh/jamesjwu/196/orig -> origin/gh/jamesjwu/196/orig 2025-12-04T08:27:07.1457614Z * [new branch] gh/jamesjwu/198/base -> origin/gh/jamesjwu/198/base 2025-12-04T08:27:07.1457949Z * [new branch] gh/jamesjwu/198/head -> origin/gh/jamesjwu/198/head 2025-12-04T08:27:07.1458190Z * [new branch] gh/jamesjwu/198/orig -> origin/gh/jamesjwu/198/orig 2025-12-04T08:27:07.1458359Z * [new branch] gh/jamesjwu/207/base -> origin/gh/jamesjwu/207/base 2025-12-04T08:27:07.1458531Z * [new branch] gh/jamesjwu/207/head -> origin/gh/jamesjwu/207/head 2025-12-04T08:27:07.1458782Z * [new branch] gh/jamesjwu/207/orig -> origin/gh/jamesjwu/207/orig 2025-12-04T08:27:07.1459491Z * [new branch] gh/jamesjwu/208/base -> origin/gh/jamesjwu/208/base 2025-12-04T08:27:07.1459684Z * [new branch] gh/jamesjwu/208/head -> origin/gh/jamesjwu/208/head 2025-12-04T08:27:07.1459832Z * [new branch] gh/jamesjwu/208/orig -> origin/gh/jamesjwu/208/orig 2025-12-04T08:27:07.1459998Z * [new branch] gh/jamesjwu/52/base -> origin/gh/jamesjwu/52/base 2025-12-04T08:27:07.1460147Z * [new branch] gh/jamesjwu/52/head -> origin/gh/jamesjwu/52/head 2025-12-04T08:27:07.1460298Z * [new branch] gh/jamesjwu/53/base -> origin/gh/jamesjwu/53/base 2025-12-04T08:27:07.1460494Z * [new branch] gh/jamesjwu/53/head -> origin/gh/jamesjwu/53/head 2025-12-04T08:27:07.1467708Z * [new branch] gh/jamesjwu/54/base -> origin/gh/jamesjwu/54/base 2025-12-04T08:27:07.1468173Z * [new branch] gh/jamesjwu/54/head -> origin/gh/jamesjwu/54/head 2025-12-04T08:27:07.1468542Z * [new branch] gh/jamesjwu/55/base -> origin/gh/jamesjwu/55/base 2025-12-04T08:27:07.1468883Z * [new branch] gh/jamesjwu/55/head -> origin/gh/jamesjwu/55/head 2025-12-04T08:27:07.1469231Z * [new branch] gh/jamesjwu/56/base -> origin/gh/jamesjwu/56/base 2025-12-04T08:27:07.1469564Z * [new branch] gh/jamesjwu/56/head -> origin/gh/jamesjwu/56/head 2025-12-04T08:27:07.1470022Z * [new branch] gh/jamesjwu/57/base -> origin/gh/jamesjwu/57/base 2025-12-04T08:27:07.1470391Z * [new branch] gh/jamesjwu/57/head -> origin/gh/jamesjwu/57/head 2025-12-04T08:27:07.1470723Z * [new branch] gh/jamesjwu/58/base -> origin/gh/jamesjwu/58/base 2025-12-04T08:27:07.1471078Z * [new branch] gh/jamesjwu/58/head -> origin/gh/jamesjwu/58/head 2025-12-04T08:27:07.1471465Z * [new branch] gh/jamesjwu/59/base -> origin/gh/jamesjwu/59/base 2025-12-04T08:27:07.1471802Z * [new branch] gh/jamesjwu/59/head -> origin/gh/jamesjwu/59/head 2025-12-04T08:27:07.1472135Z * [new branch] gh/jamesjwu/60/base -> origin/gh/jamesjwu/60/base 2025-12-04T08:27:07.1472468Z * [new branch] gh/jamesjwu/60/head -> origin/gh/jamesjwu/60/head 2025-12-04T08:27:07.1473020Z * [new branch] gh/jamesjwu/61/base -> origin/gh/jamesjwu/61/base 2025-12-04T08:27:07.1473485Z * [new branch] gh/jamesjwu/61/head -> origin/gh/jamesjwu/61/head 2025-12-04T08:27:07.1473965Z * [new branch] gh/jamesjwu/62/base -> origin/gh/jamesjwu/62/base 2025-12-04T08:27:07.1474453Z * [new branch] gh/jamesjwu/62/head -> origin/gh/jamesjwu/62/head 2025-12-04T08:27:07.1475145Z * [new branch] gh/jamesjwu/63/base -> origin/gh/jamesjwu/63/base 2025-12-04T08:27:07.1475994Z * [new branch] gh/jamesjwu/63/head -> origin/gh/jamesjwu/63/head 2025-12-04T08:27:07.1478403Z * [new branch] gh/jamesjwu/64/base -> origin/gh/jamesjwu/64/base 2025-12-04T08:27:07.1478774Z * [new branch] gh/jamesjwu/64/head -> origin/gh/jamesjwu/64/head 2025-12-04T08:27:07.1479715Z * [new branch] gh/jamesjwu/65/base -> origin/gh/jamesjwu/65/base 2025-12-04T08:27:07.1480223Z * [new branch] gh/jamesjwu/65/head -> origin/gh/jamesjwu/65/head 2025-12-04T08:27:07.1480655Z * [new branch] gh/janeyx99/165/base -> origin/gh/janeyx99/165/base 2025-12-04T08:27:07.1483633Z * [new branch] gh/janeyx99/165/head -> origin/gh/janeyx99/165/head 2025-12-04T08:27:07.1484240Z * [new branch] gh/janeyx99/165/orig -> origin/gh/janeyx99/165/orig 2025-12-04T08:27:07.1484754Z * [new branch] gh/janeyx99/201/base -> origin/gh/janeyx99/201/base 2025-12-04T08:27:07.1485124Z * [new branch] gh/janeyx99/201/head -> origin/gh/janeyx99/201/head 2025-12-04T08:27:07.1485484Z * [new branch] gh/janeyx99/201/orig -> origin/gh/janeyx99/201/orig 2025-12-04T08:27:07.1485842Z * [new branch] gh/janeyx99/225/base -> origin/gh/janeyx99/225/base 2025-12-04T08:27:07.1486432Z * [new branch] gh/janeyx99/225/head -> origin/gh/janeyx99/225/head 2025-12-04T08:27:07.1487033Z * [new branch] gh/janeyx99/225/orig -> origin/gh/janeyx99/225/orig 2025-12-04T08:27:07.1488609Z * [new branch] gh/janeyx99/299/base -> origin/gh/janeyx99/299/base 2025-12-04T08:27:07.1489111Z * [new branch] gh/janeyx99/299/head -> origin/gh/janeyx99/299/head 2025-12-04T08:27:07.1489696Z * [new branch] gh/janeyx99/299/orig -> origin/gh/janeyx99/299/orig 2025-12-04T08:27:07.1492308Z * [new branch] gh/janeyx99/302/base -> origin/gh/janeyx99/302/base 2025-12-04T08:27:07.1492938Z * [new branch] gh/janeyx99/302/head -> origin/gh/janeyx99/302/head 2025-12-04T08:27:07.1493469Z * [new branch] gh/janeyx99/303/base -> origin/gh/janeyx99/303/base 2025-12-04T08:27:07.1493860Z * [new branch] gh/janeyx99/303/head -> origin/gh/janeyx99/303/head 2025-12-04T08:27:07.1494255Z * [new branch] gh/janeyx99/305/base -> origin/gh/janeyx99/305/base 2025-12-04T08:27:07.1494648Z * [new branch] gh/janeyx99/305/head -> origin/gh/janeyx99/305/head 2025-12-04T08:27:07.1497925Z * [new branch] gh/janeyx99/306/base -> origin/gh/janeyx99/306/base 2025-12-04T08:27:07.1498510Z * [new branch] gh/janeyx99/306/head -> origin/gh/janeyx99/306/head 2025-12-04T08:27:07.1499036Z * [new branch] gh/janeyx99/314/base -> origin/gh/janeyx99/314/base 2025-12-04T08:27:07.1499832Z * [new branch] gh/janeyx99/314/head -> origin/gh/janeyx99/314/head 2025-12-04T08:27:07.1500308Z * [new branch] gh/janeyx99/314/orig -> origin/gh/janeyx99/314/orig 2025-12-04T08:27:07.1500667Z * [new branch] gh/janeyx99/315/base -> origin/gh/janeyx99/315/base 2025-12-04T08:27:07.1500998Z * [new branch] gh/janeyx99/315/head -> origin/gh/janeyx99/315/head 2025-12-04T08:27:07.1501518Z * [new branch] gh/janeyx99/315/orig -> origin/gh/janeyx99/315/orig 2025-12-04T08:27:07.1507827Z * [new branch] gh/janeyx99/316/base -> origin/gh/janeyx99/316/base 2025-12-04T08:27:07.1508474Z * [new branch] gh/janeyx99/316/head -> origin/gh/janeyx99/316/head 2025-12-04T08:27:07.1509688Z * [new branch] gh/janeyx99/316/orig -> origin/gh/janeyx99/316/orig 2025-12-04T08:27:07.1510107Z * [new branch] gh/janeyx99/317/base -> origin/gh/janeyx99/317/base 2025-12-04T08:27:07.1510471Z * [new branch] gh/janeyx99/317/head -> origin/gh/janeyx99/317/head 2025-12-04T08:27:07.1510996Z * [new branch] gh/janeyx99/317/orig -> origin/gh/janeyx99/317/orig 2025-12-04T08:27:07.1511366Z * [new branch] gh/janeyx99/325/base -> origin/gh/janeyx99/325/base 2025-12-04T08:27:07.1511735Z * [new branch] gh/janeyx99/325/head -> origin/gh/janeyx99/325/head 2025-12-04T08:27:07.1512119Z * [new branch] gh/janeyx99/325/orig -> origin/gh/janeyx99/325/orig 2025-12-04T08:27:07.1512479Z * [new branch] gh/janeyx99/327/base -> origin/gh/janeyx99/327/base 2025-12-04T08:27:07.1512840Z * [new branch] gh/janeyx99/327/head -> origin/gh/janeyx99/327/head 2025-12-04T08:27:07.1513212Z * [new branch] gh/janeyx99/327/orig -> origin/gh/janeyx99/327/orig 2025-12-04T08:27:07.1513756Z * [new branch] gh/janeyx99/328/base -> origin/gh/janeyx99/328/base 2025-12-04T08:27:07.1514266Z * [new branch] gh/janeyx99/328/head -> origin/gh/janeyx99/328/head 2025-12-04T08:27:07.1514773Z * [new branch] gh/janeyx99/328/orig -> origin/gh/janeyx99/328/orig 2025-12-04T08:27:07.1515155Z * [new branch] gh/janeyx99/329/base -> origin/gh/janeyx99/329/base 2025-12-04T08:27:07.1515524Z * [new branch] gh/janeyx99/329/head -> origin/gh/janeyx99/329/head 2025-12-04T08:27:07.1515875Z * [new branch] gh/janeyx99/329/orig -> origin/gh/janeyx99/329/orig 2025-12-04T08:27:07.1518574Z * [new branch] gh/janeyx99/330/base -> origin/gh/janeyx99/330/base 2025-12-04T08:27:07.1518967Z * [new branch] gh/janeyx99/330/head -> origin/gh/janeyx99/330/head 2025-12-04T08:27:07.1519442Z * [new branch] gh/janeyx99/330/orig -> origin/gh/janeyx99/330/orig 2025-12-04T08:27:07.1519839Z * [new branch] gh/janeyx99/331/base -> origin/gh/janeyx99/331/base 2025-12-04T08:27:07.1520215Z * [new branch] gh/janeyx99/331/head -> origin/gh/janeyx99/331/head 2025-12-04T08:27:07.1520577Z * [new branch] gh/janeyx99/331/orig -> origin/gh/janeyx99/331/orig 2025-12-04T08:27:07.1526073Z * [new branch] gh/janeyx99/332/base -> origin/gh/janeyx99/332/base 2025-12-04T08:27:07.1528075Z * [new branch] gh/janeyx99/332/head -> origin/gh/janeyx99/332/head 2025-12-04T08:27:07.1528606Z * [new branch] gh/janeyx99/332/orig -> origin/gh/janeyx99/332/orig 2025-12-04T08:27:07.1533606Z * [new branch] gh/janeyx99/333/base -> origin/gh/janeyx99/333/base 2025-12-04T08:27:07.1534185Z * [new branch] gh/janeyx99/333/head -> origin/gh/janeyx99/333/head 2025-12-04T08:27:07.1534669Z * [new branch] gh/janeyx99/333/orig -> origin/gh/janeyx99/333/orig 2025-12-04T08:27:07.1535530Z * [new branch] gh/janeyx99/88/base -> origin/gh/janeyx99/88/base 2025-12-04T08:27:07.1535973Z * [new branch] gh/janeyx99/88/head -> origin/gh/janeyx99/88/head 2025-12-04T08:27:07.1536340Z * [new branch] gh/janeyx99/88/orig -> origin/gh/janeyx99/88/orig 2025-12-04T08:27:07.1536693Z * [new branch] gh/jansel/360/base -> origin/gh/jansel/360/base 2025-12-04T08:27:07.1537048Z * [new branch] gh/jansel/360/head -> origin/gh/jansel/360/head 2025-12-04T08:27:07.1537395Z * [new branch] gh/jansel/451/base -> origin/gh/jansel/451/base 2025-12-04T08:27:07.1537720Z * [new branch] gh/jansel/451/head -> origin/gh/jansel/451/head 2025-12-04T08:27:07.1538039Z * [new branch] gh/jansel/451/orig -> origin/gh/jansel/451/orig 2025-12-04T08:27:07.1538364Z * [new branch] gh/jansel/462/base -> origin/gh/jansel/462/base 2025-12-04T08:27:07.1538691Z * [new branch] gh/jansel/462/head -> origin/gh/jansel/462/head 2025-12-04T08:27:07.1539217Z * [new branch] gh/jansel/462/orig -> origin/gh/jansel/462/orig 2025-12-04T08:27:07.1539542Z * [new branch] gh/jansel/533/base -> origin/gh/jansel/533/base 2025-12-04T08:27:07.1539869Z * [new branch] gh/jansel/533/head -> origin/gh/jansel/533/head 2025-12-04T08:27:07.1540209Z * [new branch] gh/jansel/533/orig -> origin/gh/jansel/533/orig 2025-12-04T08:27:07.1540527Z * [new branch] gh/jansel/552/base -> origin/gh/jansel/552/base 2025-12-04T08:27:07.1540858Z * [new branch] gh/jansel/552/head -> origin/gh/jansel/552/head 2025-12-04T08:27:07.1541187Z * [new branch] gh/jansel/552/orig -> origin/gh/jansel/552/orig 2025-12-04T08:27:07.1541507Z * [new branch] gh/jansel/553/base -> origin/gh/jansel/553/base 2025-12-04T08:27:07.1541819Z * [new branch] gh/jansel/553/head -> origin/gh/jansel/553/head 2025-12-04T08:27:07.1542331Z * [new branch] gh/jansel/553/orig -> origin/gh/jansel/553/orig 2025-12-04T08:27:07.1542796Z * [new branch] gh/jansel/554/base -> origin/gh/jansel/554/base 2025-12-04T08:27:07.1543249Z * [new branch] gh/jansel/554/head -> origin/gh/jansel/554/head 2025-12-04T08:27:07.1543708Z * [new branch] gh/jansel/554/orig -> origin/gh/jansel/554/orig 2025-12-04T08:27:07.1544049Z * [new branch] gh/jansel/555/base -> origin/gh/jansel/555/base 2025-12-04T08:27:07.1544379Z * [new branch] gh/jansel/555/head -> origin/gh/jansel/555/head 2025-12-04T08:27:07.1544771Z * [new branch] gh/jansel/555/orig -> origin/gh/jansel/555/orig 2025-12-04T08:27:07.1550327Z * [new branch] gh/jansel/556/base -> origin/gh/jansel/556/base 2025-12-04T08:27:07.1550902Z * [new branch] gh/jansel/556/head -> origin/gh/jansel/556/head 2025-12-04T08:27:07.1551397Z * [new branch] gh/jansel/556/orig -> origin/gh/jansel/556/orig 2025-12-04T08:27:07.1552162Z * [new branch] gh/jansel/557/base -> origin/gh/jansel/557/base 2025-12-04T08:27:07.1552598Z * [new branch] gh/jansel/557/head -> origin/gh/jansel/557/head 2025-12-04T08:27:07.1552957Z * [new branch] gh/jansel/557/orig -> origin/gh/jansel/557/orig 2025-12-04T08:27:07.1553297Z * [new branch] gh/jansel/558/base -> origin/gh/jansel/558/base 2025-12-04T08:27:07.1553644Z * [new branch] gh/jansel/558/head -> origin/gh/jansel/558/head 2025-12-04T08:27:07.1554008Z * [new branch] gh/jansel/558/orig -> origin/gh/jansel/558/orig 2025-12-04T08:27:07.1554358Z * [new branch] gh/jansel/559/base -> origin/gh/jansel/559/base 2025-12-04T08:27:07.1554691Z * [new branch] gh/jansel/559/head -> origin/gh/jansel/559/head 2025-12-04T08:27:07.1555112Z * [new branch] gh/jansel/559/orig -> origin/gh/jansel/559/orig 2025-12-04T08:27:07.1555452Z * [new branch] gh/jansel/560/base -> origin/gh/jansel/560/base 2025-12-04T08:27:07.1555783Z * [new branch] gh/jansel/560/head -> origin/gh/jansel/560/head 2025-12-04T08:27:07.1556135Z * [new branch] gh/jansel/560/orig -> origin/gh/jansel/560/orig 2025-12-04T08:27:07.1556515Z * [new branch] gh/jansel/561/base -> origin/gh/jansel/561/base 2025-12-04T08:27:07.1556840Z * [new branch] gh/jansel/561/head -> origin/gh/jansel/561/head 2025-12-04T08:27:07.1557170Z * [new branch] gh/jansel/561/orig -> origin/gh/jansel/561/orig 2025-12-04T08:27:07.1557505Z * [new branch] gh/jansel/562/base -> origin/gh/jansel/562/base 2025-12-04T08:27:07.1558264Z * [new branch] gh/jansel/562/head -> origin/gh/jansel/562/head 2025-12-04T08:27:07.1558633Z * [new branch] gh/jansel/562/orig -> origin/gh/jansel/562/orig 2025-12-04T08:27:07.1558974Z * [new branch] gh/jansel/563/base -> origin/gh/jansel/563/base 2025-12-04T08:27:07.1559338Z * [new branch] gh/jansel/563/head -> origin/gh/jansel/563/head 2025-12-04T08:27:07.1559674Z * [new branch] gh/jansel/563/orig -> origin/gh/jansel/563/orig 2025-12-04T08:27:07.1568782Z * [new branch] gh/jansel/564/base -> origin/gh/jansel/564/base 2025-12-04T08:27:07.1569514Z * [new branch] gh/jansel/564/head -> origin/gh/jansel/564/head 2025-12-04T08:27:07.1569873Z * [new branch] gh/jansel/564/orig -> origin/gh/jansel/564/orig 2025-12-04T08:27:07.1570208Z * [new branch] gh/jansel/565/base -> origin/gh/jansel/565/base 2025-12-04T08:27:07.1570568Z * [new branch] gh/jansel/565/head -> origin/gh/jansel/565/head 2025-12-04T08:27:07.1570901Z * [new branch] gh/jansel/565/orig -> origin/gh/jansel/565/orig 2025-12-04T08:27:07.1571222Z * [new branch] gh/jansel/566/base -> origin/gh/jansel/566/base 2025-12-04T08:27:07.1571559Z * [new branch] gh/jansel/566/head -> origin/gh/jansel/566/head 2025-12-04T08:27:07.1571884Z * [new branch] gh/jansel/566/orig -> origin/gh/jansel/566/orig 2025-12-04T08:27:07.1572205Z * [new branch] gh/jansel/567/base -> origin/gh/jansel/567/base 2025-12-04T08:27:07.1572531Z * [new branch] gh/jansel/567/head -> origin/gh/jansel/567/head 2025-12-04T08:27:07.1572984Z * [new branch] gh/jansel/567/orig -> origin/gh/jansel/567/orig 2025-12-04T08:27:07.1573321Z * [new branch] gh/jansel/568/base -> origin/gh/jansel/568/base 2025-12-04T08:27:07.1573642Z * [new branch] gh/jansel/568/head -> origin/gh/jansel/568/head 2025-12-04T08:27:07.1573982Z * [new branch] gh/jansel/568/orig -> origin/gh/jansel/568/orig 2025-12-04T08:27:07.1574309Z * [new branch] gh/jansel/569/base -> origin/gh/jansel/569/base 2025-12-04T08:27:07.1574634Z * [new branch] gh/jansel/569/head -> origin/gh/jansel/569/head 2025-12-04T08:27:07.1574953Z * [new branch] gh/jansel/569/orig -> origin/gh/jansel/569/orig 2025-12-04T08:27:07.1575277Z * [new branch] gh/jansel/570/base -> origin/gh/jansel/570/base 2025-12-04T08:27:07.1575605Z * [new branch] gh/jansel/570/head -> origin/gh/jansel/570/head 2025-12-04T08:27:07.1575933Z * [new branch] gh/jansel/570/orig -> origin/gh/jansel/570/orig 2025-12-04T08:27:07.1581157Z * [new branch] gh/jansel/571/base -> origin/gh/jansel/571/base 2025-12-04T08:27:07.1581581Z * [new branch] gh/jansel/571/head -> origin/gh/jansel/571/head 2025-12-04T08:27:07.1581918Z * [new branch] gh/jansel/571/orig -> origin/gh/jansel/571/orig 2025-12-04T08:27:07.1582249Z * [new branch] gh/jansel/572/base -> origin/gh/jansel/572/base 2025-12-04T08:27:07.1582571Z * [new branch] gh/jansel/572/head -> origin/gh/jansel/572/head 2025-12-04T08:27:07.1582887Z * [new branch] gh/jansel/572/orig -> origin/gh/jansel/572/orig 2025-12-04T08:27:07.1583211Z * [new branch] gh/jansel/573/base -> origin/gh/jansel/573/base 2025-12-04T08:27:07.1583541Z * [new branch] gh/jansel/573/head -> origin/gh/jansel/573/head 2025-12-04T08:27:07.1583864Z * [new branch] gh/jansel/573/orig -> origin/gh/jansel/573/orig 2025-12-04T08:27:07.1584185Z * [new branch] gh/jansel/574/base -> origin/gh/jansel/574/base 2025-12-04T08:27:07.1584706Z * [new branch] gh/jansel/574/head -> origin/gh/jansel/574/head 2025-12-04T08:27:07.1585023Z * [new branch] gh/jansel/574/orig -> origin/gh/jansel/574/orig 2025-12-04T08:27:07.1585349Z * [new branch] gh/jansel/575/base -> origin/gh/jansel/575/base 2025-12-04T08:27:07.1585671Z * [new branch] gh/jansel/575/head -> origin/gh/jansel/575/head 2025-12-04T08:27:07.1590379Z * [new branch] gh/jansel/575/orig -> origin/gh/jansel/575/orig 2025-12-04T08:27:07.1591056Z * [new branch] gh/jansel/576/base -> origin/gh/jansel/576/base 2025-12-04T08:27:07.1591416Z * [new branch] gh/jansel/576/head -> origin/gh/jansel/576/head 2025-12-04T08:27:07.1591768Z * [new branch] gh/jansel/576/orig -> origin/gh/jansel/576/orig 2025-12-04T08:27:07.1592152Z * [new branch] gh/jbschlosser/247/base -> origin/gh/jbschlosser/247/base 2025-12-04T08:27:07.1592538Z * [new branch] gh/jbschlosser/247/head -> origin/gh/jbschlosser/247/head 2025-12-04T08:27:07.1592891Z * [new branch] gh/jbschlosser/247/orig -> origin/gh/jbschlosser/247/orig 2025-12-04T08:27:07.1593250Z * [new branch] gh/jbschlosser/250/base -> origin/gh/jbschlosser/250/base 2025-12-04T08:27:07.1593614Z * [new branch] gh/jbschlosser/250/head -> origin/gh/jbschlosser/250/head 2025-12-04T08:27:07.1593987Z * [new branch] gh/jbschlosser/250/orig -> origin/gh/jbschlosser/250/orig 2025-12-04T08:27:07.1594359Z * [new branch] gh/jerryzh168/1/base -> origin/gh/jerryzh168/1/base 2025-12-04T08:27:07.1595756Z * [new branch] gh/jerryzh168/1/head -> origin/gh/jerryzh168/1/head 2025-12-04T08:27:07.1596159Z * [new branch] gh/jerryzh168/1/orig -> origin/gh/jerryzh168/1/orig 2025-12-04T08:27:07.1596553Z * [new branch] gh/jiayisunx/59/base -> origin/gh/jiayisunx/59/base 2025-12-04T08:27:07.1596939Z * [new branch] gh/jiayisunx/59/head -> origin/gh/jiayisunx/59/head 2025-12-04T08:27:07.1597324Z * [new branch] gh/jiayisunx/59/orig -> origin/gh/jiayisunx/59/orig 2025-12-04T08:27:07.1597704Z * [new branch] gh/jiayisunx/61/base -> origin/gh/jiayisunx/61/base 2025-12-04T08:27:07.1598195Z * [new branch] gh/jiayisunx/61/head -> origin/gh/jiayisunx/61/head 2025-12-04T08:27:07.1598580Z * [new branch] gh/jiayisunx/61/orig -> origin/gh/jiayisunx/61/orig 2025-12-04T08:27:07.1598961Z * [new branch] gh/jiayisunx/68/base -> origin/gh/jiayisunx/68/base 2025-12-04T08:27:07.1599348Z * [new branch] gh/jiayisunx/68/head -> origin/gh/jiayisunx/68/head 2025-12-04T08:27:07.1599677Z * [new branch] gh/jiayisunx/68/orig -> origin/gh/jiayisunx/68/orig 2025-12-04T08:27:07.1604496Z * [new branch] gh/jiayisunx/77/base -> origin/gh/jiayisunx/77/base 2025-12-04T08:27:07.1604914Z * [new branch] gh/jiayisunx/77/head -> origin/gh/jiayisunx/77/head 2025-12-04T08:27:07.1605275Z * [new branch] gh/jiayisunx/77/orig -> origin/gh/jiayisunx/77/orig 2025-12-04T08:27:07.1605641Z * [new branch] gh/jiayisunx/78/base -> origin/gh/jiayisunx/78/base 2025-12-04T08:27:07.1606014Z * [new branch] gh/jiayisunx/78/head -> origin/gh/jiayisunx/78/head 2025-12-04T08:27:07.1606376Z * [new branch] gh/jiayisunx/78/orig -> origin/gh/jiayisunx/78/orig 2025-12-04T08:27:07.1606723Z * [new branch] gh/jiayisunx/79/base -> origin/gh/jiayisunx/79/base 2025-12-04T08:27:07.1607078Z * [new branch] gh/jiayisunx/79/head -> origin/gh/jiayisunx/79/head 2025-12-04T08:27:07.1607429Z * [new branch] gh/jiayisunx/79/orig -> origin/gh/jiayisunx/79/orig 2025-12-04T08:27:07.1608120Z * [new branch] gh/jiayisunx/82/base -> origin/gh/jiayisunx/82/base 2025-12-04T08:27:07.1608483Z * [new branch] gh/jiayisunx/82/head -> origin/gh/jiayisunx/82/head 2025-12-04T08:27:07.1608832Z * [new branch] gh/jiayisunx/82/orig -> origin/gh/jiayisunx/82/orig 2025-12-04T08:27:07.1613625Z * [new branch] gh/jiayisunx/83/base -> origin/gh/jiayisunx/83/base 2025-12-04T08:27:07.1614065Z * [new branch] gh/jiayisunx/83/head -> origin/gh/jiayisunx/83/head 2025-12-04T08:27:07.1614435Z * [new branch] gh/jiayisunx/83/orig -> origin/gh/jiayisunx/83/orig 2025-12-04T08:27:07.1614807Z * [new branch] gh/jiayisunx/84/base -> origin/gh/jiayisunx/84/base 2025-12-04T08:27:07.1615167Z * [new branch] gh/jiayisunx/84/head -> origin/gh/jiayisunx/84/head 2025-12-04T08:27:07.1615524Z * [new branch] gh/jiayisunx/84/orig -> origin/gh/jiayisunx/84/orig 2025-12-04T08:27:07.1615886Z * [new branch] gh/jiayisunx/85/base -> origin/gh/jiayisunx/85/base 2025-12-04T08:27:07.1616725Z * [new branch] gh/jiayisunx/85/head -> origin/gh/jiayisunx/85/head 2025-12-04T08:27:07.1617094Z * [new branch] gh/jiayisunx/85/orig -> origin/gh/jiayisunx/85/orig 2025-12-04T08:27:07.1617443Z * [new branch] gh/jiayisunx/86/base -> origin/gh/jiayisunx/86/base 2025-12-04T08:27:07.1617797Z * [new branch] gh/jiayisunx/86/head -> origin/gh/jiayisunx/86/head 2025-12-04T08:27:07.1618141Z * [new branch] gh/jiayisunx/86/orig -> origin/gh/jiayisunx/86/orig 2025-12-04T08:27:07.1618658Z * [new branch] gh/jiayisunx/87/base -> origin/gh/jiayisunx/87/base 2025-12-04T08:27:07.1619022Z * [new branch] gh/jiayisunx/87/head -> origin/gh/jiayisunx/87/head 2025-12-04T08:27:07.1619394Z * [new branch] gh/jiayisunx/87/orig -> origin/gh/jiayisunx/87/orig 2025-12-04T08:27:07.1619762Z * [new branch] gh/jiayisunx/88/base -> origin/gh/jiayisunx/88/base 2025-12-04T08:27:07.1620129Z * [new branch] gh/jiayisunx/88/head -> origin/gh/jiayisunx/88/head 2025-12-04T08:27:07.1620496Z * [new branch] gh/jiayisunx/88/orig -> origin/gh/jiayisunx/88/orig 2025-12-04T08:27:07.1622879Z * [new branch] gh/jiayisunx/89/base -> origin/gh/jiayisunx/89/base 2025-12-04T08:27:07.1623810Z * [new branch] gh/jiayisunx/89/head -> origin/gh/jiayisunx/89/head 2025-12-04T08:27:07.1624252Z * [new branch] gh/jiayisunx/89/orig -> origin/gh/jiayisunx/89/orig 2025-12-04T08:27:07.1629438Z * [new branch] gh/jiayisunx/90/base -> origin/gh/jiayisunx/90/base 2025-12-04T08:27:07.1629896Z * [new branch] gh/jiayisunx/90/head -> origin/gh/jiayisunx/90/head 2025-12-04T08:27:07.1630336Z * [new branch] gh/jiayisunx/90/orig -> origin/gh/jiayisunx/90/orig 2025-12-04T08:27:07.1630724Z * [new branch] gh/jjwu@meta.com/1/base -> origin/gh/jjwu@meta.com/1/base 2025-12-04T08:27:07.1631107Z * [new branch] gh/jjwu@meta.com/1/head -> origin/gh/jjwu@meta.com/1/head 2025-12-04T08:27:07.1631482Z * [new branch] gh/jturney/1/base -> origin/gh/jturney/1/base 2025-12-04T08:27:07.1631842Z * [new branch] gh/jturney/1/head -> origin/gh/jturney/1/head 2025-12-04T08:27:07.1632202Z * [new branch] gh/jturney/1/orig -> origin/gh/jturney/1/orig 2025-12-04T08:27:07.1632554Z * [new branch] gh/jturney/2/base -> origin/gh/jturney/2/base 2025-12-04T08:27:07.1632925Z * [new branch] gh/jturney/2/head -> origin/gh/jturney/2/head 2025-12-04T08:27:07.1633305Z * [new branch] gh/jturney/2/orig -> origin/gh/jturney/2/orig 2025-12-04T08:27:07.1634109Z * [new branch] gh/karthickai/10/base -> origin/gh/karthickai/10/base 2025-12-04T08:27:07.1634595Z * [new branch] gh/karthickai/10/head -> origin/gh/karthickai/10/head 2025-12-04T08:27:07.1635007Z * [new branch] gh/karthickai/10/orig -> origin/gh/karthickai/10/orig 2025-12-04T08:27:07.1635684Z * [new branch] gh/karthickai/11/base -> origin/gh/karthickai/11/base 2025-12-04T08:27:07.1636082Z * [new branch] gh/karthickai/11/head -> origin/gh/karthickai/11/head 2025-12-04T08:27:07.1636467Z * [new branch] gh/karthickai/11/orig -> origin/gh/karthickai/11/orig 2025-12-04T08:27:07.1638099Z * [new branch] gh/karthickai/12/base -> origin/gh/karthickai/12/base 2025-12-04T08:27:07.1638618Z * [new branch] gh/karthickai/12/head -> origin/gh/karthickai/12/head 2025-12-04T08:27:07.1643280Z * [new branch] gh/karthickai/12/orig -> origin/gh/karthickai/12/orig 2025-12-04T08:27:07.1645379Z * [new branch] gh/karthickai/13/base -> origin/gh/karthickai/13/base 2025-12-04T08:27:07.1645952Z * [new branch] gh/karthickai/13/head -> origin/gh/karthickai/13/head 2025-12-04T08:27:07.1646357Z * [new branch] gh/karthickai/13/orig -> origin/gh/karthickai/13/orig 2025-12-04T08:27:07.1650641Z * [new branch] gh/karthickai/14/base -> origin/gh/karthickai/14/base 2025-12-04T08:27:07.1654932Z * [new branch] gh/karthickai/14/head -> origin/gh/karthickai/14/head 2025-12-04T08:27:07.1655376Z * [new branch] gh/karthickai/14/orig -> origin/gh/karthickai/14/orig 2025-12-04T08:27:07.1655928Z * [new branch] gh/karthickai/15/base -> origin/gh/karthickai/15/base 2025-12-04T08:27:07.1656276Z * [new branch] gh/karthickai/15/head -> origin/gh/karthickai/15/head 2025-12-04T08:27:07.1656612Z * [new branch] gh/karthickai/15/orig -> origin/gh/karthickai/15/orig 2025-12-04T08:27:07.1656945Z * [new branch] gh/karthickai/16/base -> origin/gh/karthickai/16/base 2025-12-04T08:27:07.1657281Z * [new branch] gh/karthickai/16/head -> origin/gh/karthickai/16/head 2025-12-04T08:27:07.1657625Z * [new branch] gh/karthickai/16/orig -> origin/gh/karthickai/16/orig 2025-12-04T08:27:07.1657967Z * [new branch] gh/karthickai/17/base -> origin/gh/karthickai/17/base 2025-12-04T08:27:07.1694584Z * [new branch] gh/karthickai/17/head -> origin/gh/karthickai/17/head 2025-12-04T08:27:07.1694977Z * [new branch] gh/karthickai/17/orig -> origin/gh/karthickai/17/orig 2025-12-04T08:27:07.1695353Z * [new branch] gh/karthickai/18/base -> origin/gh/karthickai/18/base 2025-12-04T08:27:07.1695705Z * [new branch] gh/karthickai/18/head -> origin/gh/karthickai/18/head 2025-12-04T08:27:07.1696051Z * [new branch] gh/karthickai/18/orig -> origin/gh/karthickai/18/orig 2025-12-04T08:27:07.1696423Z * [new branch] gh/karthickai/19/base -> origin/gh/karthickai/19/base 2025-12-04T08:27:07.1696762Z * [new branch] gh/karthickai/19/head -> origin/gh/karthickai/19/head 2025-12-04T08:27:07.1697095Z * [new branch] gh/karthickai/19/orig -> origin/gh/karthickai/19/orig 2025-12-04T08:27:07.1697422Z * [new branch] gh/karthickai/20/base -> origin/gh/karthickai/20/base 2025-12-04T08:27:07.1697762Z * [new branch] gh/karthickai/20/head -> origin/gh/karthickai/20/head 2025-12-04T08:27:07.1698107Z * [new branch] gh/karthickai/20/orig -> origin/gh/karthickai/20/orig 2025-12-04T08:27:07.1698443Z * [new branch] gh/karthickai/21/base -> origin/gh/karthickai/21/base 2025-12-04T08:27:07.1698771Z * [new branch] gh/karthickai/21/head -> origin/gh/karthickai/21/head 2025-12-04T08:27:07.1699228Z * [new branch] gh/karthickai/21/orig -> origin/gh/karthickai/21/orig 2025-12-04T08:27:07.1699565Z * [new branch] gh/karthickai/22/base -> origin/gh/karthickai/22/base 2025-12-04T08:27:07.1699893Z * [new branch] gh/karthickai/22/head -> origin/gh/karthickai/22/head 2025-12-04T08:27:07.1700229Z * [new branch] gh/karthickai/22/orig -> origin/gh/karthickai/22/orig 2025-12-04T08:27:07.1700563Z * [new branch] gh/karthickai/23/base -> origin/gh/karthickai/23/base 2025-12-04T08:27:07.1700901Z * [new branch] gh/karthickai/23/head -> origin/gh/karthickai/23/head 2025-12-04T08:27:07.1701233Z * [new branch] gh/karthickai/23/orig -> origin/gh/karthickai/23/orig 2025-12-04T08:27:07.1701573Z * [new branch] gh/karthickai/24/base -> origin/gh/karthickai/24/base 2025-12-04T08:27:07.1701909Z * [new branch] gh/karthickai/24/head -> origin/gh/karthickai/24/head 2025-12-04T08:27:07.1702258Z * [new branch] gh/karthickai/24/orig -> origin/gh/karthickai/24/orig 2025-12-04T08:27:07.1702601Z * [new branch] gh/karthickai/25/base -> origin/gh/karthickai/25/base 2025-12-04T08:27:07.1702943Z * [new branch] gh/karthickai/25/head -> origin/gh/karthickai/25/head 2025-12-04T08:27:07.1703288Z * [new branch] gh/karthickai/25/orig -> origin/gh/karthickai/25/orig 2025-12-04T08:27:07.1703611Z * [new branch] gh/karthickai/26/base -> origin/gh/karthickai/26/base 2025-12-04T08:27:07.1703942Z * [new branch] gh/karthickai/26/head -> origin/gh/karthickai/26/head 2025-12-04T08:27:07.1704318Z * [new branch] gh/karthickai/26/orig -> origin/gh/karthickai/26/orig 2025-12-04T08:27:07.1704667Z * [new branch] gh/karthickai/6/base -> origin/gh/karthickai/6/base 2025-12-04T08:27:07.1705005Z * [new branch] gh/karthickai/6/head -> origin/gh/karthickai/6/head 2025-12-04T08:27:07.1705338Z * [new branch] gh/karthickai/6/orig -> origin/gh/karthickai/6/orig 2025-12-04T08:27:07.1705664Z * [new branch] gh/krocki/1/base -> origin/gh/krocki/1/base 2025-12-04T08:27:07.1705983Z * [new branch] gh/krocki/1/head -> origin/gh/krocki/1/head 2025-12-04T08:27:07.1706284Z * [new branch] gh/krocki/1/orig -> origin/gh/krocki/1/orig 2025-12-04T08:27:07.1706592Z * [new branch] gh/krocki/2/base -> origin/gh/krocki/2/base 2025-12-04T08:27:07.1706900Z * [new branch] gh/krocki/2/head -> origin/gh/krocki/2/head 2025-12-04T08:27:07.1707207Z * [new branch] gh/krocki/2/orig -> origin/gh/krocki/2/orig 2025-12-04T08:27:07.1707542Z * [new branch] gh/kurtamohler/60/base -> origin/gh/kurtamohler/60/base 2025-12-04T08:27:07.1707906Z * [new branch] gh/kurtamohler/60/head -> origin/gh/kurtamohler/60/head 2025-12-04T08:27:07.1708285Z * [new branch] gh/kurtamohler/60/orig -> origin/gh/kurtamohler/60/orig 2025-12-04T08:27:07.1708622Z * [new branch] gh/kurtamohler/61/base -> origin/gh/kurtamohler/61/base 2025-12-04T08:27:07.1708966Z * [new branch] gh/kurtamohler/61/head -> origin/gh/kurtamohler/61/head 2025-12-04T08:27:07.1709307Z * [new branch] gh/kurtamohler/61/orig -> origin/gh/kurtamohler/61/orig 2025-12-04T08:27:07.1709649Z * [new branch] gh/kurtamohler/62/base -> origin/gh/kurtamohler/62/base 2025-12-04T08:27:07.1709985Z * [new branch] gh/kurtamohler/62/head -> origin/gh/kurtamohler/62/head 2025-12-04T08:27:07.1710327Z * [new branch] gh/kurtamohler/62/orig -> origin/gh/kurtamohler/62/orig 2025-12-04T08:27:07.1710671Z * [new branch] gh/kurtamohler/63/base -> origin/gh/kurtamohler/63/base 2025-12-04T08:27:07.1711044Z * [new branch] gh/kurtamohler/63/head -> origin/gh/kurtamohler/63/head 2025-12-04T08:27:07.1711380Z * [new branch] gh/kurtamohler/63/orig -> origin/gh/kurtamohler/63/orig 2025-12-04T08:27:07.1711729Z * [new branch] gh/kurtamohler/64/base -> origin/gh/kurtamohler/64/base 2025-12-04T08:27:07.1712071Z * [new branch] gh/kurtamohler/64/head -> origin/gh/kurtamohler/64/head 2025-12-04T08:27:07.1712413Z * [new branch] gh/kurtamohler/64/orig -> origin/gh/kurtamohler/64/orig 2025-12-04T08:27:07.1712750Z * [new branch] gh/kurtamohler/65/base -> origin/gh/kurtamohler/65/base 2025-12-04T08:27:07.1713103Z * [new branch] gh/kurtamohler/65/head -> origin/gh/kurtamohler/65/head 2025-12-04T08:27:07.1713453Z * [new branch] gh/kurtamohler/65/orig -> origin/gh/kurtamohler/65/orig 2025-12-04T08:27:07.1713808Z * [new branch] gh/kurtamohler/66/base -> origin/gh/kurtamohler/66/base 2025-12-04T08:27:07.1714150Z * [new branch] gh/kurtamohler/66/head -> origin/gh/kurtamohler/66/head 2025-12-04T08:27:07.1714499Z * [new branch] gh/kurtamohler/66/orig -> origin/gh/kurtamohler/66/orig 2025-12-04T08:27:07.1714851Z * [new branch] gh/kurtamohler/67/base -> origin/gh/kurtamohler/67/base 2025-12-04T08:27:07.1715255Z * [new branch] gh/kurtamohler/67/head -> origin/gh/kurtamohler/67/head 2025-12-04T08:27:07.1715633Z * [new branch] gh/kurtamohler/67/orig -> origin/gh/kurtamohler/67/orig 2025-12-04T08:27:07.1716004Z * [new branch] gh/kwen2501/130/base -> origin/gh/kwen2501/130/base 2025-12-04T08:27:07.1716396Z * [new branch] gh/kwen2501/130/head -> origin/gh/kwen2501/130/head 2025-12-04T08:27:07.1716743Z * [new branch] gh/kwen2501/130/orig -> origin/gh/kwen2501/130/orig 2025-12-04T08:27:07.1717094Z * [new branch] gh/kwen2501/170/base -> origin/gh/kwen2501/170/base 2025-12-04T08:27:07.1717442Z * [new branch] gh/kwen2501/170/head -> origin/gh/kwen2501/170/head 2025-12-04T08:27:07.1717784Z * [new branch] gh/kwen2501/187/base -> origin/gh/kwen2501/187/base 2025-12-04T08:27:07.1718224Z * [new branch] gh/kwen2501/187/head -> origin/gh/kwen2501/187/head 2025-12-04T08:27:07.1718596Z * [new branch] gh/kwen2501/187/orig -> origin/gh/kwen2501/187/orig 2025-12-04T08:27:07.1718960Z * [new branch] gh/kwen2501/188/base -> origin/gh/kwen2501/188/base 2025-12-04T08:27:07.1719341Z * [new branch] gh/kwen2501/188/head -> origin/gh/kwen2501/188/head 2025-12-04T08:27:07.1719694Z * [new branch] gh/kwen2501/188/orig -> origin/gh/kwen2501/188/orig 2025-12-04T08:27:07.1720056Z * [new branch] gh/kwen2501/211/base -> origin/gh/kwen2501/211/base 2025-12-04T08:27:07.1720423Z * [new branch] gh/kwen2501/211/head -> origin/gh/kwen2501/211/head 2025-12-04T08:27:07.1720957Z * [new branch] gh/kwen2501/224/base -> origin/gh/kwen2501/224/base 2025-12-04T08:27:07.1721308Z * [new branch] gh/kwen2501/224/head -> origin/gh/kwen2501/224/head 2025-12-04T08:27:07.1721655Z * [new branch] gh/kwen2501/224/orig -> origin/gh/kwen2501/224/orig 2025-12-04T08:27:07.1722004Z * [new branch] gh/kwen2501/228/base -> origin/gh/kwen2501/228/base 2025-12-04T08:27:07.1722343Z * [new branch] gh/kwen2501/228/head -> origin/gh/kwen2501/228/head 2025-12-04T08:27:07.1722697Z * [new branch] gh/kwen2501/228/orig -> origin/gh/kwen2501/228/orig 2025-12-04T08:27:07.1723044Z * [new branch] gh/kwen2501/234/base -> origin/gh/kwen2501/234/base 2025-12-04T08:27:07.1723391Z * [new branch] gh/kwen2501/234/head -> origin/gh/kwen2501/234/head 2025-12-04T08:27:07.1723818Z * [new branch] gh/kwen2501/234/orig -> origin/gh/kwen2501/234/orig 2025-12-04T08:27:07.1724166Z * [new branch] gh/kwen2501/235/base -> origin/gh/kwen2501/235/base 2025-12-04T08:27:07.1724514Z * [new branch] gh/kwen2501/235/head -> origin/gh/kwen2501/235/head 2025-12-04T08:27:07.1724859Z * [new branch] gh/kwen2501/235/orig -> origin/gh/kwen2501/235/orig 2025-12-04T08:27:07.1725200Z * [new branch] gh/kwen2501/236/base -> origin/gh/kwen2501/236/base 2025-12-04T08:27:07.1725614Z * [new branch] gh/kwen2501/236/head -> origin/gh/kwen2501/236/head 2025-12-04T08:27:07.1725959Z * [new branch] gh/kwen2501/236/orig -> origin/gh/kwen2501/236/orig 2025-12-04T08:27:07.1726293Z * [new branch] gh/kwen2501/237/base -> origin/gh/kwen2501/237/base 2025-12-04T08:27:07.1726619Z * [new branch] gh/kwen2501/237/head -> origin/gh/kwen2501/237/head 2025-12-04T08:27:07.1726949Z * [new branch] gh/kwen2501/237/orig -> origin/gh/kwen2501/237/orig 2025-12-04T08:27:07.1731907Z * [new branch] gh/kwen2501/238/base -> origin/gh/kwen2501/238/base 2025-12-04T08:27:07.1736090Z * [new branch] gh/kwen2501/238/head -> origin/gh/kwen2501/238/head 2025-12-04T08:27:07.1737874Z * [new branch] gh/kwen2501/238/orig -> origin/gh/kwen2501/238/orig 2025-12-04T08:27:07.1738310Z * [new branch] gh/kwen2501/240/base -> origin/gh/kwen2501/240/base 2025-12-04T08:27:07.1742740Z * [new branch] gh/kwen2501/240/head -> origin/gh/kwen2501/240/head 2025-12-04T08:27:07.1748424Z * [new branch] gh/kwen2501/240/orig -> origin/gh/kwen2501/240/orig 2025-12-04T08:27:07.1752413Z * [new branch] gh/kwen2501/241/base -> origin/gh/kwen2501/241/base 2025-12-04T08:27:07.1754339Z * [new branch] gh/kwen2501/241/head -> origin/gh/kwen2501/241/head 2025-12-04T08:27:07.1754716Z * [new branch] gh/kwen2501/241/orig -> origin/gh/kwen2501/241/orig 2025-12-04T08:27:07.1755076Z * [new branch] gh/kwen2501/247/base -> origin/gh/kwen2501/247/base 2025-12-04T08:27:07.1755443Z * [new branch] gh/kwen2501/247/head -> origin/gh/kwen2501/247/head 2025-12-04T08:27:07.1755802Z * [new branch] gh/kwen2501/247/orig -> origin/gh/kwen2501/247/orig 2025-12-04T08:27:07.1756156Z * [new branch] gh/kwen2501/252/base -> origin/gh/kwen2501/252/base 2025-12-04T08:27:07.1756521Z * [new branch] gh/kwen2501/252/head -> origin/gh/kwen2501/252/head 2025-12-04T08:27:07.1756886Z * [new branch] gh/kwen2501/252/orig -> origin/gh/kwen2501/252/orig 2025-12-04T08:27:07.1757251Z * [new branch] gh/kwen2501/259/base -> origin/gh/kwen2501/259/base 2025-12-04T08:27:07.1757616Z * [new branch] gh/kwen2501/259/head -> origin/gh/kwen2501/259/head 2025-12-04T08:27:07.1757963Z * [new branch] gh/kwen2501/259/orig -> origin/gh/kwen2501/259/orig 2025-12-04T08:27:07.1758589Z * [new branch] gh/kwen2501/260/base -> origin/gh/kwen2501/260/base 2025-12-04T08:27:07.1758962Z * [new branch] gh/kwen2501/260/head -> origin/gh/kwen2501/260/head 2025-12-04T08:27:07.1759383Z * [new branch] gh/kwen2501/260/orig -> origin/gh/kwen2501/260/orig 2025-12-04T08:27:07.1759735Z * [new branch] gh/kwen2501/268/base -> origin/gh/kwen2501/268/base 2025-12-04T08:27:07.1760093Z * [new branch] gh/kwen2501/268/head -> origin/gh/kwen2501/268/head 2025-12-04T08:27:07.1760447Z * [new branch] gh/kwen2501/268/orig -> origin/gh/kwen2501/268/orig 2025-12-04T08:27:07.1760846Z * [new branch] gh/kwen2501/269/base -> origin/gh/kwen2501/269/base 2025-12-04T08:27:07.1761396Z * [new branch] gh/kwen2501/269/head -> origin/gh/kwen2501/269/head 2025-12-04T08:27:07.1761758Z * [new branch] gh/kwen2501/269/orig -> origin/gh/kwen2501/269/orig 2025-12-04T08:27:07.1762115Z * [new branch] gh/kwen2501/270/base -> origin/gh/kwen2501/270/base 2025-12-04T08:27:07.1762461Z * [new branch] gh/kwen2501/270/head -> origin/gh/kwen2501/270/head 2025-12-04T08:27:07.1762825Z * [new branch] gh/kwen2501/270/orig -> origin/gh/kwen2501/270/orig 2025-12-04T08:27:07.1763190Z * [new branch] gh/kwen2501/271/base -> origin/gh/kwen2501/271/base 2025-12-04T08:27:07.1763551Z * [new branch] gh/kwen2501/271/head -> origin/gh/kwen2501/271/head 2025-12-04T08:27:07.1763947Z * [new branch] gh/kwen2501/271/orig -> origin/gh/kwen2501/271/orig 2025-12-04T08:27:07.1764362Z * [new branch] gh/kwen2501/274/base -> origin/gh/kwen2501/274/base 2025-12-04T08:27:07.1764709Z * [new branch] gh/kwen2501/274/head -> origin/gh/kwen2501/274/head 2025-12-04T08:27:07.1765061Z * [new branch] gh/kwen2501/274/orig -> origin/gh/kwen2501/274/orig 2025-12-04T08:27:07.1765446Z * [new branch] gh/kwen2501/275/base -> origin/gh/kwen2501/275/base 2025-12-04T08:27:07.1765818Z * [new branch] gh/kwen2501/275/head -> origin/gh/kwen2501/275/head 2025-12-04T08:27:07.1766176Z * [new branch] gh/kwen2501/275/orig -> origin/gh/kwen2501/275/orig 2025-12-04T08:27:07.1766526Z * [new branch] gh/kwen2501/276/base -> origin/gh/kwen2501/276/base 2025-12-04T08:27:07.1766935Z * [new branch] gh/kwen2501/276/head -> origin/gh/kwen2501/276/head 2025-12-04T08:27:07.1767286Z * [new branch] gh/kwen2501/276/orig -> origin/gh/kwen2501/276/orig 2025-12-04T08:27:07.1767668Z * [new branch] gh/kwen2501/277/base -> origin/gh/kwen2501/277/base 2025-12-04T08:27:07.1768023Z * [new branch] gh/kwen2501/277/head -> origin/gh/kwen2501/277/head 2025-12-04T08:27:07.1768383Z * [new branch] gh/kwen2501/277/orig -> origin/gh/kwen2501/277/orig 2025-12-04T08:27:07.1768727Z * [new branch] gh/kwen2501/278/base -> origin/gh/kwen2501/278/base 2025-12-04T08:27:07.1769075Z * [new branch] gh/kwen2501/278/head -> origin/gh/kwen2501/278/head 2025-12-04T08:27:07.1769415Z * [new branch] gh/kwen2501/278/orig -> origin/gh/kwen2501/278/orig 2025-12-04T08:27:07.1769756Z * [new branch] gh/kwen2501/279/base -> origin/gh/kwen2501/279/base 2025-12-04T08:27:07.1770066Z * [new branch] gh/kwen2501/279/head -> origin/gh/kwen2501/279/head 2025-12-04T08:27:07.1770376Z * [new branch] gh/kwen2501/279/orig -> origin/gh/kwen2501/279/orig 2025-12-04T08:27:07.1770681Z * [new branch] gh/kwen2501/280/base -> origin/gh/kwen2501/280/base 2025-12-04T08:27:07.1770993Z * [new branch] gh/kwen2501/280/head -> origin/gh/kwen2501/280/head 2025-12-04T08:27:07.1771309Z * [new branch] gh/kwen2501/280/orig -> origin/gh/kwen2501/280/orig 2025-12-04T08:27:07.1771624Z * [new branch] gh/kwen2501/281/base -> origin/gh/kwen2501/281/base 2025-12-04T08:27:07.1771933Z * [new branch] gh/kwen2501/281/head -> origin/gh/kwen2501/281/head 2025-12-04T08:27:07.1772249Z * [new branch] gh/kwen2501/281/orig -> origin/gh/kwen2501/281/orig 2025-12-04T08:27:07.1772563Z * [new branch] gh/kwen2501/282/base -> origin/gh/kwen2501/282/base 2025-12-04T08:27:07.1772869Z * [new branch] gh/kwen2501/282/head -> origin/gh/kwen2501/282/head 2025-12-04T08:27:07.1773183Z * [new branch] gh/kwen2501/282/orig -> origin/gh/kwen2501/282/orig 2025-12-04T08:27:07.1778548Z * [new branch] gh/kwen2501/283/base -> origin/gh/kwen2501/283/base 2025-12-04T08:27:07.1781321Z * [new branch] gh/kwen2501/283/head -> origin/gh/kwen2501/283/head 2025-12-04T08:27:07.1781718Z * [new branch] gh/kwen2501/283/orig -> origin/gh/kwen2501/283/orig 2025-12-04T08:27:07.1782042Z * [new branch] gh/kwen2501/284/base -> origin/gh/kwen2501/284/base 2025-12-04T08:27:07.1782398Z * [new branch] gh/kwen2501/284/head -> origin/gh/kwen2501/284/head 2025-12-04T08:27:07.1782728Z * [new branch] gh/kwen2501/284/orig -> origin/gh/kwen2501/284/orig 2025-12-04T08:27:07.1783073Z * [new branch] gh/kwen2501/285/base -> origin/gh/kwen2501/285/base 2025-12-04T08:27:07.1783392Z * [new branch] gh/kwen2501/285/head -> origin/gh/kwen2501/285/head 2025-12-04T08:27:07.1783738Z * [new branch] gh/kwen2501/285/orig -> origin/gh/kwen2501/285/orig 2025-12-04T08:27:07.1784054Z * [new branch] gh/kwen2501/286/base -> origin/gh/kwen2501/286/base 2025-12-04T08:27:07.1784388Z * [new branch] gh/kwen2501/286/head -> origin/gh/kwen2501/286/head 2025-12-04T08:27:07.1784706Z * [new branch] gh/kwen2501/286/orig -> origin/gh/kwen2501/286/orig 2025-12-04T08:27:07.1785028Z * [new branch] gh/kwen2501/287/base -> origin/gh/kwen2501/287/base 2025-12-04T08:27:07.1785363Z * [new branch] gh/kwen2501/287/head -> origin/gh/kwen2501/287/head 2025-12-04T08:27:07.1785735Z * [new branch] gh/kwen2501/287/orig -> origin/gh/kwen2501/287/orig 2025-12-04T08:27:07.1786309Z * [new branch] gh/kwen2501/288/base -> origin/gh/kwen2501/288/base 2025-12-04T08:27:07.1786788Z * [new branch] gh/kwen2501/288/head -> origin/gh/kwen2501/288/head 2025-12-04T08:27:07.1787247Z * [new branch] gh/kwen2501/288/orig -> origin/gh/kwen2501/288/orig 2025-12-04T08:27:07.1787604Z * [new branch] gh/laithsakka/251/base -> origin/gh/laithsakka/251/base 2025-12-04T08:27:07.1787973Z * [new branch] gh/laithsakka/251/head -> origin/gh/laithsakka/251/head 2025-12-04T08:27:07.1788326Z * [new branch] gh/laithsakka/251/orig -> origin/gh/laithsakka/251/orig 2025-12-04T08:27:07.1788671Z * [new branch] gh/laithsakka/276/base -> origin/gh/laithsakka/276/base 2025-12-04T08:27:07.1789172Z * [new branch] gh/laithsakka/276/head -> origin/gh/laithsakka/276/head 2025-12-04T08:27:07.1789633Z * [new branch] gh/laithsakka/276/orig -> origin/gh/laithsakka/276/orig 2025-12-04T08:27:07.1792011Z * [new branch] gh/laithsakka/28/base -> origin/gh/laithsakka/28/base 2025-12-04T08:27:07.1792572Z * [new branch] gh/laithsakka/29/base -> origin/gh/laithsakka/29/base 2025-12-04T08:27:07.1793074Z * [new branch] gh/laithsakka/30/base -> origin/gh/laithsakka/30/base 2025-12-04T08:27:07.1793563Z * [new branch] gh/laithsakka/30/head -> origin/gh/laithsakka/30/head 2025-12-04T08:27:07.1796823Z * [new branch] gh/laithsakka/31/base -> origin/gh/laithsakka/31/base 2025-12-04T08:27:07.1797185Z * [new branch] gh/laithsakka/31/head -> origin/gh/laithsakka/31/head 2025-12-04T08:27:07.1797549Z * [new branch] gh/laithsakka/313/base -> origin/gh/laithsakka/313/base 2025-12-04T08:27:07.1797900Z * [new branch] gh/laithsakka/313/head -> origin/gh/laithsakka/313/head 2025-12-04T08:27:07.1798337Z * [new branch] gh/laithsakka/313/orig -> origin/gh/laithsakka/313/orig 2025-12-04T08:27:07.1798694Z * [new branch] gh/laithsakka/316/base -> origin/gh/laithsakka/316/base 2025-12-04T08:27:07.1799208Z * [new branch] gh/laithsakka/316/head -> origin/gh/laithsakka/316/head 2025-12-04T08:27:07.1799598Z * [new branch] gh/laithsakka/316/orig -> origin/gh/laithsakka/316/orig 2025-12-04T08:27:07.1799976Z * [new branch] gh/laithsakka/317/base -> origin/gh/laithsakka/317/base 2025-12-04T08:27:07.1800325Z * [new branch] gh/laithsakka/317/head -> origin/gh/laithsakka/317/head 2025-12-04T08:27:07.1800872Z * [new branch] gh/laithsakka/317/orig -> origin/gh/laithsakka/317/orig 2025-12-04T08:27:07.1801233Z * [new branch] gh/laithsakka/319/base -> origin/gh/laithsakka/319/base 2025-12-04T08:27:07.1801835Z * [new branch] gh/laithsakka/319/head -> origin/gh/laithsakka/319/head 2025-12-04T08:27:07.1805658Z * [new branch] gh/laithsakka/319/orig -> origin/gh/laithsakka/319/orig 2025-12-04T08:27:07.1806244Z * [new branch] gh/laithsakka/32/base -> origin/gh/laithsakka/32/base 2025-12-04T08:27:07.1806729Z * [new branch] gh/laithsakka/32/head -> origin/gh/laithsakka/32/head 2025-12-04T08:27:07.1807100Z * [new branch] gh/laithsakka/320/base -> origin/gh/laithsakka/320/base 2025-12-04T08:27:07.1807474Z * [new branch] gh/laithsakka/320/head -> origin/gh/laithsakka/320/head 2025-12-04T08:27:07.1807826Z * [new branch] gh/laithsakka/320/orig -> origin/gh/laithsakka/320/orig 2025-12-04T08:27:07.1808166Z * [new branch] gh/laithsakka/321/base -> origin/gh/laithsakka/321/base 2025-12-04T08:27:07.1808520Z * [new branch] gh/laithsakka/321/head -> origin/gh/laithsakka/321/head 2025-12-04T08:27:07.1809006Z * [new branch] gh/laithsakka/321/orig -> origin/gh/laithsakka/321/orig 2025-12-04T08:27:07.1809385Z * [new branch] gh/laithsakka/322/base -> origin/gh/laithsakka/322/base 2025-12-04T08:27:07.1810195Z * [new branch] gh/laithsakka/322/head -> origin/gh/laithsakka/322/head 2025-12-04T08:27:07.1810807Z * [new branch] gh/laithsakka/322/orig -> origin/gh/laithsakka/322/orig 2025-12-04T08:27:07.1811825Z * [new branch] gh/laithsakka/323/base -> origin/gh/laithsakka/323/base 2025-12-04T08:27:07.1812428Z * [new branch] gh/laithsakka/323/head -> origin/gh/laithsakka/323/head 2025-12-04T08:27:07.1813139Z * [new branch] gh/laithsakka/323/orig -> origin/gh/laithsakka/323/orig 2025-12-04T08:27:07.1814189Z * [new branch] gh/laithsakka/324/base -> origin/gh/laithsakka/324/base 2025-12-04T08:27:07.1814962Z * [new branch] gh/laithsakka/324/head -> origin/gh/laithsakka/324/head 2025-12-04T08:27:07.1815546Z * [new branch] gh/laithsakka/324/orig -> origin/gh/laithsakka/324/orig 2025-12-04T08:27:07.1818185Z * [new branch] gh/laithsakka/325/base -> origin/gh/laithsakka/325/base 2025-12-04T08:27:07.1818626Z * [new branch] gh/laithsakka/325/head -> origin/gh/laithsakka/325/head 2025-12-04T08:27:07.1818974Z * [new branch] gh/laithsakka/325/orig -> origin/gh/laithsakka/325/orig 2025-12-04T08:27:07.1819342Z * [new branch] gh/laithsakka/326/base -> origin/gh/laithsakka/326/base 2025-12-04T08:27:07.1819730Z * [new branch] gh/laithsakka/326/head -> origin/gh/laithsakka/326/head 2025-12-04T08:27:07.1820457Z * [new branch] gh/laithsakka/326/orig -> origin/gh/laithsakka/326/orig 2025-12-04T08:27:07.1826024Z * [new branch] gh/laithsakka/327/base -> origin/gh/laithsakka/327/base 2025-12-04T08:27:07.1826441Z * [new branch] gh/laithsakka/327/head -> origin/gh/laithsakka/327/head 2025-12-04T08:27:07.1826808Z * [new branch] gh/laithsakka/327/orig -> origin/gh/laithsakka/327/orig 2025-12-04T08:27:07.1831170Z * [new branch] gh/laithsakka/328/base -> origin/gh/laithsakka/328/base 2025-12-04T08:27:07.1831789Z * [new branch] gh/laithsakka/328/head -> origin/gh/laithsakka/328/head 2025-12-04T08:27:07.1832131Z * [new branch] gh/laithsakka/328/orig -> origin/gh/laithsakka/328/orig 2025-12-04T08:27:07.1832478Z * [new branch] gh/liangel/4/base -> origin/gh/liangel/4/base 2025-12-04T08:27:07.1832808Z * [new branch] gh/liangel/4/head -> origin/gh/liangel/4/head 2025-12-04T08:27:07.1833125Z * [new branch] gh/liangel/4/orig -> origin/gh/liangel/4/orig 2025-12-04T08:27:07.1834654Z * [new branch] gh/lucaskabela/1/base -> origin/gh/lucaskabela/1/base 2025-12-04T08:27:07.1835021Z * [new branch] gh/lucaskabela/1/head -> origin/gh/lucaskabela/1/head 2025-12-04T08:27:07.1835365Z * [new branch] gh/lw/4/base -> origin/gh/lw/4/base 2025-12-04T08:27:07.1835663Z * [new branch] gh/lw/4/head -> origin/gh/lw/4/head 2025-12-04T08:27:07.1835952Z * [new branch] gh/lw/4/orig -> origin/gh/lw/4/orig 2025-12-04T08:27:07.1836506Z * [new branch] gh/lw/5/base -> origin/gh/lw/5/base 2025-12-04T08:27:07.1837163Z * [new branch] gh/lw/5/head -> origin/gh/lw/5/head 2025-12-04T08:27:07.1837795Z * [new branch] gh/lw/5/orig -> origin/gh/lw/5/orig 2025-12-04T08:27:07.1843902Z * [new branch] gh/lw/6/base -> origin/gh/lw/6/base 2025-12-04T08:27:07.1844449Z * [new branch] gh/lw/6/head -> origin/gh/lw/6/head 2025-12-04T08:27:07.1844884Z * [new branch] gh/lw/6/orig -> origin/gh/lw/6/orig 2025-12-04T08:27:07.1845836Z * [new branch] gh/malfet/14/base -> origin/gh/malfet/14/base 2025-12-04T08:27:07.1846257Z * [new branch] gh/malfet/417/base -> origin/gh/malfet/417/base 2025-12-04T08:27:07.1846599Z * [new branch] gh/malfet/417/head -> origin/gh/malfet/417/head 2025-12-04T08:27:07.1846927Z * [new branch] gh/malfet/417/orig -> origin/gh/malfet/417/orig 2025-12-04T08:27:07.1847259Z * [new branch] gh/malfet/506/base -> origin/gh/malfet/506/base 2025-12-04T08:27:07.1847582Z * [new branch] gh/malfet/506/head -> origin/gh/malfet/506/head 2025-12-04T08:27:07.1847901Z * [new branch] gh/malfet/506/orig -> origin/gh/malfet/506/orig 2025-12-04T08:27:07.1848228Z * [new branch] gh/malfet/517/base -> origin/gh/malfet/517/base 2025-12-04T08:27:07.1848544Z * [new branch] gh/malfet/517/head -> origin/gh/malfet/517/head 2025-12-04T08:27:07.1848854Z * [new branch] gh/malfet/528/base -> origin/gh/malfet/528/base 2025-12-04T08:27:07.1849357Z * [new branch] gh/malfet/528/head -> origin/gh/malfet/528/head 2025-12-04T08:27:07.1849516Z * [new branch] gh/malfet/528/orig -> origin/gh/malfet/528/orig 2025-12-04T08:27:07.1854048Z * [new branch] gh/malfet/537/base -> origin/gh/malfet/537/base 2025-12-04T08:27:07.1854393Z * [new branch] gh/malfet/537/head -> origin/gh/malfet/537/head 2025-12-04T08:27:07.1854666Z * [new branch] gh/malfet/537/orig -> origin/gh/malfet/537/orig 2025-12-04T08:27:07.1854958Z * [new branch] gh/malfet/546/base -> origin/gh/malfet/546/base 2025-12-04T08:27:07.1855153Z * [new branch] gh/malfet/546/head -> origin/gh/malfet/546/head 2025-12-04T08:27:07.1855814Z * [new branch] gh/malfet/546/orig -> origin/gh/malfet/546/orig 2025-12-04T08:27:07.1856150Z * [new branch] gh/malfet/565/base -> origin/gh/malfet/565/base 2025-12-04T08:27:07.1856411Z * [new branch] gh/malfet/565/head -> origin/gh/malfet/565/head 2025-12-04T08:27:07.1856836Z * [new branch] gh/malfet/565/orig -> origin/gh/malfet/565/orig 2025-12-04T08:27:07.1857012Z * [new branch] gh/malfet/575/base -> origin/gh/malfet/575/base 2025-12-04T08:27:07.1858257Z * [new branch] gh/malfet/575/head -> origin/gh/malfet/575/head 2025-12-04T08:27:07.1858546Z * [new branch] gh/malfet/575/orig -> origin/gh/malfet/575/orig 2025-12-04T08:27:07.1861268Z * [new branch] gh/malfet/580/base -> origin/gh/malfet/580/base 2025-12-04T08:27:07.1863123Z * [new branch] gh/malfet/580/head -> origin/gh/malfet/580/head 2025-12-04T08:27:07.1863313Z * [new branch] gh/malfet/580/orig -> origin/gh/malfet/580/orig 2025-12-04T08:27:07.1863449Z * [new branch] gh/malfet/581/base -> origin/gh/malfet/581/base 2025-12-04T08:27:07.1863589Z * [new branch] gh/malfet/581/head -> origin/gh/malfet/581/head 2025-12-04T08:27:07.1863733Z * [new branch] gh/malfet/581/orig -> origin/gh/malfet/581/orig 2025-12-04T08:27:07.1868536Z * [new branch] gh/malfet/583/base -> origin/gh/malfet/583/base 2025-12-04T08:27:07.1868711Z * [new branch] gh/malfet/583/head -> origin/gh/malfet/583/head 2025-12-04T08:27:07.1868848Z * [new branch] gh/malfet/583/orig -> origin/gh/malfet/583/orig 2025-12-04T08:27:07.1868988Z * [new branch] gh/malfet/586/base -> origin/gh/malfet/586/base 2025-12-04T08:27:07.1869116Z * [new branch] gh/malfet/586/head -> origin/gh/malfet/586/head 2025-12-04T08:27:07.1869398Z * [new branch] gh/malfet/586/orig -> origin/gh/malfet/586/orig 2025-12-04T08:27:07.1874802Z * [new branch] gh/malfet/587/base -> origin/gh/malfet/587/base 2025-12-04T08:27:07.1875015Z * [new branch] gh/malfet/587/head -> origin/gh/malfet/587/head 2025-12-04T08:27:07.1875390Z * [new branch] gh/malfet/587/orig -> origin/gh/malfet/587/orig 2025-12-04T08:27:07.1875564Z * [new branch] gh/malfet/588/base -> origin/gh/malfet/588/base 2025-12-04T08:27:07.1875703Z * [new branch] gh/malfet/588/head -> origin/gh/malfet/588/head 2025-12-04T08:27:07.1875852Z * [new branch] gh/malfet/588/orig -> origin/gh/malfet/588/orig 2025-12-04T08:27:07.1875986Z * [new branch] gh/malfet/589/base -> origin/gh/malfet/589/base 2025-12-04T08:27:07.1876124Z * [new branch] gh/malfet/589/head -> origin/gh/malfet/589/head 2025-12-04T08:27:07.1876270Z * [new branch] gh/malfet/589/orig -> origin/gh/malfet/589/orig 2025-12-04T08:27:07.1876418Z * [new branch] gh/malfet/590/base -> origin/gh/malfet/590/base 2025-12-04T08:27:07.1876570Z * [new branch] gh/malfet/590/head -> origin/gh/malfet/590/head 2025-12-04T08:27:07.1876737Z * [new branch] gh/malfet/590/orig -> origin/gh/malfet/590/orig 2025-12-04T08:27:07.1876891Z * [new branch] gh/malfet/591/base -> origin/gh/malfet/591/base 2025-12-04T08:27:07.1877035Z * [new branch] gh/malfet/591/head -> origin/gh/malfet/591/head 2025-12-04T08:27:07.1877185Z * [new branch] gh/malfet/591/orig -> origin/gh/malfet/591/orig 2025-12-04T08:27:07.1878167Z * [new branch] gh/malfet/592/base -> origin/gh/malfet/592/base 2025-12-04T08:27:07.1878609Z * [new branch] gh/malfet/592/head -> origin/gh/malfet/592/head 2025-12-04T08:27:07.1882731Z * [new branch] gh/malfet/592/orig -> origin/gh/malfet/592/orig 2025-12-04T08:27:07.1882920Z * [new branch] gh/malfet/593/base -> origin/gh/malfet/593/base 2025-12-04T08:27:07.1883239Z * [new branch] gh/malfet/593/head -> origin/gh/malfet/593/head 2025-12-04T08:27:07.1883384Z * [new branch] gh/malfet/593/orig -> origin/gh/malfet/593/orig 2025-12-04T08:27:07.1883525Z * [new branch] gh/malfet/594/base -> origin/gh/malfet/594/base 2025-12-04T08:27:07.1888324Z * [new branch] gh/malfet/594/head -> origin/gh/malfet/594/head 2025-12-04T08:27:07.1888501Z * [new branch] gh/malfet/594/orig -> origin/gh/malfet/594/orig 2025-12-04T08:27:07.1888655Z * [new branch] gh/malfet/595/base -> origin/gh/malfet/595/base 2025-12-04T08:27:07.1888794Z * [new branch] gh/malfet/595/head -> origin/gh/malfet/595/head 2025-12-04T08:27:07.1888953Z * [new branch] gh/malfet/595/orig -> origin/gh/malfet/595/orig 2025-12-04T08:27:07.1889839Z * [new branch] gh/malfet/596/base -> origin/gh/malfet/596/base 2025-12-04T08:27:07.1889993Z * [new branch] gh/malfet/596/head -> origin/gh/malfet/596/head 2025-12-04T08:27:07.1890143Z * [new branch] gh/malfet/596/orig -> origin/gh/malfet/596/orig 2025-12-04T08:27:07.1890284Z * [new branch] gh/malfet/597/base -> origin/gh/malfet/597/base 2025-12-04T08:27:07.1890443Z * [new branch] gh/malfet/597/head -> origin/gh/malfet/597/head 2025-12-04T08:27:07.1890578Z * [new branch] gh/malfet/597/orig -> origin/gh/malfet/597/orig 2025-12-04T08:27:07.1895664Z * [new branch] gh/malfet/598/base -> origin/gh/malfet/598/base 2025-12-04T08:27:07.1895841Z * [new branch] gh/malfet/598/head -> origin/gh/malfet/598/head 2025-12-04T08:27:07.1896123Z * [new branch] gh/malfet/598/orig -> origin/gh/malfet/598/orig 2025-12-04T08:27:07.1896267Z * [new branch] gh/malfet/599/base -> origin/gh/malfet/599/base 2025-12-04T08:27:07.1902344Z * [new branch] gh/malfet/599/head -> origin/gh/malfet/599/head 2025-12-04T08:27:07.1902695Z * [new branch] gh/malfet/599/orig -> origin/gh/malfet/599/orig 2025-12-04T08:27:07.1902963Z * [new branch] gh/malfet/600/base -> origin/gh/malfet/600/base 2025-12-04T08:27:07.1903211Z * [new branch] gh/malfet/600/head -> origin/gh/malfet/600/head 2025-12-04T08:27:07.1903471Z * [new branch] gh/malfet/600/orig -> origin/gh/malfet/600/orig 2025-12-04T08:27:07.1904194Z * [new branch] gh/malfet/601/base -> origin/gh/malfet/601/base 2025-12-04T08:27:07.1904535Z * [new branch] gh/malfet/601/head -> origin/gh/malfet/601/head 2025-12-04T08:27:07.1904718Z * [new branch] gh/malfet/601/orig -> origin/gh/malfet/601/orig 2025-12-04T08:27:07.1904886Z * [new branch] gh/malfet/602/base -> origin/gh/malfet/602/base 2025-12-04T08:27:07.1905046Z * [new branch] gh/malfet/602/head -> origin/gh/malfet/602/head 2025-12-04T08:27:07.1905201Z * [new branch] gh/malfet/602/orig -> origin/gh/malfet/602/orig 2025-12-04T08:27:07.1905349Z * [new branch] gh/malfet/603/base -> origin/gh/malfet/603/base 2025-12-04T08:27:07.1905504Z * [new branch] gh/malfet/603/head -> origin/gh/malfet/603/head 2025-12-04T08:27:07.1905653Z * [new branch] gh/malfet/603/orig -> origin/gh/malfet/603/orig 2025-12-04T08:27:07.1905801Z * [new branch] gh/malfet/604/base -> origin/gh/malfet/604/base 2025-12-04T08:27:07.1905953Z * [new branch] gh/malfet/604/head -> origin/gh/malfet/604/head 2025-12-04T08:27:07.1906111Z * [new branch] gh/malfet/604/orig -> origin/gh/malfet/604/orig 2025-12-04T08:27:07.1906260Z * [new branch] gh/malfet/605/base -> origin/gh/malfet/605/base 2025-12-04T08:27:07.1906964Z * [new branch] gh/malfet/605/head -> origin/gh/malfet/605/head 2025-12-04T08:27:07.1907412Z * [new branch] gh/malfet/605/orig -> origin/gh/malfet/605/orig 2025-12-04T08:27:07.1912577Z * [new branch] gh/malfet/606/base -> origin/gh/malfet/606/base 2025-12-04T08:27:07.1912767Z * [new branch] gh/malfet/606/head -> origin/gh/malfet/606/head 2025-12-04T08:27:07.1912903Z * [new branch] gh/malfet/606/orig -> origin/gh/malfet/606/orig 2025-12-04T08:27:07.1913045Z * [new branch] gh/malfet/607/base -> origin/gh/malfet/607/base 2025-12-04T08:27:07.1913194Z * [new branch] gh/malfet/607/head -> origin/gh/malfet/607/head 2025-12-04T08:27:07.1913326Z * [new branch] gh/malfet/607/orig -> origin/gh/malfet/607/orig 2025-12-04T08:27:07.1913469Z * [new branch] gh/malfet/608/base -> origin/gh/malfet/608/base 2025-12-04T08:27:07.1913787Z * [new branch] gh/malfet/608/head -> origin/gh/malfet/608/head 2025-12-04T08:27:07.1914279Z * [new branch] gh/malfet/608/orig -> origin/gh/malfet/608/orig 2025-12-04T08:27:07.1915656Z * [new branch] gh/malfet/609/base -> origin/gh/malfet/609/base 2025-12-04T08:27:07.1916420Z * [new branch] gh/malfet/609/head -> origin/gh/malfet/609/head 2025-12-04T08:27:07.1916900Z * [new branch] gh/malfet/609/orig -> origin/gh/malfet/609/orig 2025-12-04T08:27:07.1918491Z * [new branch] gh/malfet/610/base -> origin/gh/malfet/610/base 2025-12-04T08:27:07.1919298Z * [new branch] gh/malfet/610/head -> origin/gh/malfet/610/head 2025-12-04T08:27:07.1919807Z * [new branch] gh/malfet/610/orig -> origin/gh/malfet/610/orig 2025-12-04T08:27:07.1923283Z * [new branch] gh/malfet/611/base -> origin/gh/malfet/611/base 2025-12-04T08:27:07.1923651Z * [new branch] gh/malfet/611/head -> origin/gh/malfet/611/head 2025-12-04T08:27:07.1923919Z * [new branch] gh/malfet/611/orig -> origin/gh/malfet/611/orig 2025-12-04T08:27:07.1924178Z * [new branch] gh/malfet/612/base -> origin/gh/malfet/612/base 2025-12-04T08:27:07.1924346Z * [new branch] gh/malfet/612/head -> origin/gh/malfet/612/head 2025-12-04T08:27:07.1924621Z * [new branch] gh/malfet/612/orig -> origin/gh/malfet/612/orig 2025-12-04T08:27:07.1928845Z * [new branch] gh/malfet/64/base -> origin/gh/malfet/64/base 2025-12-04T08:27:07.1929212Z * [new branch] gh/malfet/64/head -> origin/gh/malfet/64/head 2025-12-04T08:27:07.1929501Z * [new branch] gh/manuelcandales/11/base -> origin/gh/manuelcandales/11/base 2025-12-04T08:27:07.1929782Z * [new branch] gh/manuelcandales/11/head -> origin/gh/manuelcandales/11/head 2025-12-04T08:27:07.1930073Z * [new branch] gh/manuelcandales/11/orig -> origin/gh/manuelcandales/11/orig 2025-12-04T08:27:07.1930683Z * [new branch] gh/markkm/1/base -> origin/gh/markkm/1/base 2025-12-04T08:27:07.1931038Z * [new branch] gh/masnesral/1/base -> origin/gh/masnesral/1/base 2025-12-04T08:27:07.1931413Z * [new branch] gh/masnesral/1/head -> origin/gh/masnesral/1/head 2025-12-04T08:27:07.1933055Z * [new branch] gh/masnesral/1/orig -> origin/gh/masnesral/1/orig 2025-12-04T08:27:07.1933427Z * [new branch] gh/mhorowitz/0/base -> origin/gh/mhorowitz/0/base 2025-12-04T08:27:07.1933771Z * [new branch] gh/mhorowitz/0/head -> origin/gh/mhorowitz/0/head 2025-12-04T08:27:07.1935313Z * [new branch] gh/mhorowitz/1/base -> origin/gh/mhorowitz/1/base 2025-12-04T08:27:07.1935693Z * [new branch] gh/mhorowitz/1/head -> origin/gh/mhorowitz/1/head 2025-12-04T08:27:07.1935984Z * [new branch] gh/mhorowitz/2/base -> origin/gh/mhorowitz/2/base 2025-12-04T08:27:07.1938428Z * [new branch] gh/mhorowitz/2/head -> origin/gh/mhorowitz/2/head 2025-12-04T08:27:07.1938755Z * [new branch] gh/mhorowitz/3/base -> origin/gh/mhorowitz/3/base 2025-12-04T08:27:07.1939015Z * [new branch] gh/mhorowitz/3/head -> origin/gh/mhorowitz/3/head 2025-12-04T08:27:07.1939246Z * [new branch] gh/mhorowitz/4/base -> origin/gh/mhorowitz/4/base 2025-12-04T08:27:07.1939484Z * [new branch] gh/mhorowitz/4/head -> origin/gh/mhorowitz/4/head 2025-12-04T08:27:07.1945011Z * [new branch] gh/mhorowitz/5/base -> origin/gh/mhorowitz/5/base 2025-12-04T08:27:07.1945357Z * [new branch] gh/mhorowitz/5/head -> origin/gh/mhorowitz/5/head 2025-12-04T08:27:07.1945609Z * [new branch] gh/mhorowitz/6/base -> origin/gh/mhorowitz/6/base 2025-12-04T08:27:07.1945887Z * [new branch] gh/mhorowitz/6/head -> origin/gh/mhorowitz/6/head 2025-12-04T08:27:07.1946202Z * [new branch] gh/mikaylagawarecki/234/base -> origin/gh/mikaylagawarecki/234/base 2025-12-04T08:27:07.1946972Z * [new branch] gh/mikaylagawarecki/234/head -> origin/gh/mikaylagawarecki/234/head 2025-12-04T08:27:07.1947191Z * [new branch] gh/mikaylagawarecki/235/base -> origin/gh/mikaylagawarecki/235/base 2025-12-04T08:27:07.1947380Z * [new branch] gh/mikaylagawarecki/235/head -> origin/gh/mikaylagawarecki/235/head 2025-12-04T08:27:07.1947908Z * [new branch] gh/mikaylagawarecki/236/base -> origin/gh/mikaylagawarecki/236/base 2025-12-04T08:27:07.1948121Z * [new branch] gh/mikaylagawarecki/236/head -> origin/gh/mikaylagawarecki/236/head 2025-12-04T08:27:07.1948477Z * [new branch] gh/mikaylagawarecki/237/base -> origin/gh/mikaylagawarecki/237/base 2025-12-04T08:27:07.1951843Z * [new branch] gh/mikaylagawarecki/237/head -> origin/gh/mikaylagawarecki/237/head 2025-12-04T08:27:07.1952058Z * [new branch] gh/mikaylagawarecki/238/base -> origin/gh/mikaylagawarecki/238/base 2025-12-04T08:27:07.1952225Z * [new branch] gh/mikaylagawarecki/238/head -> origin/gh/mikaylagawarecki/238/head 2025-12-04T08:27:07.1952396Z * [new branch] gh/mikaylagawarecki/336/base -> origin/gh/mikaylagawarecki/336/base 2025-12-04T08:27:07.1952621Z * [new branch] gh/mikaylagawarecki/336/head -> origin/gh/mikaylagawarecki/336/head 2025-12-04T08:27:07.1953269Z * [new branch] gh/mikaylagawarecki/336/orig -> origin/gh/mikaylagawarecki/336/orig 2025-12-04T08:27:07.1954639Z * [new branch] gh/mikaylagawarecki/341/base -> origin/gh/mikaylagawarecki/341/base 2025-12-04T08:27:07.1955629Z * [new branch] gh/mikaylagawarecki/341/head -> origin/gh/mikaylagawarecki/341/head 2025-12-04T08:27:07.1956065Z * [new branch] gh/mikaylagawarecki/341/orig -> origin/gh/mikaylagawarecki/341/orig 2025-12-04T08:27:07.1957472Z * [new branch] gh/mikaylagawarecki/342/base -> origin/gh/mikaylagawarecki/342/base 2025-12-04T08:27:07.1957802Z * [new branch] gh/mikaylagawarecki/342/head -> origin/gh/mikaylagawarecki/342/head 2025-12-04T08:27:07.1962567Z * [new branch] gh/mikaylagawarecki/342/orig -> origin/gh/mikaylagawarecki/342/orig 2025-12-04T08:27:07.1962787Z * [new branch] gh/mikaylagawarecki/345/base -> origin/gh/mikaylagawarecki/345/base 2025-12-04T08:27:07.1963290Z * [new branch] gh/mikaylagawarecki/345/head -> origin/gh/mikaylagawarecki/345/head 2025-12-04T08:27:07.1963495Z * [new branch] gh/mikaylagawarecki/345/orig -> origin/gh/mikaylagawarecki/345/orig 2025-12-04T08:27:07.1963678Z * [new branch] gh/mikaylagawarecki/346/base -> origin/gh/mikaylagawarecki/346/base 2025-12-04T08:27:07.1964009Z * [new branch] gh/mikaylagawarecki/346/head -> origin/gh/mikaylagawarecki/346/head 2025-12-04T08:27:07.1964200Z * [new branch] gh/mikaylagawarecki/346/orig -> origin/gh/mikaylagawarecki/346/orig 2025-12-04T08:27:07.1964768Z * [new branch] gh/mikaylagawarecki/347/base -> origin/gh/mikaylagawarecki/347/base 2025-12-04T08:27:07.1967722Z * [new branch] gh/mikaylagawarecki/347/head -> origin/gh/mikaylagawarecki/347/head 2025-12-04T08:27:07.1968118Z * [new branch] gh/mikaylagawarecki/347/orig -> origin/gh/mikaylagawarecki/347/orig 2025-12-04T08:27:07.1968390Z * [new branch] gh/mikaylagawarecki/350/base -> origin/gh/mikaylagawarecki/350/base 2025-12-04T08:27:07.1968723Z * [new branch] gh/mikaylagawarecki/350/head -> origin/gh/mikaylagawarecki/350/head 2025-12-04T08:27:07.1969172Z * [new branch] gh/mikaylagawarecki/350/orig -> origin/gh/mikaylagawarecki/350/orig 2025-12-04T08:27:07.1969853Z * [new branch] gh/mikaylagawarecki/351/base -> origin/gh/mikaylagawarecki/351/base 2025-12-04T08:27:07.1972769Z * [new branch] gh/mikaylagawarecki/351/head -> origin/gh/mikaylagawarecki/351/head 2025-12-04T08:27:07.1973156Z * [new branch] gh/mikaylagawarecki/351/orig -> origin/gh/mikaylagawarecki/351/orig 2025-12-04T08:27:07.1973504Z * [new branch] gh/mikaylagawarecki/352/base -> origin/gh/mikaylagawarecki/352/base 2025-12-04T08:27:07.1973821Z * [new branch] gh/mikaylagawarecki/352/head -> origin/gh/mikaylagawarecki/352/head 2025-12-04T08:27:07.1974120Z * [new branch] gh/mikaylagawarecki/352/orig -> origin/gh/mikaylagawarecki/352/orig 2025-12-04T08:27:07.1980252Z * [new branch] gh/mikaylagawarecki/353/base -> origin/gh/mikaylagawarecki/353/base 2025-12-04T08:27:07.1980487Z * [new branch] gh/mikaylagawarecki/353/head -> origin/gh/mikaylagawarecki/353/head 2025-12-04T08:27:07.1980669Z * [new branch] gh/mikaylagawarecki/353/orig -> origin/gh/mikaylagawarecki/353/orig 2025-12-04T08:27:07.1980843Z * [new branch] gh/mikaylagawarecki/354/base -> origin/gh/mikaylagawarecki/354/base 2025-12-04T08:27:07.1981023Z * [new branch] gh/mikaylagawarecki/354/head -> origin/gh/mikaylagawarecki/354/head 2025-12-04T08:27:07.1981193Z * [new branch] gh/mikaylagawarecki/354/orig -> origin/gh/mikaylagawarecki/354/orig 2025-12-04T08:27:07.1981372Z * [new branch] gh/mikaylagawarecki/356/base -> origin/gh/mikaylagawarecki/356/base 2025-12-04T08:27:07.1985290Z * [new branch] gh/mikaylagawarecki/356/head -> origin/gh/mikaylagawarecki/356/head 2025-12-04T08:27:07.1987395Z * [new branch] gh/mikaylagawarecki/356/orig -> origin/gh/mikaylagawarecki/356/orig 2025-12-04T08:27:07.1987775Z * [new branch] gh/mikaylagawarecki/357/base -> origin/gh/mikaylagawarecki/357/base 2025-12-04T08:27:07.1987985Z * [new branch] gh/mikaylagawarecki/357/head -> origin/gh/mikaylagawarecki/357/head 2025-12-04T08:27:07.1988301Z * [new branch] gh/mikaylagawarecki/357/orig -> origin/gh/mikaylagawarecki/357/orig 2025-12-04T08:27:07.1988575Z * [new branch] gh/mikaylagawarecki/359/base -> origin/gh/mikaylagawarecki/359/base 2025-12-04T08:27:07.1988880Z * [new branch] gh/mikaylagawarecki/359/head -> origin/gh/mikaylagawarecki/359/head 2025-12-04T08:27:07.1989176Z * [new branch] gh/mikaylagawarecki/359/orig -> origin/gh/mikaylagawarecki/359/orig 2025-12-04T08:27:07.1989459Z * [new branch] gh/mikaylagawarecki/360/base -> origin/gh/mikaylagawarecki/360/base 2025-12-04T08:27:07.1989743Z * [new branch] gh/mikaylagawarecki/360/head -> origin/gh/mikaylagawarecki/360/head 2025-12-04T08:27:07.1990013Z * [new branch] gh/mikaylagawarecki/360/orig -> origin/gh/mikaylagawarecki/360/orig 2025-12-04T08:27:07.1990471Z * [new branch] gh/mikaylagawarecki/361/base -> origin/gh/mikaylagawarecki/361/base 2025-12-04T08:27:07.1991130Z * [new branch] gh/mikaylagawarecki/361/head -> origin/gh/mikaylagawarecki/361/head 2025-12-04T08:27:07.1991345Z * [new branch] gh/mikaylagawarecki/361/orig -> origin/gh/mikaylagawarecki/361/orig 2025-12-04T08:27:07.1992286Z * [new branch] gh/mikaylagawarecki/362/base -> origin/gh/mikaylagawarecki/362/base 2025-12-04T08:27:07.1992680Z * [new branch] gh/mikaylagawarecki/362/head -> origin/gh/mikaylagawarecki/362/head 2025-12-04T08:27:07.1995518Z * [new branch] gh/mikaylagawarecki/362/orig -> origin/gh/mikaylagawarecki/362/orig 2025-12-04T08:27:07.1995757Z * [new branch] gh/mikaylagawarecki/363/base -> origin/gh/mikaylagawarecki/363/base 2025-12-04T08:27:07.1995941Z * [new branch] gh/mikaylagawarecki/363/head -> origin/gh/mikaylagawarecki/363/head 2025-12-04T08:27:07.1996395Z * [new branch] gh/mikaylagawarecki/363/orig -> origin/gh/mikaylagawarecki/363/orig 2025-12-04T08:27:07.1998258Z * [new branch] gh/mikaylagawarecki/364/base -> origin/gh/mikaylagawarecki/364/base 2025-12-04T08:27:07.1998653Z * [new branch] gh/mikaylagawarecki/364/head -> origin/gh/mikaylagawarecki/364/head 2025-12-04T08:27:07.1999117Z * [new branch] gh/mikaylagawarecki/364/orig -> origin/gh/mikaylagawarecki/364/orig 2025-12-04T08:27:07.2002734Z * [new branch] gh/mikaylagawarecki/365/base -> origin/gh/mikaylagawarecki/365/base 2025-12-04T08:27:07.2003124Z * [new branch] gh/mikaylagawarecki/365/head -> origin/gh/mikaylagawarecki/365/head 2025-12-04T08:27:07.2003615Z * [new branch] gh/mikaylagawarecki/365/orig -> origin/gh/mikaylagawarecki/365/orig 2025-12-04T08:27:07.2004229Z * [new branch] gh/mikaylagawarecki/366/base -> origin/gh/mikaylagawarecki/366/base 2025-12-04T08:27:07.2004448Z * [new branch] gh/mikaylagawarecki/366/head -> origin/gh/mikaylagawarecki/366/head 2025-12-04T08:27:07.2004704Z * [new branch] gh/mikaylagawarecki/366/orig -> origin/gh/mikaylagawarecki/366/orig 2025-12-04T08:27:07.2006453Z * [new branch] gh/mikaylagawarecki/367/base -> origin/gh/mikaylagawarecki/367/base 2025-12-04T08:27:07.2006846Z * [new branch] gh/mikaylagawarecki/367/head -> origin/gh/mikaylagawarecki/367/head 2025-12-04T08:27:07.2007131Z * [new branch] gh/mikaylagawarecki/367/orig -> origin/gh/mikaylagawarecki/367/orig 2025-12-04T08:27:07.2009120Z * [new branch] gh/mikaylagawarecki/368/base -> origin/gh/mikaylagawarecki/368/base 2025-12-04T08:27:07.2009516Z * [new branch] gh/mikaylagawarecki/368/head -> origin/gh/mikaylagawarecki/368/head 2025-12-04T08:27:07.2009822Z * [new branch] gh/mikaylagawarecki/368/orig -> origin/gh/mikaylagawarecki/368/orig 2025-12-04T08:27:07.2012812Z * [new branch] gh/mikaylagawarecki/369/base -> origin/gh/mikaylagawarecki/369/base 2025-12-04T08:27:07.2013200Z * [new branch] gh/mikaylagawarecki/369/head -> origin/gh/mikaylagawarecki/369/head 2025-12-04T08:27:07.2013513Z * [new branch] gh/mikaylagawarecki/369/orig -> origin/gh/mikaylagawarecki/369/orig 2025-12-04T08:27:07.2013775Z * [new branch] gh/mikaylagawarecki/370/base -> origin/gh/mikaylagawarecki/370/base 2025-12-04T08:27:07.2014040Z * [new branch] gh/mikaylagawarecki/370/head -> origin/gh/mikaylagawarecki/370/head 2025-12-04T08:27:07.2014402Z * [new branch] gh/mikaylagawarecki/370/orig -> origin/gh/mikaylagawarecki/370/orig 2025-12-04T08:27:07.2019023Z * [new branch] gh/mikaylagawarecki/371/base -> origin/gh/mikaylagawarecki/371/base 2025-12-04T08:27:07.2019245Z * [new branch] gh/mikaylagawarecki/371/head -> origin/gh/mikaylagawarecki/371/head 2025-12-04T08:27:07.2019415Z * [new branch] gh/mikaylagawarecki/371/orig -> origin/gh/mikaylagawarecki/371/orig 2025-12-04T08:27:07.2019745Z * [new branch] gh/mikaylagawarecki/372/base -> origin/gh/mikaylagawarecki/372/base 2025-12-04T08:27:07.2019909Z * [new branch] gh/mikaylagawarecki/372/head -> origin/gh/mikaylagawarecki/372/head 2025-12-04T08:27:07.2020086Z * [new branch] gh/mikaylagawarecki/372/orig -> origin/gh/mikaylagawarecki/372/orig 2025-12-04T08:27:07.2020301Z * [new branch] gh/mikaylagawarecki/373/base -> origin/gh/mikaylagawarecki/373/base 2025-12-04T08:27:07.2021306Z * [new branch] gh/mikaylagawarecki/373/head -> origin/gh/mikaylagawarecki/373/head 2025-12-04T08:27:07.2021679Z * [new branch] gh/mikaylagawarecki/373/orig -> origin/gh/mikaylagawarecki/373/orig 2025-12-04T08:27:07.2025732Z * [new branch] gh/mikaylagawarecki/374/base -> origin/gh/mikaylagawarecki/374/base 2025-12-04T08:27:07.2026101Z * [new branch] gh/mikaylagawarecki/374/head -> origin/gh/mikaylagawarecki/374/head 2025-12-04T08:27:07.2026422Z * [new branch] gh/mikaylagawarecki/374/orig -> origin/gh/mikaylagawarecki/374/orig 2025-12-04T08:27:07.2026771Z * [new branch] gh/mikaylagawarecki/375/base -> origin/gh/mikaylagawarecki/375/base 2025-12-04T08:27:07.2027078Z * [new branch] gh/mikaylagawarecki/375/head -> origin/gh/mikaylagawarecki/375/head 2025-12-04T08:27:07.2027803Z * [new branch] gh/mikaylagawarecki/375/orig -> origin/gh/mikaylagawarecki/375/orig 2025-12-04T08:27:07.2028021Z * [new branch] gh/mikaylagawarecki/376/base -> origin/gh/mikaylagawarecki/376/base 2025-12-04T08:27:07.2028209Z * [new branch] gh/mikaylagawarecki/376/head -> origin/gh/mikaylagawarecki/376/head 2025-12-04T08:27:07.2029247Z * [new branch] gh/mikaylagawarecki/376/orig -> origin/gh/mikaylagawarecki/376/orig 2025-12-04T08:27:07.2031987Z * [new branch] gh/mikaylagawarecki/377/base -> origin/gh/mikaylagawarecki/377/base 2025-12-04T08:27:07.2032382Z * [new branch] gh/mikaylagawarecki/377/head -> origin/gh/mikaylagawarecki/377/head 2025-12-04T08:27:07.2032669Z * [new branch] gh/mikaylagawarecki/377/orig -> origin/gh/mikaylagawarecki/377/orig 2025-12-04T08:27:07.2032947Z * [new branch] gh/mikaylagawarecki/378/base -> origin/gh/mikaylagawarecki/378/base 2025-12-04T08:27:07.2033232Z * [new branch] gh/mikaylagawarecki/378/head -> origin/gh/mikaylagawarecki/378/head 2025-12-04T08:27:07.2034097Z * [new branch] gh/mikaylagawarecki/378/orig -> origin/gh/mikaylagawarecki/378/orig 2025-12-04T08:27:07.2038468Z * [new branch] gh/mikaylagawarecki/379/base -> origin/gh/mikaylagawarecki/379/base 2025-12-04T08:27:07.2038717Z * [new branch] gh/mikaylagawarecki/379/head -> origin/gh/mikaylagawarecki/379/head 2025-12-04T08:27:07.2038907Z * [new branch] gh/mikaylagawarecki/379/orig -> origin/gh/mikaylagawarecki/379/orig 2025-12-04T08:27:07.2039112Z * [new branch] gh/mikaylagawarecki/380/base -> origin/gh/mikaylagawarecki/380/base 2025-12-04T08:27:07.2039287Z * [new branch] gh/mikaylagawarecki/380/head -> origin/gh/mikaylagawarecki/380/head 2025-12-04T08:27:07.2039473Z * [new branch] gh/mikaylagawarecki/380/orig -> origin/gh/mikaylagawarecki/380/orig 2025-12-04T08:27:07.2039646Z * [new branch] gh/mikaylagawarecki/381/base -> origin/gh/mikaylagawarecki/381/base 2025-12-04T08:27:07.2043100Z * [new branch] gh/mikaylagawarecki/381/head -> origin/gh/mikaylagawarecki/381/head 2025-12-04T08:27:07.2043498Z * [new branch] gh/mikaylagawarecki/381/orig -> origin/gh/mikaylagawarecki/381/orig 2025-12-04T08:27:07.2043836Z * [new branch] gh/mikaylagawarecki/382/base -> origin/gh/mikaylagawarecki/382/base 2025-12-04T08:27:07.2044113Z * [new branch] gh/mikaylagawarecki/382/head -> origin/gh/mikaylagawarecki/382/head 2025-12-04T08:27:07.2044507Z * [new branch] gh/mikaylagawarecki/382/orig -> origin/gh/mikaylagawarecki/382/orig 2025-12-04T08:27:07.2044714Z * [new branch] gh/mikaylagawarecki/383/base -> origin/gh/mikaylagawarecki/383/base 2025-12-04T08:27:07.2045482Z * [new branch] gh/mikaylagawarecki/383/head -> origin/gh/mikaylagawarecki/383/head 2025-12-04T08:27:07.2049754Z * [new branch] gh/mikaylagawarecki/383/orig -> origin/gh/mikaylagawarecki/383/orig 2025-12-04T08:27:07.2049970Z * [new branch] gh/mikaylagawarecki/384/base -> origin/gh/mikaylagawarecki/384/base 2025-12-04T08:27:07.2050144Z * [new branch] gh/mikaylagawarecki/384/head -> origin/gh/mikaylagawarecki/384/head 2025-12-04T08:27:07.2050349Z * [new branch] gh/mikaylagawarecki/384/orig -> origin/gh/mikaylagawarecki/384/orig 2025-12-04T08:27:07.2050520Z * [new branch] gh/mikaylagawarecki/385/base -> origin/gh/mikaylagawarecki/385/base 2025-12-04T08:27:07.2050696Z * [new branch] gh/mikaylagawarecki/385/head -> origin/gh/mikaylagawarecki/385/head 2025-12-04T08:27:07.2050888Z * [new branch] gh/mikaylagawarecki/385/orig -> origin/gh/mikaylagawarecki/385/orig 2025-12-04T08:27:07.2054937Z * [new branch] gh/mikaylagawarecki/386/base -> origin/gh/mikaylagawarecki/386/base 2025-12-04T08:27:07.2055153Z * [new branch] gh/mikaylagawarecki/386/head -> origin/gh/mikaylagawarecki/386/head 2025-12-04T08:27:07.2055326Z * [new branch] gh/mikaylagawarecki/386/orig -> origin/gh/mikaylagawarecki/386/orig 2025-12-04T08:27:07.2055500Z * [new branch] gh/mikaylagawarecki/387/base -> origin/gh/mikaylagawarecki/387/base 2025-12-04T08:27:07.2055842Z * [new branch] gh/mikaylagawarecki/387/head -> origin/gh/mikaylagawarecki/387/head 2025-12-04T08:27:07.2056021Z * [new branch] gh/mikaylagawarecki/387/orig -> origin/gh/mikaylagawarecki/387/orig 2025-12-04T08:27:07.2056254Z * [new branch] gh/mikaylagawarecki/388/base -> origin/gh/mikaylagawarecki/388/base 2025-12-04T08:27:07.2057120Z * [new branch] gh/mikaylagawarecki/388/head -> origin/gh/mikaylagawarecki/388/head 2025-12-04T08:27:07.2057485Z * [new branch] gh/mikaylagawarecki/388/orig -> origin/gh/mikaylagawarecki/388/orig 2025-12-04T08:27:07.2060646Z * [new branch] gh/mikaylagawarecki/389/base -> origin/gh/mikaylagawarecki/389/base 2025-12-04T08:27:07.2060868Z * [new branch] gh/mikaylagawarecki/389/head -> origin/gh/mikaylagawarecki/389/head 2025-12-04T08:27:07.2061043Z * [new branch] gh/mikaylagawarecki/389/orig -> origin/gh/mikaylagawarecki/389/orig 2025-12-04T08:27:07.2061245Z * [new branch] gh/mikaylagawarecki/390/base -> origin/gh/mikaylagawarecki/390/base 2025-12-04T08:27:07.2061470Z * [new branch] gh/mikaylagawarecki/390/head -> origin/gh/mikaylagawarecki/390/head 2025-12-04T08:27:07.2062491Z * [new branch] gh/mikaylagawarecki/390/orig -> origin/gh/mikaylagawarecki/390/orig 2025-12-04T08:27:07.2065956Z * [new branch] gh/mikaylagawarecki/391/base -> origin/gh/mikaylagawarecki/391/base 2025-12-04T08:27:07.2066322Z * [new branch] gh/mikaylagawarecki/391/head -> origin/gh/mikaylagawarecki/391/head 2025-12-04T08:27:07.2066639Z * [new branch] gh/mikaylagawarecki/391/orig -> origin/gh/mikaylagawarecki/391/orig 2025-12-04T08:27:07.2067281Z * [new branch] gh/mikaylagawarecki/392/base -> origin/gh/mikaylagawarecki/392/base 2025-12-04T08:27:07.2067491Z * [new branch] gh/mikaylagawarecki/392/head -> origin/gh/mikaylagawarecki/392/head 2025-12-04T08:27:07.2067659Z * [new branch] gh/mikaylagawarecki/392/orig -> origin/gh/mikaylagawarecki/392/orig 2025-12-04T08:27:07.2069104Z * [new branch] gh/mlazos/41/base -> origin/gh/mlazos/41/base 2025-12-04T08:27:07.2069404Z * [new branch] gh/mlazos/41/head -> origin/gh/mlazos/41/head 2025-12-04T08:27:07.2069745Z * [new branch] gh/mlazos/41/orig -> origin/gh/mlazos/41/orig 2025-12-04T08:27:07.2071451Z * [new branch] gh/mlazos/42/base -> origin/gh/mlazos/42/base 2025-12-04T08:27:07.2071755Z * [new branch] gh/mlazos/42/head -> origin/gh/mlazos/42/head 2025-12-04T08:27:07.2071992Z * [new branch] gh/mlazos/42/orig -> origin/gh/mlazos/42/orig 2025-12-04T08:27:07.2074706Z * [new branch] gh/mlazos/43/base -> origin/gh/mlazos/43/base 2025-12-04T08:27:07.2074881Z * [new branch] gh/mlazos/43/head -> origin/gh/mlazos/43/head 2025-12-04T08:27:07.2075027Z * [new branch] gh/mlazos/43/orig -> origin/gh/mlazos/43/orig 2025-12-04T08:27:07.2075981Z * [new branch] gh/mlazos/44/base -> origin/gh/mlazos/44/base 2025-12-04T08:27:07.2076292Z * [new branch] gh/mlazos/44/head -> origin/gh/mlazos/44/head 2025-12-04T08:27:07.2080983Z * [new branch] gh/mlazos/44/orig -> origin/gh/mlazos/44/orig 2025-12-04T08:27:07.2081326Z * [new branch] gh/mlazos/47/base -> origin/gh/mlazos/47/base 2025-12-04T08:27:07.2081571Z * [new branch] gh/mlazos/47/head -> origin/gh/mlazos/47/head 2025-12-04T08:27:07.2081782Z * [new branch] gh/mlazos/47/orig -> origin/gh/mlazos/47/orig 2025-12-04T08:27:07.2082105Z * [new branch] gh/mlazos/48/base -> origin/gh/mlazos/48/base 2025-12-04T08:27:07.2082316Z * [new branch] gh/mlazos/48/head -> origin/gh/mlazos/48/head 2025-12-04T08:27:07.2082677Z * [new branch] gh/mlazos/48/orig -> origin/gh/mlazos/48/orig 2025-12-04T08:27:07.2083048Z * [new branch] gh/mlazos/49/base -> origin/gh/mlazos/49/base 2025-12-04T08:27:07.2085258Z * [new branch] gh/mlazos/49/head -> origin/gh/mlazos/49/head 2025-12-04T08:27:07.2085612Z * [new branch] gh/mlazos/49/orig -> origin/gh/mlazos/49/orig 2025-12-04T08:27:07.2085865Z * [new branch] gh/mlazos/50/base -> origin/gh/mlazos/50/base 2025-12-04T08:27:07.2086027Z * [new branch] gh/mlazos/50/head -> origin/gh/mlazos/50/head 2025-12-04T08:27:07.2087043Z * [new branch] gh/mlazos/50/orig -> origin/gh/mlazos/50/orig 2025-12-04T08:27:07.2090994Z * [new branch] gh/mlazos/51/base -> origin/gh/mlazos/51/base 2025-12-04T08:27:07.2091320Z * [new branch] gh/mlazos/51/head -> origin/gh/mlazos/51/head 2025-12-04T08:27:07.2091570Z * [new branch] gh/mlazos/51/orig -> origin/gh/mlazos/51/orig 2025-12-04T08:27:07.2091792Z * [new branch] gh/mlazos/52/base -> origin/gh/mlazos/52/base 2025-12-04T08:27:07.2092029Z * [new branch] gh/mlazos/52/head -> origin/gh/mlazos/52/head 2025-12-04T08:27:07.2092245Z * [new branch] gh/mlazos/52/orig -> origin/gh/mlazos/52/orig 2025-12-04T08:27:07.2092923Z * [new branch] gh/mlazos/53/base -> origin/gh/mlazos/53/base 2025-12-04T08:27:07.2093089Z * [new branch] gh/mlazos/53/head -> origin/gh/mlazos/53/head 2025-12-04T08:27:07.2093281Z * [new branch] gh/mlazos/53/orig -> origin/gh/mlazos/53/orig 2025-12-04T08:27:07.2096434Z * [new branch] gh/mlazos/54/base -> origin/gh/mlazos/54/base 2025-12-04T08:27:07.2096757Z * [new branch] gh/mlazos/54/head -> origin/gh/mlazos/54/head 2025-12-04T08:27:07.2097007Z * [new branch] gh/mlazos/54/orig -> origin/gh/mlazos/54/orig 2025-12-04T08:27:07.2097217Z * [new branch] gh/mlazos/55/base -> origin/gh/mlazos/55/base 2025-12-04T08:27:07.2097380Z * [new branch] gh/mlazos/55/head -> origin/gh/mlazos/55/head 2025-12-04T08:27:07.2097847Z * [new branch] gh/mlazos/55/orig -> origin/gh/mlazos/55/orig 2025-12-04T08:27:07.2100785Z * [new branch] gh/mlazos/56/base -> origin/gh/mlazos/56/base 2025-12-04T08:27:07.2101124Z * [new branch] gh/mlazos/56/head -> origin/gh/mlazos/56/head 2025-12-04T08:27:07.2101414Z * [new branch] gh/mlazos/56/orig -> origin/gh/mlazos/56/orig 2025-12-04T08:27:07.2101588Z * [new branch] gh/mlazos/57/base -> origin/gh/mlazos/57/base 2025-12-04T08:27:07.2101871Z * [new branch] gh/mlazos/57/head -> origin/gh/mlazos/57/head 2025-12-04T08:27:07.2102410Z * [new branch] gh/mlazos/57/orig -> origin/gh/mlazos/57/orig 2025-12-04T08:27:07.2107955Z * [new branch] gh/mlazos/58/base -> origin/gh/mlazos/58/base 2025-12-04T08:27:07.2108313Z * [new branch] gh/mlazos/58/head -> origin/gh/mlazos/58/head 2025-12-04T08:27:07.2108613Z * [new branch] gh/mlazos/58/orig -> origin/gh/mlazos/58/orig 2025-12-04T08:27:07.2108798Z * [new branch] gh/mlazos/59/base -> origin/gh/mlazos/59/base 2025-12-04T08:27:07.2108929Z * [new branch] gh/mlazos/59/head -> origin/gh/mlazos/59/head 2025-12-04T08:27:07.2109057Z * [new branch] gh/mlazos/59/orig -> origin/gh/mlazos/59/orig 2025-12-04T08:27:07.2109328Z * [new branch] gh/mlazos/60/base -> origin/gh/mlazos/60/base 2025-12-04T08:27:07.2109459Z * [new branch] gh/mlazos/60/head -> origin/gh/mlazos/60/head 2025-12-04T08:27:07.2109746Z * [new branch] gh/mlazos/60/orig -> origin/gh/mlazos/60/orig 2025-12-04T08:27:07.2112026Z * [new branch] gh/mlazos/61/base -> origin/gh/mlazos/61/base 2025-12-04T08:27:07.2112352Z * [new branch] gh/mlazos/61/head -> origin/gh/mlazos/61/head 2025-12-04T08:27:07.2112614Z * [new branch] gh/mlazos/61/orig -> origin/gh/mlazos/61/orig 2025-12-04T08:27:07.2112784Z * [new branch] gh/mlazos/62/base -> origin/gh/mlazos/62/base 2025-12-04T08:27:07.2113242Z * [new branch] gh/mlazos/62/head -> origin/gh/mlazos/62/head 2025-12-04T08:27:07.2114332Z * [new branch] gh/mlazos/62/orig -> origin/gh/mlazos/62/orig 2025-12-04T08:27:07.2117536Z * [new branch] gh/mlazos/63/base -> origin/gh/mlazos/63/base 2025-12-04T08:27:07.2117730Z * [new branch] gh/mlazos/63/head -> origin/gh/mlazos/63/head 2025-12-04T08:27:07.2117896Z * [new branch] gh/mlazos/63/orig -> origin/gh/mlazos/63/orig 2025-12-04T08:27:07.2118197Z * [new branch] gh/mlazos/64/base -> origin/gh/mlazos/64/base 2025-12-04T08:27:07.2118371Z * [new branch] gh/mlazos/64/head -> origin/gh/mlazos/64/head 2025-12-04T08:27:07.2118996Z * [new branch] gh/mlazos/64/orig -> origin/gh/mlazos/64/orig 2025-12-04T08:27:07.2123346Z * [new branch] gh/mlazos/65/base -> origin/gh/mlazos/65/base 2025-12-04T08:27:07.2123529Z * [new branch] gh/mlazos/65/head -> origin/gh/mlazos/65/head 2025-12-04T08:27:07.2123670Z * [new branch] gh/mlazos/65/orig -> origin/gh/mlazos/65/orig 2025-12-04T08:27:07.2123822Z * [new branch] gh/mlazos/66/base -> origin/gh/mlazos/66/base 2025-12-04T08:27:07.2123971Z * [new branch] gh/mlazos/66/head -> origin/gh/mlazos/66/head 2025-12-04T08:27:07.2124133Z * [new branch] gh/mlazos/66/orig -> origin/gh/mlazos/66/orig 2025-12-04T08:27:07.2124575Z * [new branch] gh/mlazos/67/base -> origin/gh/mlazos/67/base 2025-12-04T08:27:07.2127598Z * [new branch] gh/mlazos/67/head -> origin/gh/mlazos/67/head 2025-12-04T08:27:07.2128189Z * [new branch] gh/mlazos/67/orig -> origin/gh/mlazos/67/orig 2025-12-04T08:27:07.2128352Z * [new branch] gh/mlazos/68/base -> origin/gh/mlazos/68/base 2025-12-04T08:27:07.2128497Z * [new branch] gh/mlazos/68/head -> origin/gh/mlazos/68/head 2025-12-04T08:27:07.2128727Z * [new branch] gh/mlazos/68/orig -> origin/gh/mlazos/68/orig 2025-12-04T08:27:07.2133051Z * [new branch] gh/mlazos/69/base -> origin/gh/mlazos/69/base 2025-12-04T08:27:07.2133423Z * [new branch] gh/mlazos/69/head -> origin/gh/mlazos/69/head 2025-12-04T08:27:07.2133700Z * [new branch] gh/mlazos/69/orig -> origin/gh/mlazos/69/orig 2025-12-04T08:27:07.2133939Z * [new branch] gh/mlazos/70/base -> origin/gh/mlazos/70/base 2025-12-04T08:27:07.2134093Z * [new branch] gh/mlazos/70/head -> origin/gh/mlazos/70/head 2025-12-04T08:27:07.2134837Z * [new branch] gh/mlazos/70/orig -> origin/gh/mlazos/70/orig 2025-12-04T08:27:07.2135010Z * [new branch] gh/mlazos/71/base -> origin/gh/mlazos/71/base 2025-12-04T08:27:07.2135147Z * [new branch] gh/mlazos/71/head -> origin/gh/mlazos/71/head 2025-12-04T08:27:07.2136482Z * [new branch] gh/mlazos/71/orig -> origin/gh/mlazos/71/orig 2025-12-04T08:27:07.2136664Z * [new branch] gh/mlazos/72/base -> origin/gh/mlazos/72/base 2025-12-04T08:27:07.2138014Z * [new branch] gh/mlazos/72/head -> origin/gh/mlazos/72/head 2025-12-04T08:27:07.2138479Z * [new branch] gh/mlazos/72/orig -> origin/gh/mlazos/72/orig 2025-12-04T08:27:07.2138646Z * [new branch] gh/mlazos/73/base -> origin/gh/mlazos/73/base 2025-12-04T08:27:07.2140639Z * [new branch] gh/mlazos/73/head -> origin/gh/mlazos/73/head 2025-12-04T08:27:07.2140975Z * [new branch] gh/mlazos/73/orig -> origin/gh/mlazos/73/orig 2025-12-04T08:27:07.2141205Z * [new branch] gh/mrmiywj/1/base -> origin/gh/mrmiywj/1/base 2025-12-04T08:27:07.2142973Z * [new branch] gh/mrmiywj/1/head -> origin/gh/mrmiywj/1/head 2025-12-04T08:27:07.2143253Z * [new branch] gh/muchulee8/73/base -> origin/gh/muchulee8/73/base 2025-12-04T08:27:07.2149825Z * [new branch] gh/muchulee8/73/head -> origin/gh/muchulee8/73/head 2025-12-04T08:27:07.2150178Z * [new branch] gh/muchulee8/73/orig -> origin/gh/muchulee8/73/orig 2025-12-04T08:27:07.2150506Z * [new branch] gh/naveenthangudu/1/base -> origin/gh/naveenthangudu/1/base 2025-12-04T08:27:07.2150774Z * [new branch] gh/naveenthangudu/1/head -> origin/gh/naveenthangudu/1/head 2025-12-04T08:27:07.2151013Z * [new branch] gh/naveenthangudu/1/orig -> origin/gh/naveenthangudu/1/orig 2025-12-04T08:27:07.2151647Z * [new branch] gh/naveenthangudu/2/base -> origin/gh/naveenthangudu/2/base 2025-12-04T08:27:07.2151840Z * [new branch] gh/naveenthangudu/2/head -> origin/gh/naveenthangudu/2/head 2025-12-04T08:27:07.2152001Z * [new branch] gh/naveenthangudu/2/orig -> origin/gh/naveenthangudu/2/orig 2025-12-04T08:27:07.2152150Z * [new branch] gh/naveenthangudu/3/base -> origin/gh/naveenthangudu/3/base 2025-12-04T08:27:07.2152295Z * [new branch] gh/naveenthangudu/3/head -> origin/gh/naveenthangudu/3/head 2025-12-04T08:27:07.2152449Z * [new branch] gh/naveenthangudu/3/orig -> origin/gh/naveenthangudu/3/orig 2025-12-04T08:27:07.2157149Z * [new branch] gh/naveenthangudu/4/base -> origin/gh/naveenthangudu/4/base 2025-12-04T08:27:07.2157372Z * [new branch] gh/naveenthangudu/4/head -> origin/gh/naveenthangudu/4/head 2025-12-04T08:27:07.2157687Z * [new branch] gh/naveenthangudu/4/orig -> origin/gh/naveenthangudu/4/orig 2025-12-04T08:27:07.2157852Z * [new branch] gh/naveenthangudu/5/base -> origin/gh/naveenthangudu/5/base 2025-12-04T08:27:07.2158201Z * [new branch] gh/naveenthangudu/5/head -> origin/gh/naveenthangudu/5/head 2025-12-04T08:27:07.2158380Z * [new branch] gh/naveenthangudu/5/orig -> origin/gh/naveenthangudu/5/orig 2025-12-04T08:27:07.2158547Z * [new branch] gh/naveenthangudu/6/base -> origin/gh/naveenthangudu/6/base 2025-12-04T08:27:07.2158707Z * [new branch] gh/naveenthangudu/6/head -> origin/gh/naveenthangudu/6/head 2025-12-04T08:27:07.2158874Z * [new branch] gh/naveenthangudu/6/orig -> origin/gh/naveenthangudu/6/orig 2025-12-04T08:27:07.2159064Z * [new branch] gh/naveenthangudu/7/base -> origin/gh/naveenthangudu/7/base 2025-12-04T08:27:07.2159234Z * [new branch] gh/naveenthangudu/7/head -> origin/gh/naveenthangudu/7/head 2025-12-04T08:27:07.2159502Z * [new branch] gh/naveenthangudu/7/orig -> origin/gh/naveenthangudu/7/orig 2025-12-04T08:27:07.2162373Z * [new branch] gh/naveenthangudu/8/base -> origin/gh/naveenthangudu/8/base 2025-12-04T08:27:07.2166561Z * [new branch] gh/naveenthangudu/8/head -> origin/gh/naveenthangudu/8/head 2025-12-04T08:27:07.2166757Z * [new branch] gh/naveenthangudu/8/orig -> origin/gh/naveenthangudu/8/orig 2025-12-04T08:27:07.2166923Z * [new branch] gh/naveenthangudu/9/base -> origin/gh/naveenthangudu/9/base 2025-12-04T08:27:07.2167079Z * [new branch] gh/naveenthangudu/9/head -> origin/gh/naveenthangudu/9/head 2025-12-04T08:27:07.2167409Z * [new branch] gh/naveenthangudu/9/orig -> origin/gh/naveenthangudu/9/orig 2025-12-04T08:27:07.2167589Z * [new branch] gh/nikitaved/1/base -> origin/gh/nikitaved/1/base 2025-12-04T08:27:07.2167757Z * [new branch] gh/nikitaved/1/head -> origin/gh/nikitaved/1/head 2025-12-04T08:27:07.2167911Z * [new branch] gh/nikitaved/1/orig -> origin/gh/nikitaved/1/orig 2025-12-04T08:27:07.2168487Z * [new branch] gh/nikitaved/10/base -> origin/gh/nikitaved/10/base 2025-12-04T08:27:07.2168979Z * [new branch] gh/nikitaved/10/head -> origin/gh/nikitaved/10/head 2025-12-04T08:27:07.2169427Z * [new branch] gh/nikitaved/10/orig -> origin/gh/nikitaved/10/orig 2025-12-04T08:27:07.2173360Z * [new branch] gh/nikitaved/11/base -> origin/gh/nikitaved/11/base 2025-12-04T08:27:07.2173807Z * [new branch] gh/nikitaved/11/head -> origin/gh/nikitaved/11/head 2025-12-04T08:27:07.2173962Z * [new branch] gh/nikitaved/11/orig -> origin/gh/nikitaved/11/orig 2025-12-04T08:27:07.2174104Z * [new branch] gh/nikitaved/12/base -> origin/gh/nikitaved/12/base 2025-12-04T08:27:07.2174251Z * [new branch] gh/nikitaved/12/head -> origin/gh/nikitaved/12/head 2025-12-04T08:27:07.2174404Z * [new branch] gh/nikitaved/12/orig -> origin/gh/nikitaved/12/orig 2025-12-04T08:27:07.2174980Z * [new branch] gh/nikitaved/13/base -> origin/gh/nikitaved/13/base 2025-12-04T08:27:07.2179253Z * [new branch] gh/nikitaved/13/head -> origin/gh/nikitaved/13/head 2025-12-04T08:27:07.2179439Z * [new branch] gh/nikitaved/13/orig -> origin/gh/nikitaved/13/orig 2025-12-04T08:27:07.2179600Z * [new branch] gh/nikitaved/14/base -> origin/gh/nikitaved/14/base 2025-12-04T08:27:07.2179765Z * [new branch] gh/nikitaved/14/head -> origin/gh/nikitaved/14/head 2025-12-04T08:27:07.2179908Z * [new branch] gh/nikitaved/14/orig -> origin/gh/nikitaved/14/orig 2025-12-04T08:27:07.2180057Z * [new branch] gh/nikitaved/15/base -> origin/gh/nikitaved/15/base 2025-12-04T08:27:07.2180760Z * [new branch] gh/nikitaved/15/head -> origin/gh/nikitaved/15/head 2025-12-04T08:27:07.2181369Z * [new branch] gh/nikitaved/15/orig -> origin/gh/nikitaved/15/orig 2025-12-04T08:27:07.2186725Z * [new branch] gh/nikitaved/16/base -> origin/gh/nikitaved/16/base 2025-12-04T08:27:07.2186922Z * [new branch] gh/nikitaved/16/head -> origin/gh/nikitaved/16/head 2025-12-04T08:27:07.2187112Z * [new branch] gh/nikitaved/16/orig -> origin/gh/nikitaved/16/orig 2025-12-04T08:27:07.2187279Z * [new branch] gh/nikitaved/2/base -> origin/gh/nikitaved/2/base 2025-12-04T08:27:07.2187456Z * [new branch] gh/nikitaved/2/head -> origin/gh/nikitaved/2/head 2025-12-04T08:27:07.2187603Z * [new branch] gh/nikitaved/2/orig -> origin/gh/nikitaved/2/orig 2025-12-04T08:27:07.2187765Z * [new branch] gh/nikitaved/4/base -> origin/gh/nikitaved/4/base 2025-12-04T08:27:07.2187968Z * [new branch] gh/nikitaved/4/head -> origin/gh/nikitaved/4/head 2025-12-04T08:27:07.2188116Z * [new branch] gh/nikitaved/4/orig -> origin/gh/nikitaved/4/orig 2025-12-04T08:27:07.2189676Z * [new branch] gh/nikitaved/5/base -> origin/gh/nikitaved/5/base 2025-12-04T08:27:07.2189837Z * [new branch] gh/nikitaved/5/head -> origin/gh/nikitaved/5/head 2025-12-04T08:27:07.2195193Z * [new branch] gh/nikitaved/5/orig -> origin/gh/nikitaved/5/orig 2025-12-04T08:27:07.2195553Z * [new branch] gh/nikitaved/6/base -> origin/gh/nikitaved/6/base 2025-12-04T08:27:07.2196004Z * [new branch] gh/nikitaved/6/head -> origin/gh/nikitaved/6/head 2025-12-04T08:27:07.2196305Z * [new branch] gh/nikitaved/6/orig -> origin/gh/nikitaved/6/orig 2025-12-04T08:27:07.2196593Z * [new branch] gh/nikitaved/8/base -> origin/gh/nikitaved/8/base 2025-12-04T08:27:07.2197266Z * [new branch] gh/nikitaved/8/head -> origin/gh/nikitaved/8/head 2025-12-04T08:27:07.2197469Z * [new branch] gh/nikitaved/8/orig -> origin/gh/nikitaved/8/orig 2025-12-04T08:27:07.2197632Z * [new branch] gh/nikitaved/9/base -> origin/gh/nikitaved/9/base 2025-12-04T08:27:07.2197809Z * [new branch] gh/nikitaved/9/head -> origin/gh/nikitaved/9/head 2025-12-04T08:27:07.2197963Z * [new branch] gh/nikitaved/9/orig -> origin/gh/nikitaved/9/orig 2025-12-04T08:27:07.2198968Z * [new branch] gh/oulgen/10/base -> origin/gh/oulgen/10/base 2025-12-04T08:27:07.2199836Z * [new branch] gh/oulgen/10/head -> origin/gh/oulgen/10/head 2025-12-04T08:27:07.2200187Z * [new branch] gh/oulgen/10/orig -> origin/gh/oulgen/10/orig 2025-12-04T08:27:07.2203777Z * [new branch] gh/oulgen/11/base -> origin/gh/oulgen/11/base 2025-12-04T08:27:07.2204114Z * [new branch] gh/oulgen/11/head -> origin/gh/oulgen/11/head 2025-12-04T08:27:07.2204359Z * [new branch] gh/oulgen/11/orig -> origin/gh/oulgen/11/orig 2025-12-04T08:27:07.2204602Z * [new branch] gh/oulgen/12/base -> origin/gh/oulgen/12/base 2025-12-04T08:27:07.2204821Z * [new branch] gh/oulgen/12/head -> origin/gh/oulgen/12/head 2025-12-04T08:27:07.2204986Z * [new branch] gh/oulgen/12/orig -> origin/gh/oulgen/12/orig 2025-12-04T08:27:07.2206409Z * [new branch] gh/oulgen/13/base -> origin/gh/oulgen/13/base 2025-12-04T08:27:07.2206736Z * [new branch] gh/oulgen/13/head -> origin/gh/oulgen/13/head 2025-12-04T08:27:07.2207144Z * [new branch] gh/oulgen/13/orig -> origin/gh/oulgen/13/orig 2025-12-04T08:27:07.2209289Z * [new branch] gh/oulgen/14/base -> origin/gh/oulgen/14/base 2025-12-04T08:27:07.2209626Z * [new branch] gh/oulgen/14/head -> origin/gh/oulgen/14/head 2025-12-04T08:27:07.2215067Z * [new branch] gh/oulgen/14/orig -> origin/gh/oulgen/14/orig 2025-12-04T08:27:07.2215246Z * [new branch] gh/oulgen/15/base -> origin/gh/oulgen/15/base 2025-12-04T08:27:07.2215392Z * [new branch] gh/oulgen/15/head -> origin/gh/oulgen/15/head 2025-12-04T08:27:07.2215536Z * [new branch] gh/oulgen/15/orig -> origin/gh/oulgen/15/orig 2025-12-04T08:27:07.2215688Z * [new branch] gh/oulgen/16/base -> origin/gh/oulgen/16/base 2025-12-04T08:27:07.2215835Z * [new branch] gh/oulgen/16/head -> origin/gh/oulgen/16/head 2025-12-04T08:27:07.2215970Z * [new branch] gh/oulgen/16/orig -> origin/gh/oulgen/16/orig 2025-12-04T08:27:07.2216108Z * [new branch] gh/oulgen/17/base -> origin/gh/oulgen/17/base 2025-12-04T08:27:07.2216253Z * [new branch] gh/oulgen/17/head -> origin/gh/oulgen/17/head 2025-12-04T08:27:07.2216443Z * [new branch] gh/oulgen/17/orig -> origin/gh/oulgen/17/orig 2025-12-04T08:27:07.2221371Z * [new branch] gh/oulgen/18/base -> origin/gh/oulgen/18/base 2025-12-04T08:27:07.2221559Z * [new branch] gh/oulgen/18/head -> origin/gh/oulgen/18/head 2025-12-04T08:27:07.2221711Z * [new branch] gh/oulgen/18/orig -> origin/gh/oulgen/18/orig 2025-12-04T08:27:07.2222243Z * [new branch] gh/oulgen/19/base -> origin/gh/oulgen/19/base 2025-12-04T08:27:07.2222397Z * [new branch] gh/oulgen/19/head -> origin/gh/oulgen/19/head 2025-12-04T08:27:07.2224956Z * [new branch] gh/oulgen/19/orig -> origin/gh/oulgen/19/orig 2025-12-04T08:27:07.2226521Z * [new branch] gh/oulgen/20/base -> origin/gh/oulgen/20/base 2025-12-04T08:27:07.2227605Z * [new branch] gh/oulgen/20/head -> origin/gh/oulgen/20/head 2025-12-04T08:27:07.2228068Z * [new branch] gh/oulgen/20/orig -> origin/gh/oulgen/20/orig 2025-12-04T08:27:07.2229441Z * [new branch] gh/oulgen/21/base -> origin/gh/oulgen/21/base 2025-12-04T08:27:07.2229618Z * [new branch] gh/oulgen/21/head -> origin/gh/oulgen/21/head 2025-12-04T08:27:07.2231871Z * [new branch] gh/oulgen/21/orig -> origin/gh/oulgen/21/orig 2025-12-04T08:27:07.2232074Z * [new branch] gh/oulgen/22/base -> origin/gh/oulgen/22/base 2025-12-04T08:27:07.2232215Z * [new branch] gh/oulgen/22/head -> origin/gh/oulgen/22/head 2025-12-04T08:27:07.2232824Z * [new branch] gh/oulgen/22/orig -> origin/gh/oulgen/22/orig 2025-12-04T08:27:07.2234192Z * [new branch] gh/oulgen/23/base -> origin/gh/oulgen/23/base 2025-12-04T08:27:07.2234449Z * [new branch] gh/oulgen/23/head -> origin/gh/oulgen/23/head 2025-12-04T08:27:07.2235518Z * [new branch] gh/oulgen/23/orig -> origin/gh/oulgen/23/orig 2025-12-04T08:27:07.2236170Z * [new branch] gh/oulgen/24/base -> origin/gh/oulgen/24/base 2025-12-04T08:27:07.2237097Z * [new branch] gh/oulgen/24/head -> origin/gh/oulgen/24/head 2025-12-04T08:27:07.2237587Z * [new branch] gh/oulgen/24/orig -> origin/gh/oulgen/24/orig 2025-12-04T08:27:07.2239446Z * [new branch] gh/oulgen/25/base -> origin/gh/oulgen/25/base 2025-12-04T08:27:07.2239772Z * [new branch] gh/oulgen/25/head -> origin/gh/oulgen/25/head 2025-12-04T08:27:07.2240829Z * [new branch] gh/oulgen/25/orig -> origin/gh/oulgen/25/orig 2025-12-04T08:27:07.2241576Z * [new branch] gh/oulgen/26/base -> origin/gh/oulgen/26/base 2025-12-04T08:27:07.2242183Z * [new branch] gh/oulgen/26/head -> origin/gh/oulgen/26/head 2025-12-04T08:27:07.2243012Z * [new branch] gh/oulgen/26/orig -> origin/gh/oulgen/26/orig 2025-12-04T08:27:07.2244209Z * [new branch] gh/oulgen/4/base -> origin/gh/oulgen/4/base 2025-12-04T08:27:07.2244493Z * [new branch] gh/oulgen/4/head -> origin/gh/oulgen/4/head 2025-12-04T08:27:07.2245448Z * [new branch] gh/oulgen/4/orig -> origin/gh/oulgen/4/orig 2025-12-04T08:27:07.2246856Z * [new branch] gh/oulgen/7/base -> origin/gh/oulgen/7/base 2025-12-04T08:27:07.2247214Z * [new branch] gh/oulgen/7/head -> origin/gh/oulgen/7/head 2025-12-04T08:27:07.2248330Z * [new branch] gh/oulgen/7/orig -> origin/gh/oulgen/7/orig 2025-12-04T08:27:07.2249199Z * [new branch] gh/oulgen/8/base -> origin/gh/oulgen/8/base 2025-12-04T08:27:07.2249606Z * [new branch] gh/oulgen/8/head -> origin/gh/oulgen/8/head 2025-12-04T08:27:07.2250642Z * [new branch] gh/oulgen/8/orig -> origin/gh/oulgen/8/orig 2025-12-04T08:27:07.2251600Z * [new branch] gh/oulgen/9/base -> origin/gh/oulgen/9/base 2025-12-04T08:27:07.2251825Z * [new branch] gh/oulgen/9/head -> origin/gh/oulgen/9/head 2025-12-04T08:27:07.2254154Z * [new branch] gh/oulgen/9/orig -> origin/gh/oulgen/9/orig 2025-12-04T08:27:07.2254607Z * [new branch] gh/patvig/mtia-serialization -> origin/gh/patvig/mtia-serialization 2025-12-04T08:27:07.2255488Z * [new branch] gh/pearu/108/base -> origin/gh/pearu/108/base 2025-12-04T08:27:07.2255995Z * [new branch] gh/pearu/108/head -> origin/gh/pearu/108/head 2025-12-04T08:27:07.2256868Z * [new branch] gh/pearu/108/orig -> origin/gh/pearu/108/orig 2025-12-04T08:27:07.2261284Z * [new branch] gh/pearu/109/base -> origin/gh/pearu/109/base 2025-12-04T08:27:07.2261446Z * [new branch] gh/pearu/109/head -> origin/gh/pearu/109/head 2025-12-04T08:27:07.2261585Z * [new branch] gh/pearu/109/orig -> origin/gh/pearu/109/orig 2025-12-04T08:27:07.2261713Z * [new branch] gh/pearu/110/base -> origin/gh/pearu/110/base 2025-12-04T08:27:07.2261842Z * [new branch] gh/pearu/110/head -> origin/gh/pearu/110/head 2025-12-04T08:27:07.2262163Z * [new branch] gh/pearu/110/orig -> origin/gh/pearu/110/orig 2025-12-04T08:27:07.2264189Z * [new branch] gh/pearu/111/base -> origin/gh/pearu/111/base 2025-12-04T08:27:07.2264516Z * [new branch] gh/pearu/111/head -> origin/gh/pearu/111/head 2025-12-04T08:27:07.2264708Z * [new branch] gh/pearu/111/orig -> origin/gh/pearu/111/orig 2025-12-04T08:27:07.2269068Z * [new branch] gh/pearu/112/base -> origin/gh/pearu/112/base 2025-12-04T08:27:07.2269419Z * [new branch] gh/pearu/112/head -> origin/gh/pearu/112/head 2025-12-04T08:27:07.2269594Z * [new branch] gh/pearu/112/orig -> origin/gh/pearu/112/orig 2025-12-04T08:27:07.2269738Z * [new branch] gh/pearu/115/base -> origin/gh/pearu/115/base 2025-12-04T08:27:07.2269862Z * [new branch] gh/pearu/115/head -> origin/gh/pearu/115/head 2025-12-04T08:27:07.2270135Z * [new branch] gh/pearu/115/orig -> origin/gh/pearu/115/orig 2025-12-04T08:27:07.2270870Z * [new branch] gh/pearu/116/base -> origin/gh/pearu/116/base 2025-12-04T08:27:07.2271224Z * [new branch] gh/pearu/116/head -> origin/gh/pearu/116/head 2025-12-04T08:27:07.2271798Z * [new branch] gh/pearu/116/orig -> origin/gh/pearu/116/orig 2025-12-04T08:27:07.2272963Z * [new branch] gh/pearu/117/base -> origin/gh/pearu/117/base 2025-12-04T08:27:07.2273589Z * [new branch] gh/pearu/117/head -> origin/gh/pearu/117/head 2025-12-04T08:27:07.2274086Z * [new branch] gh/pearu/117/orig -> origin/gh/pearu/117/orig 2025-12-04T08:27:07.2275243Z * [new branch] gh/pearu/118/base -> origin/gh/pearu/118/base 2025-12-04T08:27:07.2275534Z * [new branch] gh/pearu/118/head -> origin/gh/pearu/118/head 2025-12-04T08:27:07.2276404Z * [new branch] gh/pearu/118/orig -> origin/gh/pearu/118/orig 2025-12-04T08:27:07.2277286Z * [new branch] gh/pearu/119/base -> origin/gh/pearu/119/base 2025-12-04T08:27:07.2278468Z * [new branch] gh/pearu/119/head -> origin/gh/pearu/119/head 2025-12-04T08:27:07.2278911Z * [new branch] gh/pearu/119/orig -> origin/gh/pearu/119/orig 2025-12-04T08:27:07.2283813Z * [new branch] gh/pearu/139/base -> origin/gh/pearu/139/base 2025-12-04T08:27:07.2283979Z * [new branch] gh/pearu/139/head -> origin/gh/pearu/139/head 2025-12-04T08:27:07.2284119Z * [new branch] gh/pearu/139/orig -> origin/gh/pearu/139/orig 2025-12-04T08:27:07.2284257Z * [new branch] gh/pearu/140/base -> origin/gh/pearu/140/base 2025-12-04T08:27:07.2284378Z * [new branch] gh/pearu/140/head -> origin/gh/pearu/140/head 2025-12-04T08:27:07.2284661Z * [new branch] gh/pearu/140/orig -> origin/gh/pearu/140/orig 2025-12-04T08:27:07.2284956Z * [new branch] gh/pearu/142/base -> origin/gh/pearu/142/base 2025-12-04T08:27:07.2285109Z * [new branch] gh/pearu/142/head -> origin/gh/pearu/142/head 2025-12-04T08:27:07.2286671Z * [new branch] gh/pearu/142/orig -> origin/gh/pearu/142/orig 2025-12-04T08:27:07.2286943Z * [new branch] gh/pearu/143/base -> origin/gh/pearu/143/base 2025-12-04T08:27:07.2289677Z * [new branch] gh/pearu/143/head -> origin/gh/pearu/143/head 2025-12-04T08:27:07.2290003Z * [new branch] gh/pearu/143/orig -> origin/gh/pearu/143/orig 2025-12-04T08:27:07.2290173Z * [new branch] gh/pearu/147/base -> origin/gh/pearu/147/base 2025-12-04T08:27:07.2290349Z * [new branch] gh/pearu/147/head -> origin/gh/pearu/147/head 2025-12-04T08:27:07.2290513Z * [new branch] gh/pearu/147/orig -> origin/gh/pearu/147/orig 2025-12-04T08:27:07.2292128Z * [new branch] gh/pearu/149/base -> origin/gh/pearu/149/base 2025-12-04T08:27:07.2292424Z * [new branch] gh/pearu/149/head -> origin/gh/pearu/149/head 2025-12-04T08:27:07.2292728Z * [new branch] gh/pearu/149/orig -> origin/gh/pearu/149/orig 2025-12-04T08:27:07.2294555Z * [new branch] gh/pearu/150/base -> origin/gh/pearu/150/base 2025-12-04T08:27:07.2294721Z * [new branch] gh/pearu/150/head -> origin/gh/pearu/150/head 2025-12-04T08:27:07.2295578Z * [new branch] gh/pearu/150/orig -> origin/gh/pearu/150/orig 2025-12-04T08:27:07.2296793Z * [new branch] gh/pearu/151/base -> origin/gh/pearu/151/base 2025-12-04T08:27:07.2296925Z * [new branch] gh/pearu/151/head -> origin/gh/pearu/151/head 2025-12-04T08:27:07.2297646Z * [new branch] gh/pearu/151/orig -> origin/gh/pearu/151/orig 2025-12-04T08:27:07.2301289Z * [new branch] gh/pearu/152/base -> origin/gh/pearu/152/base 2025-12-04T08:27:07.2301442Z * [new branch] gh/pearu/152/head -> origin/gh/pearu/152/head 2025-12-04T08:27:07.2301728Z * [new branch] gh/pearu/152/orig -> origin/gh/pearu/152/orig 2025-12-04T08:27:07.2301854Z * [new branch] gh/pearu/153/base -> origin/gh/pearu/153/base 2025-12-04T08:27:07.2301985Z * [new branch] gh/pearu/153/head -> origin/gh/pearu/153/head 2025-12-04T08:27:07.2302335Z * [new branch] gh/pearu/153/orig -> origin/gh/pearu/153/orig 2025-12-04T08:27:07.2302963Z * [new branch] gh/pearu/154/base -> origin/gh/pearu/154/base 2025-12-04T08:27:07.2306127Z * [new branch] gh/pearu/154/head -> origin/gh/pearu/154/head 2025-12-04T08:27:07.2306333Z * [new branch] gh/pearu/154/orig -> origin/gh/pearu/154/orig 2025-12-04T08:27:07.2306490Z * [new branch] gh/pearu/155/base -> origin/gh/pearu/155/base 2025-12-04T08:27:07.2306635Z * [new branch] gh/pearu/155/head -> origin/gh/pearu/155/head 2025-12-04T08:27:07.2306787Z * [new branch] gh/pearu/155/orig -> origin/gh/pearu/155/orig 2025-12-04T08:27:07.2307362Z * [new branch] gh/pearu/156/base -> origin/gh/pearu/156/base 2025-12-04T08:27:07.2307969Z * [new branch] gh/pearu/156/head -> origin/gh/pearu/156/head 2025-12-04T08:27:07.2308887Z * [new branch] gh/pearu/156/orig -> origin/gh/pearu/156/orig 2025-12-04T08:27:07.2314436Z * [new branch] gh/pearu/56/base -> origin/gh/pearu/56/base 2025-12-04T08:27:07.2314623Z * [new branch] gh/pearu/56/head -> origin/gh/pearu/56/head 2025-12-04T08:27:07.2314910Z * [new branch] gh/pearu/56/orig -> origin/gh/pearu/56/orig 2025-12-04T08:27:07.2315050Z * [new branch] gh/pearu/97/base -> origin/gh/pearu/97/base 2025-12-04T08:27:07.2315192Z * [new branch] gh/pearu/97/head -> origin/gh/pearu/97/head 2025-12-04T08:27:07.2315327Z * [new branch] gh/pearu/97/orig -> origin/gh/pearu/97/orig 2025-12-04T08:27:07.2315478Z * [new branch] gh/pianpwk/21/base -> origin/gh/pianpwk/21/base 2025-12-04T08:27:07.2315630Z * [new branch] gh/pianpwk/21/head -> origin/gh/pianpwk/21/head 2025-12-04T08:27:07.2318241Z * [new branch] gh/pianpwk/28/base -> origin/gh/pianpwk/28/base 2025-12-04T08:27:07.2318430Z * [new branch] gh/pianpwk/28/head -> origin/gh/pianpwk/28/head 2025-12-04T08:27:07.2318575Z * [new branch] gh/pianpwk/28/orig -> origin/gh/pianpwk/28/orig 2025-12-04T08:27:07.2319742Z * [new branch] gh/pianpwk/29/base -> origin/gh/pianpwk/29/base 2025-12-04T08:27:07.2320054Z * [new branch] gh/pianpwk/29/head -> origin/gh/pianpwk/29/head 2025-12-04T08:27:07.2321077Z * [new branch] gh/pianpwk/29/orig -> origin/gh/pianpwk/29/orig 2025-12-04T08:27:07.2324976Z * [new branch] gh/pianpwk/30/base -> origin/gh/pianpwk/30/base 2025-12-04T08:27:07.2325156Z * [new branch] gh/pianpwk/30/head -> origin/gh/pianpwk/30/head 2025-12-04T08:27:07.2325302Z * [new branch] gh/pianpwk/30/orig -> origin/gh/pianpwk/30/orig 2025-12-04T08:27:07.2325440Z * [new branch] gh/pianpwk/31/base -> origin/gh/pianpwk/31/base 2025-12-04T08:27:07.2325586Z * [new branch] gh/pianpwk/31/head -> origin/gh/pianpwk/31/head 2025-12-04T08:27:07.2326184Z * [new branch] gh/pianpwk/31/orig -> origin/gh/pianpwk/31/orig 2025-12-04T08:27:07.2327272Z * [new branch] gh/pianpwk/32/base -> origin/gh/pianpwk/32/base 2025-12-04T08:27:07.2327611Z * [new branch] gh/pianpwk/32/head -> origin/gh/pianpwk/32/head 2025-12-04T08:27:07.2328448Z * [new branch] gh/pianpwk/32/orig -> origin/gh/pianpwk/32/orig 2025-12-04T08:27:07.2329043Z * [new branch] gh/pianpwk/33/base -> origin/gh/pianpwk/33/base 2025-12-04T08:27:07.2331621Z * [new branch] gh/pianpwk/33/head -> origin/gh/pianpwk/33/head 2025-12-04T08:27:07.2331793Z * [new branch] gh/pianpwk/33/orig -> origin/gh/pianpwk/33/orig 2025-12-04T08:27:07.2331942Z * [new branch] gh/pianpwk/34/base -> origin/gh/pianpwk/34/base 2025-12-04T08:27:07.2332813Z * [new branch] gh/pianpwk/34/head -> origin/gh/pianpwk/34/head 2025-12-04T08:27:07.2333397Z * [new branch] gh/pianpwk/34/orig -> origin/gh/pianpwk/34/orig 2025-12-04T08:27:07.2337448Z * [new branch] gh/pianpwk/35/base -> origin/gh/pianpwk/35/base 2025-12-04T08:27:07.2337621Z * [new branch] gh/pianpwk/35/head -> origin/gh/pianpwk/35/head 2025-12-04T08:27:07.2337775Z * [new branch] gh/pianpwk/35/orig -> origin/gh/pianpwk/35/orig 2025-12-04T08:27:07.2337919Z * [new branch] gh/rec/141/base -> origin/gh/rec/141/base 2025-12-04T08:27:07.2338054Z * [new branch] gh/rec/141/head -> origin/gh/rec/141/head 2025-12-04T08:27:07.2338749Z * [new branch] gh/rec/153/base -> origin/gh/rec/153/base 2025-12-04T08:27:07.2343758Z * [new branch] gh/rec/153/head -> origin/gh/rec/153/head 2025-12-04T08:27:07.2344062Z * [new branch] gh/rec/153/orig -> origin/gh/rec/153/orig 2025-12-04T08:27:07.2344294Z * [new branch] gh/rec/154/base -> origin/gh/rec/154/base 2025-12-04T08:27:07.2344647Z * [new branch] gh/rec/154/head -> origin/gh/rec/154/head 2025-12-04T08:27:07.2344779Z * [new branch] gh/rec/154/orig -> origin/gh/rec/154/orig 2025-12-04T08:27:07.2344903Z * [new branch] gh/rec/164/base -> origin/gh/rec/164/base 2025-12-04T08:27:07.2345194Z * [new branch] gh/rec/164/head -> origin/gh/rec/164/head 2025-12-04T08:27:07.2345764Z * [new branch] gh/rec/164/orig -> origin/gh/rec/164/orig 2025-12-04T08:27:07.2346070Z * [new branch] gh/rec/166/base -> origin/gh/rec/166/base 2025-12-04T08:27:07.2349103Z * [new branch] gh/rec/166/head -> origin/gh/rec/166/head 2025-12-04T08:27:07.2349268Z * [new branch] gh/rec/166/orig -> origin/gh/rec/166/orig 2025-12-04T08:27:07.2349392Z * [new branch] gh/rec/167/base -> origin/gh/rec/167/base 2025-12-04T08:27:07.2349534Z * [new branch] gh/rec/167/head -> origin/gh/rec/167/head 2025-12-04T08:27:07.2349657Z * [new branch] gh/rec/167/orig -> origin/gh/rec/167/orig 2025-12-04T08:27:07.2353781Z * [new branch] gh/rec/168/base -> origin/gh/rec/168/base 2025-12-04T08:27:07.2353970Z * [new branch] gh/rec/168/head -> origin/gh/rec/168/head 2025-12-04T08:27:07.2354096Z * [new branch] gh/rec/168/orig -> origin/gh/rec/168/orig 2025-12-04T08:27:07.2354230Z * [new branch] gh/rec/169/base -> origin/gh/rec/169/base 2025-12-04T08:27:07.2354350Z * [new branch] gh/rec/169/head -> origin/gh/rec/169/head 2025-12-04T08:27:07.2354473Z * [new branch] gh/rec/169/orig -> origin/gh/rec/169/orig 2025-12-04T08:27:07.2355322Z * [new branch] gh/rec/170/base -> origin/gh/rec/170/base 2025-12-04T08:27:07.2355769Z * [new branch] gh/rec/170/head -> origin/gh/rec/170/head 2025-12-04T08:27:07.2359218Z * [new branch] gh/rec/170/orig -> origin/gh/rec/170/orig 2025-12-04T08:27:07.2359491Z * [new branch] gh/rec/171/base -> origin/gh/rec/171/base 2025-12-04T08:27:07.2359935Z * [new branch] gh/rec/171/head -> origin/gh/rec/171/head 2025-12-04T08:27:07.2360082Z * [new branch] gh/rec/171/orig -> origin/gh/rec/171/orig 2025-12-04T08:27:07.2360314Z * [new branch] gh/rec/172/base -> origin/gh/rec/172/base 2025-12-04T08:27:07.2362091Z * [new branch] gh/rec/172/head -> origin/gh/rec/172/head 2025-12-04T08:27:07.2362403Z * [new branch] gh/rec/172/orig -> origin/gh/rec/172/orig 2025-12-04T08:27:07.2362791Z * [new branch] gh/rec/173/base -> origin/gh/rec/173/base 2025-12-04T08:27:07.2365634Z * [new branch] gh/rec/173/head -> origin/gh/rec/173/head 2025-12-04T08:27:07.2365981Z * [new branch] gh/rec/173/orig -> origin/gh/rec/173/orig 2025-12-04T08:27:07.2366128Z * [new branch] gh/rec/174/base -> origin/gh/rec/174/base 2025-12-04T08:27:07.2366508Z * [new branch] gh/rec/174/head -> origin/gh/rec/174/head 2025-12-04T08:27:07.2366642Z * [new branch] gh/rec/174/orig -> origin/gh/rec/174/orig 2025-12-04T08:27:07.2368096Z * [new branch] gh/rec/175/base -> origin/gh/rec/175/base 2025-12-04T08:27:07.2368419Z * [new branch] gh/rec/175/head -> origin/gh/rec/175/head 2025-12-04T08:27:07.2368720Z * [new branch] gh/rec/175/orig -> origin/gh/rec/175/orig 2025-12-04T08:27:07.2373472Z * [new branch] gh/rec/176/base -> origin/gh/rec/176/base 2025-12-04T08:27:07.2378521Z * [new branch] gh/rec/176/head -> origin/gh/rec/176/head 2025-12-04T08:27:07.2380804Z * [new branch] gh/rec/176/orig -> origin/gh/rec/176/orig 2025-12-04T08:27:07.2381060Z * [new branch] gh/rec/177/base -> origin/gh/rec/177/base 2025-12-04T08:27:07.2387307Z * [new branch] gh/rec/177/head -> origin/gh/rec/177/head 2025-12-04T08:27:07.2387633Z * [new branch] gh/rec/177/orig -> origin/gh/rec/177/orig 2025-12-04T08:27:07.2387917Z * [new branch] gh/robert-hardwick/3/base -> origin/gh/robert-hardwick/3/base 2025-12-04T08:27:07.2388121Z * [new branch] gh/robert-hardwick/3/head -> origin/gh/robert-hardwick/3/head 2025-12-04T08:27:07.2388354Z * [new branch] gh/robert-hardwick/3/orig -> origin/gh/robert-hardwick/3/orig 2025-12-04T08:27:07.2389114Z * [new branch] gh/robert-hardwick/4/base -> origin/gh/robert-hardwick/4/base 2025-12-04T08:27:07.2389389Z * [new branch] gh/robert-hardwick/4/head -> origin/gh/robert-hardwick/4/head 2025-12-04T08:27:07.2389555Z * [new branch] gh/robert-hardwick/4/orig -> origin/gh/robert-hardwick/4/orig 2025-12-04T08:27:07.2389902Z * [new branch] gh/robert-hardwick/5/base -> origin/gh/robert-hardwick/5/base 2025-12-04T08:27:07.2390109Z * [new branch] gh/robert-hardwick/5/head -> origin/gh/robert-hardwick/5/head 2025-12-04T08:27:07.2390270Z * [new branch] gh/robert-hardwick/5/orig -> origin/gh/robert-hardwick/5/orig 2025-12-04T08:27:07.2390442Z * [new branch] gh/robert-hardwick/6/base -> origin/gh/robert-hardwick/6/base 2025-12-04T08:27:07.2390605Z * [new branch] gh/robert-hardwick/6/head -> origin/gh/robert-hardwick/6/head 2025-12-04T08:27:07.2390763Z * [new branch] gh/robert-hardwick/6/orig -> origin/gh/robert-hardwick/6/orig 2025-12-04T08:27:07.2390930Z * [new branch] gh/robert-hardwick/7/base -> origin/gh/robert-hardwick/7/base 2025-12-04T08:27:07.2391099Z * [new branch] gh/robert-hardwick/7/head -> origin/gh/robert-hardwick/7/head 2025-12-04T08:27:07.2391273Z * [new branch] gh/robert-hardwick/7/orig -> origin/gh/robert-hardwick/7/orig 2025-12-04T08:27:07.2391575Z * [new branch] gh/robert-hardwick/8/base -> origin/gh/robert-hardwick/8/base 2025-12-04T08:27:07.2391727Z * [new branch] gh/robert-hardwick/8/head -> origin/gh/robert-hardwick/8/head 2025-12-04T08:27:07.2391888Z * [new branch] gh/robert-hardwick/8/orig -> origin/gh/robert-hardwick/8/orig 2025-12-04T08:27:07.2392039Z * [new branch] gh/robert-hardwick/9/base -> origin/gh/robert-hardwick/9/base 2025-12-04T08:27:07.2392337Z * [new branch] gh/robert-hardwick/9/head -> origin/gh/robert-hardwick/9/head 2025-12-04T08:27:07.2392595Z * [new branch] gh/robert-hardwick/9/orig -> origin/gh/robert-hardwick/9/orig 2025-12-04T08:27:07.2392842Z * [new branch] gh/rtimpe/1/base -> origin/gh/rtimpe/1/base 2025-12-04T08:27:07.2393011Z * [new branch] gh/rtimpe/1/head -> origin/gh/rtimpe/1/head 2025-12-04T08:27:07.2395701Z * [new branch] gh/rtimpe/2/base -> origin/gh/rtimpe/2/base 2025-12-04T08:27:07.2395898Z * [new branch] gh/rtimpe/2/head -> origin/gh/rtimpe/2/head 2025-12-04T08:27:07.2396062Z * [new branch] gh/rtimpe/22/base -> origin/gh/rtimpe/22/base 2025-12-04T08:27:07.2396211Z * [new branch] gh/rtimpe/22/head -> origin/gh/rtimpe/22/head 2025-12-04T08:27:07.2396352Z * [new branch] gh/rtimpe/22/orig -> origin/gh/rtimpe/22/orig 2025-12-04T08:27:07.2396495Z * [new branch] gh/rtimpe/23/base -> origin/gh/rtimpe/23/base 2025-12-04T08:27:07.2396922Z * [new branch] gh/rtimpe/23/head -> origin/gh/rtimpe/23/head 2025-12-04T08:27:07.2397339Z * [new branch] gh/rtimpe/23/orig -> origin/gh/rtimpe/23/orig 2025-12-04T08:27:07.2398484Z * [new branch] gh/rtimpe/24/base -> origin/gh/rtimpe/24/base 2025-12-04T08:27:07.2398905Z * [new branch] gh/rtimpe/24/head -> origin/gh/rtimpe/24/head 2025-12-04T08:27:07.2399924Z * [new branch] gh/rtimpe/24/orig -> origin/gh/rtimpe/24/orig 2025-12-04T08:27:07.2404077Z * [new branch] gh/rtimpe/25/base -> origin/gh/rtimpe/25/base 2025-12-04T08:27:07.2404249Z * [new branch] gh/rtimpe/25/head -> origin/gh/rtimpe/25/head 2025-12-04T08:27:07.2404389Z * [new branch] gh/rtimpe/25/orig -> origin/gh/rtimpe/25/orig 2025-12-04T08:27:07.2404533Z * [new branch] gh/rtimpe/26/base -> origin/gh/rtimpe/26/base 2025-12-04T08:27:07.2404666Z * [new branch] gh/rtimpe/26/head -> origin/gh/rtimpe/26/head 2025-12-04T08:27:07.2404832Z * [new branch] gh/rtimpe/26/orig -> origin/gh/rtimpe/26/orig 2025-12-04T08:27:07.2405021Z * [new branch] gh/rtimpe/27/base -> origin/gh/rtimpe/27/base 2025-12-04T08:27:07.2406956Z * [new branch] gh/rtimpe/27/head -> origin/gh/rtimpe/27/head 2025-12-04T08:27:07.2407324Z * [new branch] gh/rtimpe/27/orig -> origin/gh/rtimpe/27/orig 2025-12-04T08:27:07.2407493Z * [new branch] gh/rtimpe/28/base -> origin/gh/rtimpe/28/base 2025-12-04T08:27:07.2407949Z * [new branch] gh/rtimpe/28/head -> origin/gh/rtimpe/28/head 2025-12-04T08:27:07.2410431Z * [new branch] gh/rtimpe/28/orig -> origin/gh/rtimpe/28/orig 2025-12-04T08:27:07.2410606Z * [new branch] gh/rtimpe/29/base -> origin/gh/rtimpe/29/base 2025-12-04T08:27:07.2410752Z * [new branch] gh/rtimpe/29/head -> origin/gh/rtimpe/29/head 2025-12-04T08:27:07.2410919Z * [new branch] gh/rtimpe/29/orig -> origin/gh/rtimpe/29/orig 2025-12-04T08:27:07.2416240Z * [new branch] gh/rtimpe/3/base -> origin/gh/rtimpe/3/base 2025-12-04T08:27:07.2416411Z * [new branch] gh/rtimpe/3/head -> origin/gh/rtimpe/3/head 2025-12-04T08:27:07.2416705Z * [new branch] gh/rtimpe/30/base -> origin/gh/rtimpe/30/base 2025-12-04T08:27:07.2416837Z * [new branch] gh/rtimpe/30/head -> origin/gh/rtimpe/30/head 2025-12-04T08:27:07.2416970Z * [new branch] gh/rtimpe/30/orig -> origin/gh/rtimpe/30/orig 2025-12-04T08:27:07.2417097Z * [new branch] gh/rtimpe/31/base -> origin/gh/rtimpe/31/base 2025-12-04T08:27:07.2417232Z * [new branch] gh/rtimpe/31/head -> origin/gh/rtimpe/31/head 2025-12-04T08:27:07.2417362Z * [new branch] gh/rtimpe/31/orig -> origin/gh/rtimpe/31/orig 2025-12-04T08:27:07.2417530Z * [new branch] gh/rtimpe/32/base -> origin/gh/rtimpe/32/base 2025-12-04T08:27:07.2418390Z * [new branch] gh/rtimpe/32/head -> origin/gh/rtimpe/32/head 2025-12-04T08:27:07.2418819Z * [new branch] gh/rtimpe/32/orig -> origin/gh/rtimpe/32/orig 2025-12-04T08:27:07.2423772Z * [new branch] gh/rtimpe/33/base -> origin/gh/rtimpe/33/base 2025-12-04T08:27:07.2424082Z * [new branch] gh/rtimpe/33/head -> origin/gh/rtimpe/33/head 2025-12-04T08:27:07.2424225Z * [new branch] gh/rtimpe/33/orig -> origin/gh/rtimpe/33/orig 2025-12-04T08:27:07.2424405Z * [new branch] gh/rtimpe/34/base -> origin/gh/rtimpe/34/base 2025-12-04T08:27:07.2424543Z * [new branch] gh/rtimpe/34/head -> origin/gh/rtimpe/34/head 2025-12-04T08:27:07.2424691Z * [new branch] gh/rtimpe/34/orig -> origin/gh/rtimpe/34/orig 2025-12-04T08:27:07.2425025Z * [new branch] gh/rtimpe/35/base -> origin/gh/rtimpe/35/base 2025-12-04T08:27:07.2425247Z * [new branch] gh/rtimpe/35/head -> origin/gh/rtimpe/35/head 2025-12-04T08:27:07.2429540Z * [new branch] gh/rtimpe/35/orig -> origin/gh/rtimpe/35/orig 2025-12-04T08:27:07.2429920Z * [new branch] gh/rtimpe/4/base -> origin/gh/rtimpe/4/base 2025-12-04T08:27:07.2430194Z * [new branch] gh/rtimpe/4/head -> origin/gh/rtimpe/4/head 2025-12-04T08:27:07.2430523Z * [new branch] gh/ruisizhang123/1/base -> origin/gh/ruisizhang123/1/base 2025-12-04T08:27:07.2430848Z * [new branch] gh/ruisizhang123/1/head -> origin/gh/ruisizhang123/1/head 2025-12-04T08:27:07.2431475Z * [new branch] gh/ruisizhang123/1/orig -> origin/gh/ruisizhang123/1/orig 2025-12-04T08:27:07.2431675Z * [new branch] gh/ruisizhang123/4/base -> origin/gh/ruisizhang123/4/base 2025-12-04T08:27:07.2432934Z * [new branch] gh/ruisizhang123/4/head -> origin/gh/ruisizhang123/4/head 2025-12-04T08:27:07.2433098Z * [new branch] gh/ruisizhang123/4/orig -> origin/gh/ruisizhang123/4/orig 2025-12-04T08:27:07.2434900Z * [new branch] gh/ruisizhang123/5/base -> origin/gh/ruisizhang123/5/base 2025-12-04T08:27:07.2435109Z * [new branch] gh/ruisizhang123/5/head -> origin/gh/ruisizhang123/5/head 2025-12-04T08:27:07.2435291Z * [new branch] gh/ruisizhang123/5/orig -> origin/gh/ruisizhang123/5/orig 2025-12-04T08:27:07.2435982Z * [new branch] gh/ruisizhang123/6/base -> origin/gh/ruisizhang123/6/base 2025-12-04T08:27:07.2436650Z * [new branch] gh/ruisizhang123/6/head -> origin/gh/ruisizhang123/6/head 2025-12-04T08:27:07.2437507Z * [new branch] gh/ruisizhang123/6/orig -> origin/gh/ruisizhang123/6/orig 2025-12-04T08:27:07.2438565Z * [new branch] gh/ruisizhang123/7/base -> origin/gh/ruisizhang123/7/base 2025-12-04T08:27:07.2439094Z * [new branch] gh/ruisizhang123/7/head -> origin/gh/ruisizhang123/7/head 2025-12-04T08:27:07.2440958Z * [new branch] gh/ruisizhang123/7/orig -> origin/gh/ruisizhang123/7/orig 2025-12-04T08:27:07.2441355Z * [new branch] gh/ruisizhang123/8/base -> origin/gh/ruisizhang123/8/base 2025-12-04T08:27:07.2441536Z * [new branch] gh/ruisizhang123/8/head -> origin/gh/ruisizhang123/8/head 2025-12-04T08:27:07.2442250Z * [new branch] gh/ruisizhang123/8/orig -> origin/gh/ruisizhang123/8/orig 2025-12-04T08:27:07.2445421Z * [new branch] gh/ruisizhang123/9/base -> origin/gh/ruisizhang123/9/base 2025-12-04T08:27:07.2445775Z * [new branch] gh/ruisizhang123/9/head -> origin/gh/ruisizhang123/9/head 2025-12-04T08:27:07.2446046Z * [new branch] gh/ruisizhang123/9/orig -> origin/gh/ruisizhang123/9/orig 2025-12-04T08:27:07.2446320Z * [new branch] gh/seemethere/52/base -> origin/gh/seemethere/52/base 2025-12-04T08:27:07.2446809Z * [new branch] gh/seemethere/52/head -> origin/gh/seemethere/52/head 2025-12-04T08:27:07.2448147Z * [new branch] gh/seemethere/52/orig -> origin/gh/seemethere/52/orig 2025-12-04T08:27:07.2448800Z * [new branch] gh/seemethere/53/base -> origin/gh/seemethere/53/base 2025-12-04T08:27:07.2449400Z * [new branch] gh/seemethere/53/head -> origin/gh/seemethere/53/head 2025-12-04T08:27:07.2449738Z * [new branch] gh/seemethere/53/orig -> origin/gh/seemethere/53/orig 2025-12-04T08:27:07.2452496Z * [new branch] gh/seemethere/54/base -> origin/gh/seemethere/54/base 2025-12-04T08:27:07.2457726Z * [new branch] gh/seemethere/54/head -> origin/gh/seemethere/54/head 2025-12-04T08:27:07.2461844Z * [new branch] gh/seemethere/54/orig -> origin/gh/seemethere/54/orig 2025-12-04T08:27:07.2466321Z * [new branch] gh/seemethere/55/base -> origin/gh/seemethere/55/base 2025-12-04T08:27:07.2466660Z * [new branch] gh/seemethere/55/head -> origin/gh/seemethere/55/head 2025-12-04T08:27:07.2472485Z * [new branch] gh/seemethere/55/orig -> origin/gh/seemethere/55/orig 2025-12-04T08:27:07.2472684Z * [new branch] gh/seemethere/59/base -> origin/gh/seemethere/59/base 2025-12-04T08:27:07.2472859Z * [new branch] gh/seemethere/59/head -> origin/gh/seemethere/59/head 2025-12-04T08:27:07.2473016Z * [new branch] gh/seemethere/59/orig -> origin/gh/seemethere/59/orig 2025-12-04T08:27:07.2473164Z * [new branch] gh/seemethere/62/base -> origin/gh/seemethere/62/base 2025-12-04T08:27:07.2473310Z * [new branch] gh/seemethere/62/head -> origin/gh/seemethere/62/head 2025-12-04T08:27:07.2473466Z * [new branch] gh/seemethere/62/orig -> origin/gh/seemethere/62/orig 2025-12-04T08:27:07.2473604Z * [new branch] gh/seemethere/63/base -> origin/gh/seemethere/63/base 2025-12-04T08:27:07.2473751Z * [new branch] gh/seemethere/63/head -> origin/gh/seemethere/63/head 2025-12-04T08:27:07.2473909Z * [new branch] gh/seemethere/63/orig -> origin/gh/seemethere/63/orig 2025-12-04T08:27:07.2474070Z * [new branch] gh/seemethere/71/base -> origin/gh/seemethere/71/base 2025-12-04T08:27:07.2474215Z * [new branch] gh/seemethere/71/head -> origin/gh/seemethere/71/head 2025-12-04T08:27:07.2474360Z * [new branch] gh/seemethere/71/orig -> origin/gh/seemethere/71/orig 2025-12-04T08:27:07.2474513Z * [new branch] gh/seemethere/72/base -> origin/gh/seemethere/72/base 2025-12-04T08:27:07.2474655Z * [new branch] gh/seemethere/72/head -> origin/gh/seemethere/72/head 2025-12-04T08:27:07.2474811Z * [new branch] gh/seemethere/72/orig -> origin/gh/seemethere/72/orig 2025-12-04T08:27:07.2474956Z * [new branch] gh/seemethere/73/base -> origin/gh/seemethere/73/base 2025-12-04T08:27:07.2475103Z * [new branch] gh/seemethere/73/head -> origin/gh/seemethere/73/head 2025-12-04T08:27:07.2475416Z * [new branch] gh/seemethere/73/orig -> origin/gh/seemethere/73/orig 2025-12-04T08:27:07.2475565Z * [new branch] gh/seemethere/74/base -> origin/gh/seemethere/74/base 2025-12-04T08:27:07.2475721Z * [new branch] gh/seemethere/74/head -> origin/gh/seemethere/74/head 2025-12-04T08:27:07.2475868Z * [new branch] gh/seemethere/74/orig -> origin/gh/seemethere/74/orig 2025-12-04T08:27:07.2476015Z * [new branch] gh/seemethere/75/base -> origin/gh/seemethere/75/base 2025-12-04T08:27:07.2476172Z * [new branch] gh/seemethere/75/head -> origin/gh/seemethere/75/head 2025-12-04T08:27:07.2476325Z * [new branch] gh/seemethere/75/orig -> origin/gh/seemethere/75/orig 2025-12-04T08:27:07.2476480Z * [new branch] gh/seemethere/76/base -> origin/gh/seemethere/76/base 2025-12-04T08:27:07.2476629Z * [new branch] gh/seemethere/76/head -> origin/gh/seemethere/76/head 2025-12-04T08:27:07.2476781Z * [new branch] gh/seemethere/76/orig -> origin/gh/seemethere/76/orig 2025-12-04T08:27:07.2476951Z * [new branch] gh/shunting314/145/base -> origin/gh/shunting314/145/base 2025-12-04T08:27:07.2477119Z * [new branch] gh/shunting314/145/head -> origin/gh/shunting314/145/head 2025-12-04T08:27:07.2478262Z * [new branch] gh/shunting314/145/orig -> origin/gh/shunting314/145/orig 2025-12-04T08:27:07.2482592Z * [new branch] gh/shunting314/176/base -> origin/gh/shunting314/176/base 2025-12-04T08:27:07.2483096Z * [new branch] gh/shunting314/176/head -> origin/gh/shunting314/176/head 2025-12-04T08:27:07.2483415Z * [new branch] gh/shunting314/176/orig -> origin/gh/shunting314/176/orig 2025-12-04T08:27:07.2483613Z * [new branch] gh/shunting314/249/base -> origin/gh/shunting314/249/base 2025-12-04T08:27:07.2483806Z * [new branch] gh/shunting314/249/head -> origin/gh/shunting314/249/head 2025-12-04T08:27:07.2483988Z * [new branch] gh/shunting314/249/orig -> origin/gh/shunting314/249/orig 2025-12-04T08:27:07.2484158Z * [new branch] gh/shunting314/253/base -> origin/gh/shunting314/253/base 2025-12-04T08:27:07.2485441Z * [new branch] gh/shunting314/253/head -> origin/gh/shunting314/253/head 2025-12-04T08:27:07.2485760Z * [new branch] gh/shunting314/253/orig -> origin/gh/shunting314/253/orig 2025-12-04T08:27:07.2489048Z * [new branch] gh/shunting314/256/base -> origin/gh/shunting314/256/base 2025-12-04T08:27:07.2489263Z * [new branch] gh/shunting314/256/head -> origin/gh/shunting314/256/head 2025-12-04T08:27:07.2489442Z * [new branch] gh/shunting314/256/orig -> origin/gh/shunting314/256/orig 2025-12-04T08:27:07.2489613Z * [new branch] gh/shunting314/257/base -> origin/gh/shunting314/257/base 2025-12-04T08:27:07.2490162Z * [new branch] gh/shunting314/257/head -> origin/gh/shunting314/257/head 2025-12-04T08:27:07.2490872Z * [new branch] gh/shunting314/257/orig -> origin/gh/shunting314/257/orig 2025-12-04T08:27:07.2492134Z * [new branch] gh/shunting314/258/base -> origin/gh/shunting314/258/base 2025-12-04T08:27:07.2492373Z * [new branch] gh/shunting314/258/head -> origin/gh/shunting314/258/head 2025-12-04T08:27:07.2493345Z * [new branch] gh/shunting314/258/orig -> origin/gh/shunting314/258/orig 2025-12-04T08:27:07.2493900Z * [new branch] gh/shunting314/259/base -> origin/gh/shunting314/259/base 2025-12-04T08:27:07.2494738Z * [new branch] gh/shunting314/259/head -> origin/gh/shunting314/259/head 2025-12-04T08:27:07.2495129Z * [new branch] gh/shunting314/259/orig -> origin/gh/shunting314/259/orig 2025-12-04T08:27:07.2496455Z * [new branch] gh/shunting314/260/base -> origin/gh/shunting314/260/base 2025-12-04T08:27:07.2497314Z * [new branch] gh/shunting314/260/head -> origin/gh/shunting314/260/head 2025-12-04T08:27:07.2497820Z * [new branch] gh/shunting314/260/orig -> origin/gh/shunting314/260/orig 2025-12-04T08:27:07.2498912Z * [new branch] gh/shunting314/261/base -> origin/gh/shunting314/261/base 2025-12-04T08:27:07.2499799Z * [new branch] gh/shunting314/261/head -> origin/gh/shunting314/261/head 2025-12-04T08:27:07.2500056Z * [new branch] gh/shunting314/261/orig -> origin/gh/shunting314/261/orig 2025-12-04T08:27:07.2501396Z * [new branch] gh/shunting314/262/base -> origin/gh/shunting314/262/base 2025-12-04T08:27:07.2501708Z * [new branch] gh/shunting314/262/head -> origin/gh/shunting314/262/head 2025-12-04T08:27:07.2502732Z * [new branch] gh/shunting314/262/orig -> origin/gh/shunting314/262/orig 2025-12-04T08:27:07.2503719Z * [new branch] gh/shunting314/263/base -> origin/gh/shunting314/263/base 2025-12-04T08:27:07.2504214Z * [new branch] gh/shunting314/263/head -> origin/gh/shunting314/263/head 2025-12-04T08:27:07.2505494Z * [new branch] gh/shunting314/263/orig -> origin/gh/shunting314/263/orig 2025-12-04T08:27:07.2506748Z * [new branch] gh/shunting314/264/base -> origin/gh/shunting314/264/base 2025-12-04T08:27:07.2507061Z * [new branch] gh/shunting314/264/head -> origin/gh/shunting314/264/head 2025-12-04T08:27:07.2508323Z * [new branch] gh/shunting314/264/orig -> origin/gh/shunting314/264/orig 2025-12-04T08:27:07.2508831Z * [new branch] gh/shunting314/265/base -> origin/gh/shunting314/265/base 2025-12-04T08:27:07.2509576Z * [new branch] gh/shunting314/265/head -> origin/gh/shunting314/265/head 2025-12-04T08:27:07.2510115Z * [new branch] gh/shunting314/265/orig -> origin/gh/shunting314/265/orig 2025-12-04T08:27:07.2511316Z * [new branch] gh/shunting314/266/base -> origin/gh/shunting314/266/base 2025-12-04T08:27:07.2511857Z * [new branch] gh/shunting314/266/head -> origin/gh/shunting314/266/head 2025-12-04T08:27:07.2512796Z * [new branch] gh/shunting314/266/orig -> origin/gh/shunting314/266/orig 2025-12-04T08:27:07.2513872Z * [new branch] gh/shunting314/267/base -> origin/gh/shunting314/267/base 2025-12-04T08:27:07.2514542Z * [new branch] gh/shunting314/267/head -> origin/gh/shunting314/267/head 2025-12-04T08:27:07.2515229Z * [new branch] gh/shunting314/267/orig -> origin/gh/shunting314/267/orig 2025-12-04T08:27:07.2516742Z * [new branch] gh/shunting314/268/base -> origin/gh/shunting314/268/base 2025-12-04T08:27:07.2519012Z * [new branch] gh/shunting314/268/head -> origin/gh/shunting314/268/head 2025-12-04T08:27:07.2519178Z * [new branch] gh/shunting314/268/orig -> origin/gh/shunting314/268/orig 2025-12-04T08:27:07.2519428Z * [new branch] gh/shunting314/269/base -> origin/gh/shunting314/269/base 2025-12-04T08:27:07.2519709Z * [new branch] gh/shunting314/269/head -> origin/gh/shunting314/269/head 2025-12-04T08:27:07.2521227Z * [new branch] gh/shunting314/269/orig -> origin/gh/shunting314/269/orig 2025-12-04T08:27:07.2521578Z * [new branch] gh/silverguo/1/base -> origin/gh/silverguo/1/base 2025-12-04T08:27:07.2525894Z * [new branch] gh/silverguo/1/head -> origin/gh/silverguo/1/head 2025-12-04T08:27:07.2526293Z * [new branch] gh/silverguo/2/base -> origin/gh/silverguo/2/base 2025-12-04T08:27:07.2526515Z * [new branch] gh/silverguo/2/head -> origin/gh/silverguo/2/head 2025-12-04T08:27:07.2526820Z * [new branch] gh/silverguo/3/base -> origin/gh/silverguo/3/base 2025-12-04T08:27:07.2526969Z * [new branch] gh/silverguo/3/head -> origin/gh/silverguo/3/head 2025-12-04T08:27:07.2527107Z * [new branch] gh/silverguo/4/base -> origin/gh/silverguo/4/base 2025-12-04T08:27:07.2532840Z * [new branch] gh/silverguo/4/head -> origin/gh/silverguo/4/head 2025-12-04T08:27:07.2533483Z * [new branch] gh/slayton58/39/base -> origin/gh/slayton58/39/base 2025-12-04T08:27:07.2533633Z * [new branch] gh/slayton58/39/head -> origin/gh/slayton58/39/head 2025-12-04T08:27:07.2533808Z * [new branch] gh/slayton58/39/orig -> origin/gh/slayton58/39/orig 2025-12-04T08:27:07.2533965Z * [new branch] gh/slayton58/42/base -> origin/gh/slayton58/42/base 2025-12-04T08:27:07.2534113Z * [new branch] gh/slayton58/42/head -> origin/gh/slayton58/42/head 2025-12-04T08:27:07.2534287Z * [new branch] gh/slayton58/42/orig -> origin/gh/slayton58/42/orig 2025-12-04T08:27:07.2534432Z * [new branch] gh/slayton58/43/base -> origin/gh/slayton58/43/base 2025-12-04T08:27:07.2534582Z * [new branch] gh/slayton58/43/head -> origin/gh/slayton58/43/head 2025-12-04T08:27:07.2539608Z * [new branch] gh/slayton58/43/orig -> origin/gh/slayton58/43/orig 2025-12-04T08:27:07.2545233Z * [new branch] gh/slayton58/44/base -> origin/gh/slayton58/44/base 2025-12-04T08:27:07.2549942Z * [new branch] gh/slayton58/44/head -> origin/gh/slayton58/44/head 2025-12-04T08:27:07.2554770Z * [new branch] gh/slayton58/44/orig -> origin/gh/slayton58/44/orig 2025-12-04T08:27:07.2554994Z * [new branch] gh/slayton58/45/base -> origin/gh/slayton58/45/base 2025-12-04T08:27:07.2555146Z * [new branch] gh/slayton58/45/head -> origin/gh/slayton58/45/head 2025-12-04T08:27:07.2555304Z * [new branch] gh/slayton58/45/orig -> origin/gh/slayton58/45/orig 2025-12-04T08:27:07.2555457Z * [new branch] gh/slayton58/46/base -> origin/gh/slayton58/46/base 2025-12-04T08:27:07.2555598Z * [new branch] gh/slayton58/46/head -> origin/gh/slayton58/46/head 2025-12-04T08:27:07.2555749Z * [new branch] gh/slayton58/46/orig -> origin/gh/slayton58/46/orig 2025-12-04T08:27:07.2555905Z * [new branch] gh/slayton58/6/base -> origin/gh/slayton58/6/base 2025-12-04T08:27:07.2556052Z * [new branch] gh/slayton58/6/head -> origin/gh/slayton58/6/head 2025-12-04T08:27:07.2556206Z * [new branch] gh/slayton58/7/base -> origin/gh/slayton58/7/base 2025-12-04T08:27:07.2556347Z * [new branch] gh/slayton58/7/head -> origin/gh/slayton58/7/head 2025-12-04T08:27:07.2556515Z * [new branch] gh/soulitzer/269/base -> origin/gh/soulitzer/269/base 2025-12-04T08:27:07.2556665Z * [new branch] gh/soulitzer/269/head -> origin/gh/soulitzer/269/head 2025-12-04T08:27:07.2556848Z * [new branch] gh/soulitzer/269/orig -> origin/gh/soulitzer/269/orig 2025-12-04T08:27:07.2557006Z * [new branch] gh/soulitzer/276/base -> origin/gh/soulitzer/276/base 2025-12-04T08:27:07.2557150Z * [new branch] gh/soulitzer/276/head -> origin/gh/soulitzer/276/head 2025-12-04T08:27:07.2557304Z * [new branch] gh/soulitzer/276/orig -> origin/gh/soulitzer/276/orig 2025-12-04T08:27:07.2557453Z * [new branch] gh/soulitzer/287/base -> origin/gh/soulitzer/287/base 2025-12-04T08:27:07.2557601Z * [new branch] gh/soulitzer/287/head -> origin/gh/soulitzer/287/head 2025-12-04T08:27:07.2557754Z * [new branch] gh/soulitzer/287/orig -> origin/gh/soulitzer/287/orig 2025-12-04T08:27:07.2557968Z * [new branch] gh/soulitzer/296/base -> origin/gh/soulitzer/296/base 2025-12-04T08:27:07.2558295Z * [new branch] gh/soulitzer/296/head -> origin/gh/soulitzer/296/head 2025-12-04T08:27:07.2558457Z * [new branch] gh/soulitzer/296/orig -> origin/gh/soulitzer/296/orig 2025-12-04T08:27:07.2558603Z * [new branch] gh/soulitzer/299/base -> origin/gh/soulitzer/299/base 2025-12-04T08:27:07.2558765Z * [new branch] gh/soulitzer/299/head -> origin/gh/soulitzer/299/head 2025-12-04T08:27:07.2558928Z * [new branch] gh/soulitzer/299/orig -> origin/gh/soulitzer/299/orig 2025-12-04T08:27:07.2559090Z * [new branch] gh/soulitzer/300/base -> origin/gh/soulitzer/300/base 2025-12-04T08:27:07.2559248Z * [new branch] gh/soulitzer/300/head -> origin/gh/soulitzer/300/head 2025-12-04T08:27:07.2559408Z * [new branch] gh/soulitzer/300/orig -> origin/gh/soulitzer/300/orig 2025-12-04T08:27:07.2559565Z * [new branch] gh/soulitzer/301/base -> origin/gh/soulitzer/301/base 2025-12-04T08:27:07.2565477Z * [new branch] gh/soulitzer/301/head -> origin/gh/soulitzer/301/head 2025-12-04T08:27:07.2565843Z * [new branch] gh/soulitzer/301/orig -> origin/gh/soulitzer/301/orig 2025-12-04T08:27:07.2566006Z * [new branch] gh/soulitzer/313/base -> origin/gh/soulitzer/313/base 2025-12-04T08:27:07.2566302Z * [new branch] gh/soulitzer/313/head -> origin/gh/soulitzer/313/head 2025-12-04T08:27:07.2566477Z * [new branch] gh/soulitzer/313/orig -> origin/gh/soulitzer/313/orig 2025-12-04T08:27:07.2566784Z * [new branch] gh/soulitzer/319/base -> origin/gh/soulitzer/319/base 2025-12-04T08:27:07.2566949Z * [new branch] gh/soulitzer/319/head -> origin/gh/soulitzer/319/head 2025-12-04T08:27:07.2567114Z * [new branch] gh/soulitzer/319/orig -> origin/gh/soulitzer/319/orig 2025-12-04T08:27:07.2567270Z * [new branch] gh/soulitzer/320/base -> origin/gh/soulitzer/320/base 2025-12-04T08:27:07.2567433Z * [new branch] gh/soulitzer/320/head -> origin/gh/soulitzer/320/head 2025-12-04T08:27:07.2567739Z * [new branch] gh/soulitzer/320/orig -> origin/gh/soulitzer/320/orig 2025-12-04T08:27:07.2572955Z * [new branch] gh/soulitzer/336/base -> origin/gh/soulitzer/336/base 2025-12-04T08:27:07.2573301Z * [new branch] gh/soulitzer/336/head -> origin/gh/soulitzer/336/head 2025-12-04T08:27:07.2573477Z * [new branch] gh/soulitzer/336/orig -> origin/gh/soulitzer/336/orig 2025-12-04T08:27:07.2573652Z * [new branch] gh/soulitzer/347/base -> origin/gh/soulitzer/347/base 2025-12-04T08:27:07.2573928Z * [new branch] gh/soulitzer/347/head -> origin/gh/soulitzer/347/head 2025-12-04T08:27:07.2574107Z * [new branch] gh/soulitzer/347/orig -> origin/gh/soulitzer/347/orig 2025-12-04T08:27:07.2574825Z * [new branch] gh/soulitzer/349/base -> origin/gh/soulitzer/349/base 2025-12-04T08:27:07.2575037Z * [new branch] gh/soulitzer/349/head -> origin/gh/soulitzer/349/head 2025-12-04T08:27:07.2575218Z * [new branch] gh/soulitzer/349/orig -> origin/gh/soulitzer/349/orig 2025-12-04T08:27:07.2575477Z * [new branch] gh/soulitzer/350/base -> origin/gh/soulitzer/350/base 2025-12-04T08:27:07.2579683Z * [new branch] gh/soulitzer/350/head -> origin/gh/soulitzer/350/head 2025-12-04T08:27:07.2580062Z * [new branch] gh/soulitzer/350/orig -> origin/gh/soulitzer/350/orig 2025-12-04T08:27:07.2580331Z * [new branch] gh/soulitzer/351/base -> origin/gh/soulitzer/351/base 2025-12-04T08:27:07.2580512Z * [new branch] gh/soulitzer/351/head -> origin/gh/soulitzer/351/head 2025-12-04T08:27:07.2580965Z * [new branch] gh/soulitzer/351/orig -> origin/gh/soulitzer/351/orig 2025-12-04T08:27:07.2581893Z * [new branch] gh/soulitzer/353/base -> origin/gh/soulitzer/353/base 2025-12-04T08:27:07.2582085Z * [new branch] gh/soulitzer/353/head -> origin/gh/soulitzer/353/head 2025-12-04T08:27:07.2582337Z * [new branch] gh/soulitzer/353/orig -> origin/gh/soulitzer/353/orig 2025-12-04T08:27:07.2582543Z * [new branch] gh/soulitzer/358/base -> origin/gh/soulitzer/358/base 2025-12-04T08:27:07.2582812Z * [new branch] gh/soulitzer/358/head -> origin/gh/soulitzer/358/head 2025-12-04T08:27:07.2583003Z * [new branch] gh/soulitzer/358/orig -> origin/gh/soulitzer/358/orig 2025-12-04T08:27:07.2585393Z * [new branch] gh/soulitzer/359/base -> origin/gh/soulitzer/359/base 2025-12-04T08:27:07.2585608Z * [new branch] gh/soulitzer/359/head -> origin/gh/soulitzer/359/head 2025-12-04T08:27:07.2585773Z * [new branch] gh/soulitzer/359/orig -> origin/gh/soulitzer/359/orig 2025-12-04T08:27:07.2589633Z * [new branch] gh/soulitzer/374/base -> origin/gh/soulitzer/374/base 2025-12-04T08:27:07.2589824Z * [new branch] gh/soulitzer/374/head -> origin/gh/soulitzer/374/head 2025-12-04T08:27:07.2590024Z * [new branch] gh/soulitzer/374/orig -> origin/gh/soulitzer/374/orig 2025-12-04T08:27:07.2590212Z * [new branch] gh/soulitzer/375/base -> origin/gh/soulitzer/375/base 2025-12-04T08:27:07.2590415Z * [new branch] gh/soulitzer/375/head -> origin/gh/soulitzer/375/head 2025-12-04T08:27:07.2590677Z * [new branch] gh/soulitzer/375/orig -> origin/gh/soulitzer/375/orig 2025-12-04T08:27:07.2593313Z * [new branch] gh/soulitzer/380/base -> origin/gh/soulitzer/380/base 2025-12-04T08:27:07.2593489Z * [new branch] gh/soulitzer/380/head -> origin/gh/soulitzer/380/head 2025-12-04T08:27:07.2593633Z * [new branch] gh/soulitzer/380/orig -> origin/gh/soulitzer/380/orig 2025-12-04T08:27:07.2593904Z * [new branch] gh/soulitzer/385/base -> origin/gh/soulitzer/385/base 2025-12-04T08:27:07.2594071Z * [new branch] gh/soulitzer/385/head -> origin/gh/soulitzer/385/head 2025-12-04T08:27:07.2594304Z * [new branch] gh/soulitzer/385/orig -> origin/gh/soulitzer/385/orig 2025-12-04T08:27:07.2595910Z * [new branch] gh/soulitzer/386/base -> origin/gh/soulitzer/386/base 2025-12-04T08:27:07.2596112Z * [new branch] gh/soulitzer/386/head -> origin/gh/soulitzer/386/head 2025-12-04T08:27:07.2596791Z * [new branch] gh/soulitzer/386/orig -> origin/gh/soulitzer/386/orig 2025-12-04T08:27:07.2597976Z * [new branch] gh/soulitzer/387/base -> origin/gh/soulitzer/387/base 2025-12-04T08:27:07.2598379Z * [new branch] gh/soulitzer/387/head -> origin/gh/soulitzer/387/head 2025-12-04T08:27:07.2599435Z * [new branch] gh/soulitzer/387/orig -> origin/gh/soulitzer/387/orig 2025-12-04T08:27:07.2600417Z * [new branch] gh/soulitzer/388/base -> origin/gh/soulitzer/388/base 2025-12-04T08:27:07.2600745Z * [new branch] gh/soulitzer/388/head -> origin/gh/soulitzer/388/head 2025-12-04T08:27:07.2601754Z * [new branch] gh/soulitzer/388/orig -> origin/gh/soulitzer/388/orig 2025-12-04T08:27:07.2602990Z * [new branch] gh/soulitzer/389/base -> origin/gh/soulitzer/389/base 2025-12-04T08:27:07.2603402Z * [new branch] gh/soulitzer/389/head -> origin/gh/soulitzer/389/head 2025-12-04T08:27:07.2604382Z * [new branch] gh/soulitzer/389/orig -> origin/gh/soulitzer/389/orig 2025-12-04T08:27:07.2605303Z * [new branch] gh/soulitzer/390/base -> origin/gh/soulitzer/390/base 2025-12-04T08:27:07.2605977Z * [new branch] gh/soulitzer/390/head -> origin/gh/soulitzer/390/head 2025-12-04T08:27:07.2606750Z * [new branch] gh/soulitzer/390/orig -> origin/gh/soulitzer/390/orig 2025-12-04T08:27:07.2607674Z * [new branch] gh/soulitzer/391/base -> origin/gh/soulitzer/391/base 2025-12-04T08:27:07.2608045Z * [new branch] gh/soulitzer/391/head -> origin/gh/soulitzer/391/head 2025-12-04T08:27:07.2609013Z * [new branch] gh/soulitzer/391/orig -> origin/gh/soulitzer/391/orig 2025-12-04T08:27:07.2610256Z * [new branch] gh/soulitzer/392/base -> origin/gh/soulitzer/392/base 2025-12-04T08:27:07.2610470Z * [new branch] gh/soulitzer/392/head -> origin/gh/soulitzer/392/head 2025-12-04T08:27:07.2611580Z * [new branch] gh/soulitzer/392/orig -> origin/gh/soulitzer/392/orig 2025-12-04T08:27:07.2612737Z * [new branch] gh/swolchok/728/next -> origin/gh/swolchok/728/next 2025-12-04T08:27:07.2613819Z * [new branch] gh/swolchok/819/base -> origin/gh/swolchok/819/base 2025-12-04T08:27:07.2614209Z * [new branch] gh/swolchok/819/head -> origin/gh/swolchok/819/head 2025-12-04T08:27:07.2615176Z * [new branch] gh/swolchok/819/orig -> origin/gh/swolchok/819/orig 2025-12-04T08:27:07.2617003Z * [new branch] gh/swolchok/824/base -> origin/gh/swolchok/824/base 2025-12-04T08:27:07.2617157Z * [new branch] gh/swolchok/824/head -> origin/gh/swolchok/824/head 2025-12-04T08:27:07.2617483Z * [new branch] gh/swolchok/824/orig -> origin/gh/swolchok/824/orig 2025-12-04T08:27:07.2619131Z * [new branch] gh/swolchok/829/base -> origin/gh/swolchok/829/base 2025-12-04T08:27:07.2619406Z * [new branch] gh/swolchok/829/head -> origin/gh/swolchok/829/head 2025-12-04T08:27:07.2619735Z * [new branch] gh/swolchok/829/orig -> origin/gh/swolchok/829/orig 2025-12-04T08:27:07.2621694Z * [new branch] gh/swolchok/839/base -> origin/gh/swolchok/839/base 2025-12-04T08:27:07.2621870Z * [new branch] gh/swolchok/839/head -> origin/gh/swolchok/839/head 2025-12-04T08:27:07.2622120Z * [new branch] gh/swolchok/839/orig -> origin/gh/swolchok/839/orig 2025-12-04T08:27:07.2626857Z * [new branch] gh/swolchok/841/base -> origin/gh/swolchok/841/base 2025-12-04T08:27:07.2627157Z * [new branch] gh/swolchok/841/head -> origin/gh/swolchok/841/head 2025-12-04T08:27:07.2627341Z * [new branch] gh/swolchok/841/orig -> origin/gh/swolchok/841/orig 2025-12-04T08:27:07.2627481Z * [new branch] gh/swolchok/842/base -> origin/gh/swolchok/842/base 2025-12-04T08:27:07.2627617Z * [new branch] gh/swolchok/842/head -> origin/gh/swolchok/842/head 2025-12-04T08:27:07.2629754Z * [new branch] gh/swolchok/842/orig -> origin/gh/swolchok/842/orig 2025-12-04T08:27:07.2629935Z * [new branch] gh/swolchok/845/base -> origin/gh/swolchok/845/base 2025-12-04T08:27:07.2630104Z * [new branch] gh/swolchok/845/head -> origin/gh/swolchok/845/head 2025-12-04T08:27:07.2630243Z * [new branch] gh/swolchok/845/orig -> origin/gh/swolchok/845/orig 2025-12-04T08:27:07.2631099Z * [new branch] gh/swolchok/848/base -> origin/gh/swolchok/848/base 2025-12-04T08:27:07.2635137Z * [new branch] gh/swolchok/848/head -> origin/gh/swolchok/848/head 2025-12-04T08:27:07.2635363Z * [new branch] gh/swolchok/848/orig -> origin/gh/swolchok/848/orig 2025-12-04T08:27:07.2635516Z * [new branch] gh/swolchok/856/base -> origin/gh/swolchok/856/base 2025-12-04T08:27:07.2635881Z * [new branch] gh/swolchok/856/head -> origin/gh/swolchok/856/head 2025-12-04T08:27:07.2636033Z * [new branch] gh/swolchok/856/orig -> origin/gh/swolchok/856/orig 2025-12-04T08:27:07.2636219Z * [new branch] gh/swolchok/860/base -> origin/gh/swolchok/860/base 2025-12-04T08:27:07.2637100Z * [new branch] gh/swolchok/860/head -> origin/gh/swolchok/860/head 2025-12-04T08:27:07.2637613Z * [new branch] gh/swolchok/860/orig -> origin/gh/swolchok/860/orig 2025-12-04T08:27:07.2642688Z * [new branch] gh/swolchok/861/base -> origin/gh/swolchok/861/base 2025-12-04T08:27:07.2643028Z * [new branch] gh/swolchok/861/head -> origin/gh/swolchok/861/head 2025-12-04T08:27:07.2643324Z * [new branch] gh/swolchok/861/orig -> origin/gh/swolchok/861/orig 2025-12-04T08:27:07.2648486Z * [new branch] gh/swolchok/862/base -> origin/gh/swolchok/862/base 2025-12-04T08:27:07.2653413Z * [new branch] gh/swolchok/862/head -> origin/gh/swolchok/862/head 2025-12-04T08:27:07.2653600Z * [new branch] gh/swolchok/862/orig -> origin/gh/swolchok/862/orig 2025-12-04T08:27:07.2653787Z * [new branch] gh/swolchok/863/base -> origin/gh/swolchok/863/base 2025-12-04T08:27:07.2653963Z * [new branch] gh/swolchok/863/head -> origin/gh/swolchok/863/head 2025-12-04T08:27:07.2654113Z * [new branch] gh/swolchok/863/orig -> origin/gh/swolchok/863/orig 2025-12-04T08:27:07.2654276Z * [new branch] gh/swolchok/864/base -> origin/gh/swolchok/864/base 2025-12-04T08:27:07.2654442Z * [new branch] gh/swolchok/864/head -> origin/gh/swolchok/864/head 2025-12-04T08:27:07.2654828Z * [new branch] gh/swolchok/864/orig -> origin/gh/swolchok/864/orig 2025-12-04T08:27:07.2655006Z * [new branch] gh/swolchok/865/base -> origin/gh/swolchok/865/base 2025-12-04T08:27:07.2655162Z * [new branch] gh/swolchok/865/head -> origin/gh/swolchok/865/head 2025-12-04T08:27:07.2655317Z * [new branch] gh/swolchok/865/orig -> origin/gh/swolchok/865/orig 2025-12-04T08:27:07.2655464Z * [new branch] gh/swolchok/866/base -> origin/gh/swolchok/866/base 2025-12-04T08:27:07.2655611Z * [new branch] gh/swolchok/866/head -> origin/gh/swolchok/866/head 2025-12-04T08:27:07.2655767Z * [new branch] gh/swolchok/866/orig -> origin/gh/swolchok/866/orig 2025-12-04T08:27:07.2655921Z * [new branch] gh/swolchok/867/base -> origin/gh/swolchok/867/base 2025-12-04T08:27:07.2656076Z * [new branch] gh/swolchok/867/head -> origin/gh/swolchok/867/head 2025-12-04T08:27:07.2656243Z * [new branch] gh/swolchok/867/orig -> origin/gh/swolchok/867/orig 2025-12-04T08:27:07.2656721Z * [new branch] gh/swolchok/868/base -> origin/gh/swolchok/868/base 2025-12-04T08:27:07.2656934Z * [new branch] gh/swolchok/868/head -> origin/gh/swolchok/868/head 2025-12-04T08:27:07.2663668Z * [new branch] gh/swolchok/868/orig -> origin/gh/swolchok/868/orig 2025-12-04T08:27:07.2663852Z * [new branch] gh/swolchok/869/base -> origin/gh/swolchok/869/base 2025-12-04T08:27:07.2663999Z * [new branch] gh/swolchok/869/head -> origin/gh/swolchok/869/head 2025-12-04T08:27:07.2664157Z * [new branch] gh/swolchok/869/orig -> origin/gh/swolchok/869/orig 2025-12-04T08:27:07.2664301Z * [new branch] gh/swolchok/870/base -> origin/gh/swolchok/870/base 2025-12-04T08:27:07.2665992Z * [new branch] gh/swolchok/870/head -> origin/gh/swolchok/870/head 2025-12-04T08:27:07.2666356Z * [new branch] gh/swolchok/870/orig -> origin/gh/swolchok/870/orig 2025-12-04T08:27:07.2666917Z * [new branch] gh/swolchok/871/base -> origin/gh/swolchok/871/base 2025-12-04T08:27:07.2667188Z * [new branch] gh/swolchok/871/head -> origin/gh/swolchok/871/head 2025-12-04T08:27:07.2667410Z * [new branch] gh/swolchok/871/orig -> origin/gh/swolchok/871/orig 2025-12-04T08:27:07.2667654Z * [new branch] gh/teja-rao/4/base -> origin/gh/teja-rao/4/base 2025-12-04T08:27:07.2672578Z * [new branch] gh/teja-rao/4/head -> origin/gh/teja-rao/4/head 2025-12-04T08:27:07.2672913Z * [new branch] gh/teja-rao/4/orig -> origin/gh/teja-rao/4/orig 2025-12-04T08:27:07.2673152Z * [new branch] gh/tianyu-l/2/base -> origin/gh/tianyu-l/2/base 2025-12-04T08:27:07.2673336Z * [new branch] gh/tianyu-l/2/head -> origin/gh/tianyu-l/2/head 2025-12-04T08:27:07.2673604Z * [new branch] gh/tianyu-l/2/orig -> origin/gh/tianyu-l/2/orig 2025-12-04T08:27:07.2673798Z * [new branch] gh/tianyu-l/3/base -> origin/gh/tianyu-l/3/base 2025-12-04T08:27:07.2674467Z * [new branch] gh/tianyu-l/3/orig -> origin/gh/tianyu-l/3/orig 2025-12-04T08:27:07.2674656Z * [new branch] gh/tianyu-l/4/base -> origin/gh/tianyu-l/4/base 2025-12-04T08:27:07.2674796Z * [new branch] gh/tianyu-l/4/head -> origin/gh/tianyu-l/4/head 2025-12-04T08:27:07.2674926Z * [new branch] gh/tianyu-l/4/orig -> origin/gh/tianyu-l/4/orig 2025-12-04T08:27:07.2675104Z * [new branch] gh/tugsbayasgalan/10/base -> origin/gh/tugsbayasgalan/10/base 2025-12-04T08:27:07.2675261Z * [new branch] gh/tugsbayasgalan/10/head -> origin/gh/tugsbayasgalan/10/head 2025-12-04T08:27:07.2675805Z * [new branch] gh/tugsbayasgalan/10/orig -> origin/gh/tugsbayasgalan/10/orig 2025-12-04T08:27:07.2677343Z * [new branch] gh/tugsbayasgalan/13/base -> origin/gh/tugsbayasgalan/13/base 2025-12-04T08:27:07.2677732Z * [new branch] gh/tugsbayasgalan/13/head -> origin/gh/tugsbayasgalan/13/head 2025-12-04T08:27:07.2678162Z * [new branch] gh/tugsbayasgalan/13/orig -> origin/gh/tugsbayasgalan/13/orig 2025-12-04T08:27:07.2685066Z * [new branch] gh/tugsbayasgalan/17/base -> origin/gh/tugsbayasgalan/17/base 2025-12-04T08:27:07.2690133Z * [new branch] gh/tugsbayasgalan/17/head -> origin/gh/tugsbayasgalan/17/head 2025-12-04T08:27:07.2690336Z * [new branch] gh/tugsbayasgalan/17/orig -> origin/gh/tugsbayasgalan/17/orig 2025-12-04T08:27:07.2690520Z * [new branch] gh/tugsbayasgalan/2/base -> origin/gh/tugsbayasgalan/2/base 2025-12-04T08:27:07.2690710Z * [new branch] gh/tugsbayasgalan/2/head -> origin/gh/tugsbayasgalan/2/head 2025-12-04T08:27:07.2690879Z * [new branch] gh/tugsbayasgalan/2/orig -> origin/gh/tugsbayasgalan/2/orig 2025-12-04T08:27:07.2691051Z * [new branch] gh/tugsbayasgalan/28/base -> origin/gh/tugsbayasgalan/28/base 2025-12-04T08:27:07.2691221Z * [new branch] gh/tugsbayasgalan/28/head -> origin/gh/tugsbayasgalan/28/head 2025-12-04T08:27:07.2691382Z * [new branch] gh/tugsbayasgalan/28/orig -> origin/gh/tugsbayasgalan/28/orig 2025-12-04T08:27:07.2691549Z * [new branch] gh/tugsbayasgalan/32/base -> origin/gh/tugsbayasgalan/32/base 2025-12-04T08:27:07.2691707Z * [new branch] gh/tugsbayasgalan/32/head -> origin/gh/tugsbayasgalan/32/head 2025-12-04T08:27:07.2691863Z * [new branch] gh/tugsbayasgalan/32/orig -> origin/gh/tugsbayasgalan/32/orig 2025-12-04T08:27:07.2692031Z * [new branch] gh/tugsbayasgalan/35/base -> origin/gh/tugsbayasgalan/35/base 2025-12-04T08:27:07.2692187Z * [new branch] gh/tugsbayasgalan/35/head -> origin/gh/tugsbayasgalan/35/head 2025-12-04T08:27:07.2697835Z * [new branch] gh/tugsbayasgalan/35/orig -> origin/gh/tugsbayasgalan/35/orig 2025-12-04T08:27:07.2703369Z * [new branch] gh/tugsbayasgalan/36/base -> origin/gh/tugsbayasgalan/36/base 2025-12-04T08:27:07.2703566Z * [new branch] gh/tugsbayasgalan/36/head -> origin/gh/tugsbayasgalan/36/head 2025-12-04T08:27:07.2703762Z * [new branch] gh/tugsbayasgalan/36/orig -> origin/gh/tugsbayasgalan/36/orig 2025-12-04T08:27:07.2703923Z * [new branch] gh/tugsbayasgalan/37/base -> origin/gh/tugsbayasgalan/37/base 2025-12-04T08:27:07.2704092Z * [new branch] gh/tugsbayasgalan/37/head -> origin/gh/tugsbayasgalan/37/head 2025-12-04T08:27:07.2704254Z * [new branch] gh/tugsbayasgalan/37/orig -> origin/gh/tugsbayasgalan/37/orig 2025-12-04T08:27:07.2704438Z * [new branch] gh/tugsbayasgalan/43/base -> origin/gh/tugsbayasgalan/43/base 2025-12-04T08:27:07.2704600Z * [new branch] gh/tugsbayasgalan/43/head -> origin/gh/tugsbayasgalan/43/head 2025-12-04T08:27:07.2704769Z * [new branch] gh/tugsbayasgalan/43/orig -> origin/gh/tugsbayasgalan/43/orig 2025-12-04T08:27:07.2704937Z * [new branch] gh/tugsbayasgalan/48/base -> origin/gh/tugsbayasgalan/48/base 2025-12-04T08:27:07.2705096Z * [new branch] gh/tugsbayasgalan/48/head -> origin/gh/tugsbayasgalan/48/head 2025-12-04T08:27:07.2705253Z * [new branch] gh/tugsbayasgalan/48/orig -> origin/gh/tugsbayasgalan/48/orig 2025-12-04T08:27:07.2705419Z * [new branch] gh/tugsbayasgalan/51/base -> origin/gh/tugsbayasgalan/51/base 2025-12-04T08:27:07.2705581Z * [new branch] gh/tugsbayasgalan/51/head -> origin/gh/tugsbayasgalan/51/head 2025-12-04T08:27:07.2705878Z * [new branch] gh/tugsbayasgalan/51/orig -> origin/gh/tugsbayasgalan/51/orig 2025-12-04T08:27:07.2706046Z * [new branch] gh/tugsbayasgalan/52/base -> origin/gh/tugsbayasgalan/52/base 2025-12-04T08:27:07.2706206Z * [new branch] gh/tugsbayasgalan/52/head -> origin/gh/tugsbayasgalan/52/head 2025-12-04T08:27:07.2706387Z * [new branch] gh/tugsbayasgalan/52/orig -> origin/gh/tugsbayasgalan/52/orig 2025-12-04T08:27:07.2706555Z * [new branch] gh/tugsbayasgalan/53/base -> origin/gh/tugsbayasgalan/53/base 2025-12-04T08:27:07.2706725Z * [new branch] gh/tugsbayasgalan/53/head -> origin/gh/tugsbayasgalan/53/head 2025-12-04T08:27:07.2707320Z * [new branch] gh/tugsbayasgalan/53/orig -> origin/gh/tugsbayasgalan/53/orig 2025-12-04T08:27:07.2707944Z * [new branch] gh/tugsbayasgalan/55/base -> origin/gh/tugsbayasgalan/55/base 2025-12-04T08:27:07.2710374Z * [new branch] gh/tugsbayasgalan/55/head -> origin/gh/tugsbayasgalan/55/head 2025-12-04T08:27:07.2713578Z * [new branch] gh/tugsbayasgalan/55/orig -> origin/gh/tugsbayasgalan/55/orig 2025-12-04T08:27:07.2713771Z * [new branch] gh/tugsbayasgalan/59/base -> origin/gh/tugsbayasgalan/59/base 2025-12-04T08:27:07.2713989Z * [new branch] gh/tugsbayasgalan/59/head -> origin/gh/tugsbayasgalan/59/head 2025-12-04T08:27:07.2714162Z * [new branch] gh/tugsbayasgalan/59/orig -> origin/gh/tugsbayasgalan/59/orig 2025-12-04T08:27:07.2714342Z * [new branch] gh/tugsbayasgalan/6/base -> origin/gh/tugsbayasgalan/6/base 2025-12-04T08:27:07.2714517Z * [new branch] gh/tugsbayasgalan/6/head -> origin/gh/tugsbayasgalan/6/head 2025-12-04T08:27:07.2714681Z * [new branch] gh/tugsbayasgalan/6/orig -> origin/gh/tugsbayasgalan/6/orig 2025-12-04T08:27:07.2715303Z * [new branch] gh/tugsbayasgalan/60/base -> origin/gh/tugsbayasgalan/60/base 2025-12-04T08:27:07.2719234Z * [new branch] gh/tugsbayasgalan/60/head -> origin/gh/tugsbayasgalan/60/head 2025-12-04T08:27:07.2719434Z * [new branch] gh/tugsbayasgalan/60/orig -> origin/gh/tugsbayasgalan/60/orig 2025-12-04T08:27:07.2719770Z * [new branch] gh/tugsbayasgalan/61/base -> origin/gh/tugsbayasgalan/61/base 2025-12-04T08:27:07.2719932Z * [new branch] gh/tugsbayasgalan/61/head -> origin/gh/tugsbayasgalan/61/head 2025-12-04T08:27:07.2720131Z * [new branch] gh/tugsbayasgalan/61/orig -> origin/gh/tugsbayasgalan/61/orig 2025-12-04T08:27:07.2724277Z * [new branch] gh/tugsbayasgalan/63/base -> origin/gh/tugsbayasgalan/63/base 2025-12-04T08:27:07.2724629Z * [new branch] gh/tugsbayasgalan/63/head -> origin/gh/tugsbayasgalan/63/head 2025-12-04T08:27:07.2724899Z * [new branch] gh/tugsbayasgalan/63/orig -> origin/gh/tugsbayasgalan/63/orig 2025-12-04T08:27:07.2725165Z * [new branch] gh/tugsbayasgalan/67/base -> origin/gh/tugsbayasgalan/67/base 2025-12-04T08:27:07.2725351Z * [new branch] gh/tugsbayasgalan/67/head -> origin/gh/tugsbayasgalan/67/head 2025-12-04T08:27:07.2725514Z * [new branch] gh/tugsbayasgalan/67/orig -> origin/gh/tugsbayasgalan/67/orig 2025-12-04T08:27:07.2727483Z * [new branch] gh/tugsbayasgalan/68/base -> origin/gh/tugsbayasgalan/68/base 2025-12-04T08:27:07.2727840Z * [new branch] gh/tugsbayasgalan/68/head -> origin/gh/tugsbayasgalan/68/head 2025-12-04T08:27:07.2733095Z * [new branch] gh/tugsbayasgalan/68/orig -> origin/gh/tugsbayasgalan/68/orig 2025-12-04T08:27:07.2733475Z * [new branch] gh/tugsbayasgalan/7/base -> origin/gh/tugsbayasgalan/7/base 2025-12-04T08:27:07.2733743Z * [new branch] gh/tugsbayasgalan/7/head -> origin/gh/tugsbayasgalan/7/head 2025-12-04T08:27:07.2733937Z * [new branch] gh/tugsbayasgalan/7/orig -> origin/gh/tugsbayasgalan/7/orig 2025-12-04T08:27:07.2734425Z * [new branch] gh/tugsbayasgalan/70/base -> origin/gh/tugsbayasgalan/70/base 2025-12-04T08:27:07.2735783Z * [new branch] gh/tugsbayasgalan/70/head -> origin/gh/tugsbayasgalan/70/head 2025-12-04T08:27:07.2736062Z * [new branch] gh/tugsbayasgalan/70/orig -> origin/gh/tugsbayasgalan/70/orig 2025-12-04T08:27:07.2736241Z * [new branch] gh/tugsbayasgalan/71/base -> origin/gh/tugsbayasgalan/71/base 2025-12-04T08:27:07.2736537Z * [new branch] gh/tugsbayasgalan/71/head -> origin/gh/tugsbayasgalan/71/head 2025-12-04T08:27:07.2736716Z * [new branch] gh/tugsbayasgalan/71/orig -> origin/gh/tugsbayasgalan/71/orig 2025-12-04T08:27:07.2742532Z * [new branch] gh/tugsbayasgalan/72/base -> origin/gh/tugsbayasgalan/72/base 2025-12-04T08:27:07.2742892Z * [new branch] gh/tugsbayasgalan/72/head -> origin/gh/tugsbayasgalan/72/head 2025-12-04T08:27:07.2743173Z * [new branch] gh/tugsbayasgalan/72/orig -> origin/gh/tugsbayasgalan/72/orig 2025-12-04T08:27:07.2743421Z * [new branch] gh/tugsbayasgalan/73/base -> origin/gh/tugsbayasgalan/73/base 2025-12-04T08:27:07.2744105Z * [new branch] gh/tugsbayasgalan/73/head -> origin/gh/tugsbayasgalan/73/head 2025-12-04T08:27:07.2744317Z * [new branch] gh/tugsbayasgalan/73/orig -> origin/gh/tugsbayasgalan/73/orig 2025-12-04T08:27:07.2744487Z * [new branch] gh/tugsbayasgalan/74/base -> origin/gh/tugsbayasgalan/74/base 2025-12-04T08:27:07.2744645Z * [new branch] gh/tugsbayasgalan/74/head -> origin/gh/tugsbayasgalan/74/head 2025-12-04T08:27:07.2744805Z * [new branch] gh/tugsbayasgalan/74/orig -> origin/gh/tugsbayasgalan/74/orig 2025-12-04T08:27:07.2746956Z * [new branch] gh/tugsbayasgalan/75/base -> origin/gh/tugsbayasgalan/75/base 2025-12-04T08:27:07.2747359Z * [new branch] gh/tugsbayasgalan/75/head -> origin/gh/tugsbayasgalan/75/head 2025-12-04T08:27:07.2747766Z * [new branch] gh/tugsbayasgalan/75/orig -> origin/gh/tugsbayasgalan/75/orig 2025-12-04T08:27:07.2748407Z * [new branch] gh/tugsbayasgalan/76/base -> origin/gh/tugsbayasgalan/76/base 2025-12-04T08:27:07.2748961Z * [new branch] gh/tugsbayasgalan/76/head -> origin/gh/tugsbayasgalan/76/head 2025-12-04T08:27:07.2751256Z * [new branch] gh/tugsbayasgalan/76/orig -> origin/gh/tugsbayasgalan/76/orig 2025-12-04T08:27:07.2751474Z * [new branch] gh/tugsbayasgalan/77/base -> origin/gh/tugsbayasgalan/77/base 2025-12-04T08:27:07.2751635Z * [new branch] gh/tugsbayasgalan/77/head -> origin/gh/tugsbayasgalan/77/head 2025-12-04T08:27:07.2751796Z * [new branch] gh/tugsbayasgalan/77/orig -> origin/gh/tugsbayasgalan/77/orig 2025-12-04T08:27:07.2751962Z * [new branch] gh/tugsbayasgalan/78/base -> origin/gh/tugsbayasgalan/78/base 2025-12-04T08:27:07.2752127Z * [new branch] gh/tugsbayasgalan/78/head -> origin/gh/tugsbayasgalan/78/head 2025-12-04T08:27:07.2752495Z * [new branch] gh/tugsbayasgalan/78/orig -> origin/gh/tugsbayasgalan/78/orig 2025-12-04T08:27:07.2755512Z * [new branch] gh/tugsbayasgalan/79/base -> origin/gh/tugsbayasgalan/79/base 2025-12-04T08:27:07.2756145Z * [new branch] gh/tugsbayasgalan/79/head -> origin/gh/tugsbayasgalan/79/head 2025-12-04T08:27:07.2756344Z * [new branch] gh/tugsbayasgalan/79/orig -> origin/gh/tugsbayasgalan/79/orig 2025-12-04T08:27:07.2756531Z * [new branch] gh/tugsbayasgalan/8/base -> origin/gh/tugsbayasgalan/8/base 2025-12-04T08:27:07.2756697Z * [new branch] gh/tugsbayasgalan/8/head -> origin/gh/tugsbayasgalan/8/head 2025-12-04T08:27:07.2756861Z * [new branch] gh/tugsbayasgalan/8/orig -> origin/gh/tugsbayasgalan/8/orig 2025-12-04T08:27:07.2757213Z * [new branch] gh/tugsbayasgalan/80/base -> origin/gh/tugsbayasgalan/80/base 2025-12-04T08:27:07.2757389Z * [new branch] gh/tugsbayasgalan/80/head -> origin/gh/tugsbayasgalan/80/head 2025-12-04T08:27:07.2758670Z * [new branch] gh/tugsbayasgalan/80/orig -> origin/gh/tugsbayasgalan/80/orig 2025-12-04T08:27:07.2758950Z * [new branch] gh/tugsbayasgalan/81/base -> origin/gh/tugsbayasgalan/81/base 2025-12-04T08:27:07.2768206Z * [new branch] gh/tugsbayasgalan/81/head -> origin/gh/tugsbayasgalan/81/head 2025-12-04T08:27:07.2768560Z * [new branch] gh/tugsbayasgalan/81/orig -> origin/gh/tugsbayasgalan/81/orig 2025-12-04T08:27:07.2768826Z * [new branch] gh/tugsbayasgalan/82/base -> origin/gh/tugsbayasgalan/82/base 2025-12-04T08:27:07.2769004Z * [new branch] gh/tugsbayasgalan/82/head -> origin/gh/tugsbayasgalan/82/head 2025-12-04T08:27:07.2769259Z * [new branch] gh/tugsbayasgalan/82/orig -> origin/gh/tugsbayasgalan/82/orig 2025-12-04T08:27:07.2769620Z * [new branch] gh/tugsbayasgalan/83/base -> origin/gh/tugsbayasgalan/83/base 2025-12-04T08:27:07.2770190Z * [new branch] gh/tugsbayasgalan/83/head -> origin/gh/tugsbayasgalan/83/head 2025-12-04T08:27:07.2774813Z * [new branch] gh/tugsbayasgalan/83/orig -> origin/gh/tugsbayasgalan/83/orig 2025-12-04T08:27:07.2779588Z * [new branch] gh/tugsbayasgalan/84/base -> origin/gh/tugsbayasgalan/84/base 2025-12-04T08:27:07.2781795Z * [new branch] gh/tugsbayasgalan/84/head -> origin/gh/tugsbayasgalan/84/head 2025-12-04T08:27:07.2782093Z * [new branch] gh/tugsbayasgalan/84/orig -> origin/gh/tugsbayasgalan/84/orig 2025-12-04T08:27:07.2786010Z * [new branch] gh/tugsbayasgalan/85/base -> origin/gh/tugsbayasgalan/85/base 2025-12-04T08:27:07.2786322Z * [new branch] gh/tugsbayasgalan/85/head -> origin/gh/tugsbayasgalan/85/head 2025-12-04T08:27:07.2789442Z * [new branch] gh/tugsbayasgalan/85/orig -> origin/gh/tugsbayasgalan/85/orig 2025-12-04T08:27:07.2789663Z * [new branch] gh/tugsbayasgalan/86/base -> origin/gh/tugsbayasgalan/86/base 2025-12-04T08:27:07.2789830Z * [new branch] gh/tugsbayasgalan/86/head -> origin/gh/tugsbayasgalan/86/head 2025-12-04T08:27:07.2790107Z * [new branch] gh/tugsbayasgalan/86/orig -> origin/gh/tugsbayasgalan/86/orig 2025-12-04T08:27:07.2790259Z * [new branch] gh/tugsbayasgalan/87/base -> origin/gh/tugsbayasgalan/87/base 2025-12-04T08:27:07.2790425Z * [new branch] gh/tugsbayasgalan/87/head -> origin/gh/tugsbayasgalan/87/head 2025-12-04T08:27:07.2790580Z * [new branch] gh/tugsbayasgalan/87/orig -> origin/gh/tugsbayasgalan/87/orig 2025-12-04T08:27:07.2790743Z * [new branch] gh/tugsbayasgalan/88/base -> origin/gh/tugsbayasgalan/88/base 2025-12-04T08:27:07.2790895Z * [new branch] gh/tugsbayasgalan/88/head -> origin/gh/tugsbayasgalan/88/head 2025-12-04T08:27:07.2791051Z * [new branch] gh/tugsbayasgalan/88/orig -> origin/gh/tugsbayasgalan/88/orig 2025-12-04T08:27:07.2791214Z * [new branch] gh/tugsbayasgalan/89/base -> origin/gh/tugsbayasgalan/89/base 2025-12-04T08:27:07.2791370Z * [new branch] gh/tugsbayasgalan/89/head -> origin/gh/tugsbayasgalan/89/head 2025-12-04T08:27:07.2791531Z * [new branch] gh/tugsbayasgalan/89/orig -> origin/gh/tugsbayasgalan/89/orig 2025-12-04T08:27:07.2791699Z * [new branch] gh/tugsbayasgalan/9/base -> origin/gh/tugsbayasgalan/9/base 2025-12-04T08:27:07.2791858Z * [new branch] gh/tugsbayasgalan/9/head -> origin/gh/tugsbayasgalan/9/head 2025-12-04T08:27:07.2792019Z * [new branch] gh/tugsbayasgalan/9/orig -> origin/gh/tugsbayasgalan/9/orig 2025-12-04T08:27:07.2792173Z * [new branch] gh/tugsbayasgalan/90/base -> origin/gh/tugsbayasgalan/90/base 2025-12-04T08:27:07.2792373Z * [new branch] gh/tugsbayasgalan/90/head -> origin/gh/tugsbayasgalan/90/head 2025-12-04T08:27:07.2792529Z * [new branch] gh/tugsbayasgalan/90/orig -> origin/gh/tugsbayasgalan/90/orig 2025-12-04T08:27:07.2792684Z * [new branch] gh/tugsbayasgalan/91/base -> origin/gh/tugsbayasgalan/91/base 2025-12-04T08:27:07.2792845Z * [new branch] gh/tugsbayasgalan/91/head -> origin/gh/tugsbayasgalan/91/head 2025-12-04T08:27:07.2793003Z * [new branch] gh/tugsbayasgalan/91/orig -> origin/gh/tugsbayasgalan/91/orig 2025-12-04T08:27:07.2793165Z * [new branch] gh/tugsbayasgalan/92/base -> origin/gh/tugsbayasgalan/92/base 2025-12-04T08:27:07.2793317Z * [new branch] gh/tugsbayasgalan/92/head -> origin/gh/tugsbayasgalan/92/head 2025-12-04T08:27:07.2793468Z * [new branch] gh/tugsbayasgalan/92/orig -> origin/gh/tugsbayasgalan/92/orig 2025-12-04T08:27:07.2793672Z * [new branch] gh/tugsbayasgalan/93/base -> origin/gh/tugsbayasgalan/93/base 2025-12-04T08:27:07.2794032Z * [new branch] gh/tugsbayasgalan/93/head -> origin/gh/tugsbayasgalan/93/head 2025-12-04T08:27:07.2794195Z * [new branch] gh/tugsbayasgalan/93/orig -> origin/gh/tugsbayasgalan/93/orig 2025-12-04T08:27:07.2794369Z * [new branch] gh/v0i0/14/base -> origin/gh/v0i0/14/base 2025-12-04T08:27:07.2794493Z * [new branch] gh/v0i0/14/head -> origin/gh/v0i0/14/head 2025-12-04T08:27:07.2794612Z * [new branch] gh/v0i0/14/orig -> origin/gh/v0i0/14/orig 2025-12-04T08:27:07.2794740Z * [new branch] gh/v0i0/15/base -> origin/gh/v0i0/15/base 2025-12-04T08:27:07.2794857Z * [new branch] gh/v0i0/15/head -> origin/gh/v0i0/15/head 2025-12-04T08:27:07.2795551Z * [new branch] gh/v0i0/15/orig -> origin/gh/v0i0/15/orig 2025-12-04T08:27:07.2796604Z * [new branch] gh/v0i0/16/base -> origin/gh/v0i0/16/base 2025-12-04T08:27:07.2796879Z * [new branch] gh/v0i0/16/head -> origin/gh/v0i0/16/head 2025-12-04T08:27:07.2798233Z * [new branch] gh/v0i0/16/orig -> origin/gh/v0i0/16/orig 2025-12-04T08:27:07.2798522Z * [new branch] gh/v0i0/17/base -> origin/gh/v0i0/17/base 2025-12-04T08:27:07.2802337Z * [new branch] gh/v0i0/17/head -> origin/gh/v0i0/17/head 2025-12-04T08:27:07.2802906Z * [new branch] gh/v0i0/17/orig -> origin/gh/v0i0/17/orig 2025-12-04T08:27:07.2803080Z * [new branch] gh/v0i0/18/base -> origin/gh/v0i0/18/base 2025-12-04T08:27:07.2803210Z * [new branch] gh/v0i0/18/head -> origin/gh/v0i0/18/head 2025-12-04T08:27:07.2803338Z * [new branch] gh/v0i0/18/orig -> origin/gh/v0i0/18/orig 2025-12-04T08:27:07.2803508Z * [new branch] gh/v0i0/19/base -> origin/gh/v0i0/19/base 2025-12-04T08:27:07.2804591Z * [new branch] gh/v0i0/19/head -> origin/gh/v0i0/19/head 2025-12-04T08:27:07.2804839Z * [new branch] gh/v0i0/19/orig -> origin/gh/v0i0/19/orig 2025-12-04T08:27:07.2806478Z * [new branch] gh/vishal9-team/1/base -> origin/gh/vishal9-team/1/base 2025-12-04T08:27:07.2806882Z * [new branch] gh/vishal9-team/1/head -> origin/gh/vishal9-team/1/head 2025-12-04T08:27:07.2808432Z * [new branch] gh/vishal9-team/2/base -> origin/gh/vishal9-team/2/base 2025-12-04T08:27:07.2808716Z * [new branch] gh/vishal9-team/2/head -> origin/gh/vishal9-team/2/head 2025-12-04T08:27:07.2811351Z * [new branch] gh/vishal9-team/2/orig -> origin/gh/vishal9-team/2/orig 2025-12-04T08:27:07.2811561Z * [new branch] gh/vishal9-team/3/base -> origin/gh/vishal9-team/3/base 2025-12-04T08:27:07.2811918Z * [new branch] gh/vishal9-team/3/head -> origin/gh/vishal9-team/3/head 2025-12-04T08:27:07.2812418Z * [new branch] gh/vishal9-team/3/orig -> origin/gh/vishal9-team/3/orig 2025-12-04T08:27:07.2813733Z * [new branch] gh/vishal9-team/4/base -> origin/gh/vishal9-team/4/base 2025-12-04T08:27:07.2813902Z * [new branch] gh/vishal9-team/4/head -> origin/gh/vishal9-team/4/head 2025-12-04T08:27:07.2814801Z * [new branch] gh/vishal9-team/4/orig -> origin/gh/vishal9-team/4/orig 2025-12-04T08:27:07.2815970Z * [new branch] gh/vkuzo/1/next -> origin/gh/vkuzo/1/next 2025-12-04T08:27:07.2816873Z * [new branch] gh/vkuzo/2/next -> origin/gh/vkuzo/2/next 2025-12-04T08:27:07.2817863Z * [new branch] gh/vkuzo/3/next -> origin/gh/vkuzo/3/next 2025-12-04T08:27:07.2818847Z * [new branch] gh/wconstab/424/base -> origin/gh/wconstab/424/base 2025-12-04T08:27:07.2819328Z * [new branch] gh/wconstab/424/head -> origin/gh/wconstab/424/head 2025-12-04T08:27:07.2820200Z * [new branch] gh/wconstab/424/orig -> origin/gh/wconstab/424/orig 2025-12-04T08:27:07.2821262Z * [new branch] gh/wconstab/435/base -> origin/gh/wconstab/435/base 2025-12-04T08:27:07.2824306Z * [new branch] gh/wconstab/435/head -> origin/gh/wconstab/435/head 2025-12-04T08:27:07.2824594Z * [new branch] gh/wconstab/435/orig -> origin/gh/wconstab/435/orig 2025-12-04T08:27:07.2826062Z * [new branch] gh/wconstab/444/base -> origin/gh/wconstab/444/base 2025-12-04T08:27:07.2826441Z * [new branch] gh/wconstab/444/head -> origin/gh/wconstab/444/head 2025-12-04T08:27:07.2827618Z * [new branch] gh/wconstab/444/orig -> origin/gh/wconstab/444/orig 2025-12-04T08:27:07.2831107Z * [new branch] gh/wconstab/447/base -> origin/gh/wconstab/447/base 2025-12-04T08:27:07.2831650Z * [new branch] gh/wconstab/447/head -> origin/gh/wconstab/447/head 2025-12-04T08:27:07.2831833Z * [new branch] gh/wconstab/447/orig -> origin/gh/wconstab/447/orig 2025-12-04T08:27:07.2832192Z * [new branch] gh/wconstab/448/base -> origin/gh/wconstab/448/base 2025-12-04T08:27:07.2832343Z * [new branch] gh/wconstab/448/head -> origin/gh/wconstab/448/head 2025-12-04T08:27:07.2835515Z * [new branch] gh/wconstab/448/orig -> origin/gh/wconstab/448/orig 2025-12-04T08:27:07.2835710Z * [new branch] gh/wconstab/449/base -> origin/gh/wconstab/449/base 2025-12-04T08:27:07.2835870Z * [new branch] gh/wconstab/449/head -> origin/gh/wconstab/449/head 2025-12-04T08:27:07.2836017Z * [new branch] gh/wconstab/449/orig -> origin/gh/wconstab/449/orig 2025-12-04T08:27:07.2836173Z * [new branch] gh/wconstab/450/base -> origin/gh/wconstab/450/base 2025-12-04T08:27:07.2836381Z * [new branch] gh/wconstab/450/head -> origin/gh/wconstab/450/head 2025-12-04T08:27:07.2837150Z * [new branch] gh/wconstab/450/orig -> origin/gh/wconstab/450/orig 2025-12-04T08:27:07.2838344Z * [new branch] gh/wconstab/451/base -> origin/gh/wconstab/451/base 2025-12-04T08:27:07.2842390Z * [new branch] gh/wconstab/451/head -> origin/gh/wconstab/451/head 2025-12-04T08:27:07.2842546Z * [new branch] gh/wconstab/451/orig -> origin/gh/wconstab/451/orig 2025-12-04T08:27:07.2842766Z * [new branch] gh/wconstab/452/base -> origin/gh/wconstab/452/base 2025-12-04T08:27:07.2847456Z * [new branch] gh/wconstab/452/head -> origin/gh/wconstab/452/head 2025-12-04T08:27:07.2852624Z * [new branch] gh/wconstab/452/orig -> origin/gh/wconstab/452/orig 2025-12-04T08:27:07.2856888Z * [new branch] gh/wconstab/453/base -> origin/gh/wconstab/453/base 2025-12-04T08:27:07.2860919Z * [new branch] gh/wconstab/453/head -> origin/gh/wconstab/453/head 2025-12-04T08:27:07.2861096Z * [new branch] gh/wconstab/453/orig -> origin/gh/wconstab/453/orig 2025-12-04T08:27:07.2861256Z * [new branch] gh/wconstab/454/base -> origin/gh/wconstab/454/base 2025-12-04T08:27:07.2861393Z * [new branch] gh/wconstab/454/head -> origin/gh/wconstab/454/head 2025-12-04T08:27:07.2861535Z * [new branch] gh/wconstab/454/orig -> origin/gh/wconstab/454/orig 2025-12-04T08:27:07.2861670Z * [new branch] gh/wconstab/455/base -> origin/gh/wconstab/455/base 2025-12-04T08:27:07.2861807Z * [new branch] gh/wconstab/455/head -> origin/gh/wconstab/455/head 2025-12-04T08:27:07.2861949Z * [new branch] gh/wconstab/455/orig -> origin/gh/wconstab/455/orig 2025-12-04T08:27:07.2862088Z * [new branch] gh/wconstab/456/base -> origin/gh/wconstab/456/base 2025-12-04T08:27:07.2862227Z * [new branch] gh/wconstab/456/head -> origin/gh/wconstab/456/head 2025-12-04T08:27:07.2862356Z * [new branch] gh/wconstab/456/orig -> origin/gh/wconstab/456/orig 2025-12-04T08:27:07.2862488Z * [new branch] gh/wconstab/457/base -> origin/gh/wconstab/457/base 2025-12-04T08:27:07.2862625Z * [new branch] gh/wconstab/457/head -> origin/gh/wconstab/457/head 2025-12-04T08:27:07.2862752Z * [new branch] gh/wconstab/457/orig -> origin/gh/wconstab/457/orig 2025-12-04T08:27:07.2862888Z * [new branch] gh/wconstab/458/base -> origin/gh/wconstab/458/base 2025-12-04T08:27:07.2863019Z * [new branch] gh/wconstab/458/head -> origin/gh/wconstab/458/head 2025-12-04T08:27:07.2863148Z * [new branch] gh/wconstab/458/orig -> origin/gh/wconstab/458/orig 2025-12-04T08:27:07.2863287Z * [new branch] gh/wconstab/459/base -> origin/gh/wconstab/459/base 2025-12-04T08:27:07.2863415Z * [new branch] gh/wconstab/459/head -> origin/gh/wconstab/459/head 2025-12-04T08:27:07.2863684Z * [new branch] gh/wconstab/459/orig -> origin/gh/wconstab/459/orig 2025-12-04T08:27:07.2863825Z * [new branch] gh/wconstab/460/base -> origin/gh/wconstab/460/base 2025-12-04T08:27:07.2863958Z * [new branch] gh/wconstab/460/head -> origin/gh/wconstab/460/head 2025-12-04T08:27:07.2864106Z * [new branch] gh/wconstab/460/orig -> origin/gh/wconstab/460/orig 2025-12-04T08:27:07.2864241Z * [new branch] gh/wconstab/461/base -> origin/gh/wconstab/461/base 2025-12-04T08:27:07.2864373Z * [new branch] gh/wconstab/461/head -> origin/gh/wconstab/461/head 2025-12-04T08:27:07.2864527Z * [new branch] gh/wconstab/461/orig -> origin/gh/wconstab/461/orig 2025-12-04T08:27:07.2864892Z * [new branch] gh/wconstab/462/base -> origin/gh/wconstab/462/base 2025-12-04T08:27:07.2866989Z * [new branch] gh/wconstab/462/head -> origin/gh/wconstab/462/head 2025-12-04T08:27:07.2867362Z * [new branch] gh/wconstab/462/orig -> origin/gh/wconstab/462/orig 2025-12-04T08:27:07.2867624Z * [new branch] gh/wconstab/463/base -> origin/gh/wconstab/463/base 2025-12-04T08:27:07.2867990Z * [new branch] gh/wconstab/463/head -> origin/gh/wconstab/463/head 2025-12-04T08:27:07.2869217Z * [new branch] gh/wconstab/463/orig -> origin/gh/wconstab/463/orig 2025-12-04T08:27:07.2869800Z * [new branch] gh/wconstab/464/base -> origin/gh/wconstab/464/base 2025-12-04T08:27:07.2870686Z * [new branch] gh/wconstab/464/head -> origin/gh/wconstab/464/head 2025-12-04T08:27:07.2871241Z * [new branch] gh/wconstab/464/orig -> origin/gh/wconstab/464/orig 2025-12-04T08:27:07.2872309Z * [new branch] gh/wconstab/465/base -> origin/gh/wconstab/465/base 2025-12-04T08:27:07.2872829Z * [new branch] gh/wconstab/465/head -> origin/gh/wconstab/465/head 2025-12-04T08:27:07.2873682Z * [new branch] gh/wconstab/465/orig -> origin/gh/wconstab/465/orig 2025-12-04T08:27:07.2874714Z * [new branch] gh/wconstab/466/base -> origin/gh/wconstab/466/base 2025-12-04T08:27:07.2874893Z * [new branch] gh/wconstab/466/head -> origin/gh/wconstab/466/head 2025-12-04T08:27:07.2875981Z * [new branch] gh/wconstab/466/orig -> origin/gh/wconstab/466/orig 2025-12-04T08:27:07.2877076Z * [new branch] gh/wconstab/467/base -> origin/gh/wconstab/467/base 2025-12-04T08:27:07.2877469Z * [new branch] gh/wconstab/467/head -> origin/gh/wconstab/467/head 2025-12-04T08:27:07.2878639Z * [new branch] gh/wconstab/467/orig -> origin/gh/wconstab/467/orig 2025-12-04T08:27:07.2881230Z * [new branch] gh/wconstab/468/base -> origin/gh/wconstab/468/base 2025-12-04T08:27:07.2881552Z * [new branch] gh/wconstab/468/head -> origin/gh/wconstab/468/head 2025-12-04T08:27:07.2881816Z * [new branch] gh/wconstab/468/orig -> origin/gh/wconstab/468/orig 2025-12-04T08:27:07.2881986Z * [new branch] gh/weifengpy/39/base -> origin/gh/weifengpy/39/base 2025-12-04T08:27:07.2882325Z * [new branch] gh/weifengpy/39/head -> origin/gh/weifengpy/39/head 2025-12-04T08:27:07.2883399Z * [new branch] gh/weifengpy/39/orig -> origin/gh/weifengpy/39/orig 2025-12-04T08:27:07.2886279Z * [new branch] gh/weifengpy/40/base -> origin/gh/weifengpy/40/base 2025-12-04T08:27:07.2886620Z * [new branch] gh/weifengpy/40/head -> origin/gh/weifengpy/40/head 2025-12-04T08:27:07.2886892Z * [new branch] gh/weifengpy/40/orig -> origin/gh/weifengpy/40/orig 2025-12-04T08:27:07.2887119Z * [new branch] gh/weifengpy/41/base -> origin/gh/weifengpy/41/base 2025-12-04T08:27:07.2887537Z * [new branch] gh/weifengpy/41/head -> origin/gh/weifengpy/41/head 2025-12-04T08:27:07.2888413Z * [new branch] gh/weifengpy/41/orig -> origin/gh/weifengpy/41/orig 2025-12-04T08:27:07.2891721Z * [new branch] gh/williamwen42/250/base -> origin/gh/williamwen42/250/base 2025-12-04T08:27:07.2892055Z * [new branch] gh/williamwen42/250/head -> origin/gh/williamwen42/250/head 2025-12-04T08:27:07.2892297Z * [new branch] gh/williamwen42/250/orig -> origin/gh/williamwen42/250/orig 2025-12-04T08:27:07.2892473Z * [new branch] gh/williamwen42/279/base -> origin/gh/williamwen42/279/base 2025-12-04T08:27:07.2892760Z * [new branch] gh/williamwen42/279/head -> origin/gh/williamwen42/279/head 2025-12-04T08:27:07.2892947Z * [new branch] gh/williamwen42/279/orig -> origin/gh/williamwen42/279/orig 2025-12-04T08:27:07.2896658Z * [new branch] gh/williamwen42/282/base -> origin/gh/williamwen42/282/base 2025-12-04T08:27:07.2897010Z * [new branch] gh/williamwen42/282/head -> origin/gh/williamwen42/282/head 2025-12-04T08:27:07.2897258Z * [new branch] gh/williamwen42/282/orig -> origin/gh/williamwen42/282/orig 2025-12-04T08:27:07.2897465Z * [new branch] gh/williamwen42/287/base -> origin/gh/williamwen42/287/base 2025-12-04T08:27:07.2897695Z * [new branch] gh/williamwen42/287/head -> origin/gh/williamwen42/287/head 2025-12-04T08:27:07.2901558Z * [new branch] gh/williamwen42/287/orig -> origin/gh/williamwen42/287/orig 2025-12-04T08:27:07.2901888Z * [new branch] gh/williamwen42/288/base -> origin/gh/williamwen42/288/base 2025-12-04T08:27:07.2902276Z * [new branch] gh/williamwen42/288/head -> origin/gh/williamwen42/288/head 2025-12-04T08:27:07.2902552Z * [new branch] gh/williamwen42/288/orig -> origin/gh/williamwen42/288/orig 2025-12-04T08:27:07.2903671Z * [new branch] gh/williamwen42/296/base -> origin/gh/williamwen42/296/base 2025-12-04T08:27:07.2903905Z * [new branch] gh/williamwen42/296/head -> origin/gh/williamwen42/296/head 2025-12-04T08:27:07.2904081Z * [new branch] gh/williamwen42/296/orig -> origin/gh/williamwen42/296/orig 2025-12-04T08:27:07.2904229Z * [new branch] gh/williamwen42/297/base -> origin/gh/williamwen42/297/base 2025-12-04T08:27:07.2904369Z * [new branch] gh/williamwen42/297/head -> origin/gh/williamwen42/297/head 2025-12-04T08:27:07.2904641Z * [new branch] gh/williamwen42/297/orig -> origin/gh/williamwen42/297/orig 2025-12-04T08:27:07.2905070Z * [new branch] gh/williamwen42/306/base -> origin/gh/williamwen42/306/base 2025-12-04T08:27:07.2906488Z * [new branch] gh/williamwen42/306/head -> origin/gh/williamwen42/306/head 2025-12-04T08:27:07.2906666Z * [new branch] gh/williamwen42/306/orig -> origin/gh/williamwen42/306/orig 2025-12-04T08:27:07.2908742Z * [new branch] gh/williamwen42/309/base -> origin/gh/williamwen42/309/base 2025-12-04T08:27:07.2909086Z * [new branch] gh/williamwen42/309/head -> origin/gh/williamwen42/309/head 2025-12-04T08:27:07.2909341Z * [new branch] gh/williamwen42/309/orig -> origin/gh/williamwen42/309/orig 2025-12-04T08:27:07.2909504Z * [new branch] gh/williamwen42/310/base -> origin/gh/williamwen42/310/base 2025-12-04T08:27:07.2911799Z * [new branch] gh/williamwen42/310/head -> origin/gh/williamwen42/310/head 2025-12-04T08:27:07.2912134Z * [new branch] gh/williamwen42/310/orig -> origin/gh/williamwen42/310/orig 2025-12-04T08:27:07.2914983Z * [new branch] gh/williamwen42/311/base -> origin/gh/williamwen42/311/base 2025-12-04T08:27:07.2915326Z * [new branch] gh/williamwen42/311/head -> origin/gh/williamwen42/311/head 2025-12-04T08:27:07.2915773Z * [new branch] gh/williamwen42/311/orig -> origin/gh/williamwen42/311/orig 2025-12-04T08:27:07.2915967Z * [new branch] gh/williamwen42/319/base -> origin/gh/williamwen42/319/base 2025-12-04T08:27:07.2916205Z * [new branch] gh/williamwen42/319/head -> origin/gh/williamwen42/319/head 2025-12-04T08:27:07.2916774Z * [new branch] gh/williamwen42/319/orig -> origin/gh/williamwen42/319/orig 2025-12-04T08:27:07.2920168Z * [new branch] gh/williamwen42/325/base -> origin/gh/williamwen42/325/base 2025-12-04T08:27:07.2920514Z * [new branch] gh/williamwen42/325/head -> origin/gh/williamwen42/325/head 2025-12-04T08:27:07.2920964Z * [new branch] gh/williamwen42/325/orig -> origin/gh/williamwen42/325/orig 2025-12-04T08:27:07.2921151Z * [new branch] gh/williamwen42/326/base -> origin/gh/williamwen42/326/base 2025-12-04T08:27:07.2922375Z * [new branch] gh/williamwen42/326/head -> origin/gh/williamwen42/326/head 2025-12-04T08:27:07.2922698Z * [new branch] gh/williamwen42/326/orig -> origin/gh/williamwen42/326/orig 2025-12-04T08:27:07.2925261Z * [new branch] gh/williamwen42/327/base -> origin/gh/williamwen42/327/base 2025-12-04T08:27:07.2925593Z * [new branch] gh/williamwen42/327/head -> origin/gh/williamwen42/327/head 2025-12-04T08:27:07.2925848Z * [new branch] gh/williamwen42/327/orig -> origin/gh/williamwen42/327/orig 2025-12-04T08:27:07.2926085Z * [new branch] gh/williamwen42/328/base -> origin/gh/williamwen42/328/base 2025-12-04T08:27:07.2927142Z * [new branch] gh/williamwen42/328/head -> origin/gh/williamwen42/328/head 2025-12-04T08:27:07.2927846Z * [new branch] gh/williamwen42/328/orig -> origin/gh/williamwen42/328/orig 2025-12-04T08:27:07.2932371Z * [new branch] gh/williamwen42/329/base -> origin/gh/williamwen42/329/base 2025-12-04T08:27:07.2937353Z * [new branch] gh/williamwen42/329/head -> origin/gh/williamwen42/329/head 2025-12-04T08:27:07.2942125Z * [new branch] gh/williamwen42/329/orig -> origin/gh/williamwen42/329/orig 2025-12-04T08:27:07.2944313Z * [new branch] gh/williamwen42/330/base -> origin/gh/williamwen42/330/base 2025-12-04T08:27:07.2944518Z * [new branch] gh/williamwen42/330/head -> origin/gh/williamwen42/330/head 2025-12-04T08:27:07.2944678Z * [new branch] gh/williamwen42/330/orig -> origin/gh/williamwen42/330/orig 2025-12-04T08:27:07.2944840Z * [new branch] gh/williamwen42/331/base -> origin/gh/williamwen42/331/base 2025-12-04T08:27:07.2945220Z * [new branch] gh/williamwen42/331/head -> origin/gh/williamwen42/331/head 2025-12-04T08:27:07.2945556Z * [new branch] gh/williamwen42/331/orig -> origin/gh/williamwen42/331/orig 2025-12-04T08:27:07.2945893Z * [new branch] gh/williamwen42/332/base -> origin/gh/williamwen42/332/base 2025-12-04T08:27:07.2946221Z * [new branch] gh/williamwen42/332/head -> origin/gh/williamwen42/332/head 2025-12-04T08:27:07.2946557Z * [new branch] gh/williamwen42/332/orig -> origin/gh/williamwen42/332/orig 2025-12-04T08:27:07.2946783Z * [new branch] gh/williamwen42/333/base -> origin/gh/williamwen42/333/base 2025-12-04T08:27:07.2947028Z * [new branch] gh/williamwen42/333/head -> origin/gh/williamwen42/333/head 2025-12-04T08:27:07.2947206Z * [new branch] gh/williamwen42/333/orig -> origin/gh/williamwen42/333/orig 2025-12-04T08:27:07.2947367Z * [new branch] gh/williamwen42/334/base -> origin/gh/williamwen42/334/base 2025-12-04T08:27:07.2947536Z * [new branch] gh/williamwen42/334/head -> origin/gh/williamwen42/334/head 2025-12-04T08:27:07.2947695Z * [new branch] gh/williamwen42/334/orig -> origin/gh/williamwen42/334/orig 2025-12-04T08:27:07.2948034Z * [new branch] gh/williamwen42/335/base -> origin/gh/williamwen42/335/base 2025-12-04T08:27:07.2948201Z * [new branch] gh/williamwen42/335/head -> origin/gh/williamwen42/335/head 2025-12-04T08:27:07.2948752Z * [new branch] gh/williamwen42/335/orig -> origin/gh/williamwen42/335/orig 2025-12-04T08:27:07.2950619Z * [new branch] gh/williamwen42/336/base -> origin/gh/williamwen42/336/base 2025-12-04T08:27:07.2950794Z * [new branch] gh/williamwen42/336/head -> origin/gh/williamwen42/336/head 2025-12-04T08:27:07.2952526Z * [new branch] gh/williamwen42/336/orig -> origin/gh/williamwen42/336/orig 2025-12-04T08:27:07.2952737Z * [new branch] gh/williamwen42/337/base -> origin/gh/williamwen42/337/base 2025-12-04T08:27:07.2952895Z * [new branch] gh/williamwen42/337/head -> origin/gh/williamwen42/337/head 2025-12-04T08:27:07.2953221Z * [new branch] gh/williamwen42/337/orig -> origin/gh/williamwen42/337/orig 2025-12-04T08:27:07.2955734Z * [new branch] gh/williamwen42/338/base -> origin/gh/williamwen42/338/base 2025-12-04T08:27:07.2955920Z * [new branch] gh/williamwen42/338/head -> origin/gh/williamwen42/338/head 2025-12-04T08:27:07.2956080Z * [new branch] gh/williamwen42/338/orig -> origin/gh/williamwen42/338/orig 2025-12-04T08:27:07.2956490Z * [new branch] gh/williamwen42/339/base -> origin/gh/williamwen42/339/base 2025-12-04T08:27:07.2957377Z * [new branch] gh/williamwen42/339/head -> origin/gh/williamwen42/339/head 2025-12-04T08:27:07.2957824Z * [new branch] gh/williamwen42/339/orig -> origin/gh/williamwen42/339/orig 2025-12-04T08:27:07.2966177Z * [new branch] gh/williamwen42/340/base -> origin/gh/williamwen42/340/base 2025-12-04T08:27:07.2966385Z * [new branch] gh/williamwen42/340/head -> origin/gh/williamwen42/340/head 2025-12-04T08:27:07.2966555Z * [new branch] gh/williamwen42/340/orig -> origin/gh/williamwen42/340/orig 2025-12-04T08:27:07.2966701Z * [new branch] gh/williamwen42/341/base -> origin/gh/williamwen42/341/base 2025-12-04T08:27:07.2966857Z * [new branch] gh/williamwen42/341/head -> origin/gh/williamwen42/341/head 2025-12-04T08:27:07.2967000Z * [new branch] gh/williamwen42/341/orig -> origin/gh/williamwen42/341/orig 2025-12-04T08:27:07.2967153Z * [new branch] gh/williamwen42/342/base -> origin/gh/williamwen42/342/base 2025-12-04T08:27:07.2967298Z * [new branch] gh/williamwen42/342/head -> origin/gh/williamwen42/342/head 2025-12-04T08:27:07.2967442Z * [new branch] gh/williamwen42/342/orig -> origin/gh/williamwen42/342/orig 2025-12-04T08:27:07.2967590Z * [new branch] gh/williamwen42/343/base -> origin/gh/williamwen42/343/base 2025-12-04T08:27:07.2967738Z * [new branch] gh/williamwen42/343/head -> origin/gh/williamwen42/343/head 2025-12-04T08:27:07.2968076Z * [new branch] gh/williamwen42/343/orig -> origin/gh/williamwen42/343/orig 2025-12-04T08:27:07.2968821Z * [new branch] gh/williamwen42/344/base -> origin/gh/williamwen42/344/base 2025-12-04T08:27:07.2969041Z * [new branch] gh/williamwen42/344/head -> origin/gh/williamwen42/344/head 2025-12-04T08:27:07.2969420Z * [new branch] gh/williamwen42/344/orig -> origin/gh/williamwen42/344/orig 2025-12-04T08:27:07.2973914Z * [new branch] gh/williamwen42/345/base -> origin/gh/williamwen42/345/base 2025-12-04T08:27:07.2974098Z * [new branch] gh/williamwen42/345/head -> origin/gh/williamwen42/345/head 2025-12-04T08:27:07.2974275Z * [new branch] gh/williamwen42/345/orig -> origin/gh/williamwen42/345/orig 2025-12-04T08:27:07.2974422Z * [new branch] gh/williamwen42/346/base -> origin/gh/williamwen42/346/base 2025-12-04T08:27:07.2974713Z * [new branch] gh/williamwen42/346/head -> origin/gh/williamwen42/346/head 2025-12-04T08:27:07.2975029Z * [new branch] gh/williamwen42/346/orig -> origin/gh/williamwen42/346/orig 2025-12-04T08:27:07.2977678Z * [new branch] gh/williamwen42/347/base -> origin/gh/williamwen42/347/base 2025-12-04T08:27:07.2977860Z * [new branch] gh/williamwen42/347/head -> origin/gh/williamwen42/347/head 2025-12-04T08:27:07.2978048Z * [new branch] gh/williamwen42/347/orig -> origin/gh/williamwen42/347/orig 2025-12-04T08:27:07.2978198Z * [new branch] gh/williamwen42/348/base -> origin/gh/williamwen42/348/base 2025-12-04T08:27:07.2978399Z * [new branch] gh/williamwen42/348/head -> origin/gh/williamwen42/348/head 2025-12-04T08:27:07.2979360Z * [new branch] gh/williamwen42/348/orig -> origin/gh/williamwen42/348/orig 2025-12-04T08:27:07.2979987Z * [new branch] gh/williamwen42/349/base -> origin/gh/williamwen42/349/base 2025-12-04T08:27:07.2980785Z * [new branch] gh/williamwen42/349/head -> origin/gh/williamwen42/349/head 2025-12-04T08:27:07.2981137Z * [new branch] gh/williamwen42/349/orig -> origin/gh/williamwen42/349/orig 2025-12-04T08:27:07.2982798Z * [new branch] gh/williamwen42/350/base -> origin/gh/williamwen42/350/base 2025-12-04T08:27:07.2983052Z * [new branch] gh/williamwen42/350/head -> origin/gh/williamwen42/350/head 2025-12-04T08:27:07.2984122Z * [new branch] gh/williamwen42/350/orig -> origin/gh/williamwen42/350/orig 2025-12-04T08:27:07.2985125Z * [new branch] gh/williamwen42/351/base -> origin/gh/williamwen42/351/base 2025-12-04T08:27:07.2985510Z * [new branch] gh/williamwen42/351/head -> origin/gh/williamwen42/351/head 2025-12-04T08:27:07.2986355Z * [new branch] gh/williamwen42/351/orig -> origin/gh/williamwen42/351/orig 2025-12-04T08:27:07.2987246Z * [new branch] gh/williamwen42/352/base -> origin/gh/williamwen42/352/base 2025-12-04T08:27:07.2987686Z * [new branch] gh/williamwen42/352/head -> origin/gh/williamwen42/352/head 2025-12-04T08:27:07.2988698Z * [new branch] gh/williamwen42/352/orig -> origin/gh/williamwen42/352/orig 2025-12-04T08:27:07.2989844Z * [new branch] gh/williamwen42/353/base -> origin/gh/williamwen42/353/base 2025-12-04T08:27:07.2990433Z * [new branch] gh/williamwen42/353/head -> origin/gh/williamwen42/353/head 2025-12-04T08:27:07.2991322Z * [new branch] gh/williamwen42/353/orig -> origin/gh/williamwen42/353/orig 2025-12-04T08:27:07.2992869Z * [new branch] gh/williamwen42/354/base -> origin/gh/williamwen42/354/base 2025-12-04T08:27:07.2993412Z * [new branch] gh/williamwen42/354/head -> origin/gh/williamwen42/354/head 2025-12-04T08:27:07.2993614Z * [new branch] gh/williamwen42/354/orig -> origin/gh/williamwen42/354/orig 2025-12-04T08:27:07.2994653Z * [new branch] gh/williamwen42/355/base -> origin/gh/williamwen42/355/base 2025-12-04T08:27:07.2995218Z * [new branch] gh/williamwen42/355/head -> origin/gh/williamwen42/355/head 2025-12-04T08:27:07.2995838Z * [new branch] gh/williamwen42/355/orig -> origin/gh/williamwen42/355/orig 2025-12-04T08:27:07.2997270Z * [new branch] gh/williamwen42/356/base -> origin/gh/williamwen42/356/base 2025-12-04T08:27:07.2997504Z * [new branch] gh/williamwen42/356/head -> origin/gh/williamwen42/356/head 2025-12-04T08:27:07.2999183Z * [new branch] gh/williamwen42/356/orig -> origin/gh/williamwen42/356/orig 2025-12-04T08:27:07.2999722Z * [new branch] gh/williamwen42/357/base -> origin/gh/williamwen42/357/base 2025-12-04T08:27:07.2999884Z * [new branch] gh/williamwen42/357/head -> origin/gh/williamwen42/357/head 2025-12-04T08:27:07.3002768Z * [new branch] gh/williamwen42/357/orig -> origin/gh/williamwen42/357/orig 2025-12-04T08:27:07.3002934Z * [new branch] gh/williamwen42/358/base -> origin/gh/williamwen42/358/base 2025-12-04T08:27:07.3006163Z * [new branch] gh/williamwen42/358/head -> origin/gh/williamwen42/358/head 2025-12-04T08:27:07.3006462Z * [new branch] gh/williamwen42/358/orig -> origin/gh/williamwen42/358/orig 2025-12-04T08:27:07.3006623Z * [new branch] gh/xmfan/169/base -> origin/gh/xmfan/169/base 2025-12-04T08:27:07.3006756Z * [new branch] gh/xmfan/169/head -> origin/gh/xmfan/169/head 2025-12-04T08:27:07.3006915Z * [new branch] gh/xmfan/170/base -> origin/gh/xmfan/170/base 2025-12-04T08:27:07.3007045Z * [new branch] gh/xmfan/170/head -> origin/gh/xmfan/170/head 2025-12-04T08:27:07.3007416Z * [new branch] gh/xmfan/274/base -> origin/gh/xmfan/274/base 2025-12-04T08:27:07.3007799Z * [new branch] gh/xmfan/274/head -> origin/gh/xmfan/274/head 2025-12-04T08:27:07.3008461Z * [new branch] gh/xmfan/274/orig -> origin/gh/xmfan/274/orig 2025-12-04T08:27:07.3014187Z * [new branch] gh/xmfan/277/base -> origin/gh/xmfan/277/base 2025-12-04T08:27:07.3018346Z * [new branch] gh/xmfan/277/head -> origin/gh/xmfan/277/head 2025-12-04T08:27:07.3022581Z * [new branch] gh/xmfan/277/orig -> origin/gh/xmfan/277/orig 2025-12-04T08:27:07.3028336Z * [new branch] gh/xmfan/301/base -> origin/gh/xmfan/301/base 2025-12-04T08:27:07.3031978Z * [new branch] gh/xmfan/301/head -> origin/gh/xmfan/301/head 2025-12-04T08:27:07.3036180Z * [new branch] gh/xmfan/301/orig -> origin/gh/xmfan/301/orig 2025-12-04T08:27:07.3036345Z * [new branch] gh/xmfan/304/base -> origin/gh/xmfan/304/base 2025-12-04T08:27:07.3036603Z * [new branch] gh/xmfan/304/head -> origin/gh/xmfan/304/head 2025-12-04T08:27:07.3036751Z * [new branch] gh/xmfan/304/orig -> origin/gh/xmfan/304/orig 2025-12-04T08:27:07.3036892Z * [new branch] gh/xmfan/309/base -> origin/gh/xmfan/309/base 2025-12-04T08:27:07.3037027Z * [new branch] gh/xmfan/309/head -> origin/gh/xmfan/309/head 2025-12-04T08:27:07.3037165Z * [new branch] gh/xmfan/309/orig -> origin/gh/xmfan/309/orig 2025-12-04T08:27:07.3037310Z * [new branch] gh/xmfan/310/base -> origin/gh/xmfan/310/base 2025-12-04T08:27:07.3037453Z * [new branch] gh/xmfan/310/head -> origin/gh/xmfan/310/head 2025-12-04T08:27:07.3037686Z * [new branch] gh/xmfan/310/orig -> origin/gh/xmfan/310/orig 2025-12-04T08:27:07.3037837Z * [new branch] gh/xmfan/311/base -> origin/gh/xmfan/311/base 2025-12-04T08:27:07.3037973Z * [new branch] gh/xmfan/311/head -> origin/gh/xmfan/311/head 2025-12-04T08:27:07.3038194Z * [new branch] gh/xmfan/311/orig -> origin/gh/xmfan/311/orig 2025-12-04T08:27:07.3038337Z * [new branch] gh/xmfan/312/base -> origin/gh/xmfan/312/base 2025-12-04T08:27:07.3038546Z * [new branch] gh/xmfan/312/head -> origin/gh/xmfan/312/head 2025-12-04T08:27:07.3038697Z * [new branch] gh/xmfan/312/orig -> origin/gh/xmfan/312/orig 2025-12-04T08:27:07.3038837Z * [new branch] gh/xmfan/313/base -> origin/gh/xmfan/313/base 2025-12-04T08:27:07.3038985Z * [new branch] gh/xmfan/313/head -> origin/gh/xmfan/313/head 2025-12-04T08:27:07.3039128Z * [new branch] gh/xmfan/313/orig -> origin/gh/xmfan/313/orig 2025-12-04T08:27:07.3039388Z * [new branch] gh/xuanzhang816/27/base -> origin/gh/xuanzhang816/27/base 2025-12-04T08:27:07.3039571Z * [new branch] gh/xuanzhang816/27/head -> origin/gh/xuanzhang816/27/head 2025-12-04T08:27:07.3039737Z * [new branch] gh/xuanzhang816/27/orig -> origin/gh/xuanzhang816/27/orig 2025-12-04T08:27:07.3039903Z * [new branch] gh/xuanzhang816/32/base -> origin/gh/xuanzhang816/32/base 2025-12-04T08:27:07.3040085Z * [new branch] gh/xuanzhang816/32/head -> origin/gh/xuanzhang816/32/head 2025-12-04T08:27:07.3040232Z * [new branch] gh/xuanzhang816/32/orig -> origin/gh/xuanzhang816/32/orig 2025-12-04T08:27:07.3040386Z * [new branch] gh/xuanzhang816/33/base -> origin/gh/xuanzhang816/33/base 2025-12-04T08:27:07.3040539Z * [new branch] gh/xuanzhang816/33/head -> origin/gh/xuanzhang816/33/head 2025-12-04T08:27:07.3040694Z * [new branch] gh/xuanzhang816/33/orig -> origin/gh/xuanzhang816/33/orig 2025-12-04T08:27:07.3040844Z * [new branch] gh/xuanzhang816/34/base -> origin/gh/xuanzhang816/34/base 2025-12-04T08:27:07.3040993Z * [new branch] gh/xuanzhang816/34/head -> origin/gh/xuanzhang816/34/head 2025-12-04T08:27:07.3041146Z * [new branch] gh/xuanzhang816/34/orig -> origin/gh/xuanzhang816/34/orig 2025-12-04T08:27:07.3041292Z * [new branch] gh/xuanzhang816/35/base -> origin/gh/xuanzhang816/35/base 2025-12-04T08:27:07.3041443Z * [new branch] gh/xuanzhang816/35/head -> origin/gh/xuanzhang816/35/head 2025-12-04T08:27:07.3041587Z * [new branch] gh/xuanzhang816/35/orig -> origin/gh/xuanzhang816/35/orig 2025-12-04T08:27:07.3041938Z * [new branch] gh/yanbing-j/11/base -> origin/gh/yanbing-j/11/base 2025-12-04T08:27:07.3042119Z * [new branch] gh/yanbing-j/11/head -> origin/gh/yanbing-j/11/head 2025-12-04T08:27:07.3046970Z * [new branch] gh/yanbing-j/11/orig -> origin/gh/yanbing-j/11/orig 2025-12-04T08:27:07.3047333Z * [new branch] gh/yanbing-j/12/base -> origin/gh/yanbing-j/12/base 2025-12-04T08:27:07.3047574Z * [new branch] gh/yanbing-j/12/head -> origin/gh/yanbing-j/12/head 2025-12-04T08:27:07.3047820Z * [new branch] gh/yanbing-j/12/orig -> origin/gh/yanbing-j/12/orig 2025-12-04T08:27:07.3047978Z * [new branch] gh/yanbing-j/13/base -> origin/gh/yanbing-j/13/base 2025-12-04T08:27:07.3048119Z * [new branch] gh/yanbing-j/13/head -> origin/gh/yanbing-j/13/head 2025-12-04T08:27:07.3048415Z * [new branch] gh/yanbing-j/13/orig -> origin/gh/yanbing-j/13/orig 2025-12-04T08:27:07.3048569Z * [new branch] gh/yanbing-j/14/base -> origin/gh/yanbing-j/14/base 2025-12-04T08:27:07.3049601Z * [new branch] gh/yanbing-j/14/head -> origin/gh/yanbing-j/14/head 2025-12-04T08:27:07.3049950Z * [new branch] gh/yanbing-j/14/orig -> origin/gh/yanbing-j/14/orig 2025-12-04T08:27:07.3055429Z * [new branch] gh/yanbing-j/15/base -> origin/gh/yanbing-j/15/base 2025-12-04T08:27:07.3055603Z * [new branch] gh/yanbing-j/15/head -> origin/gh/yanbing-j/15/head 2025-12-04T08:27:07.3055748Z * [new branch] gh/yanbing-j/15/orig -> origin/gh/yanbing-j/15/orig 2025-12-04T08:27:07.3055889Z * [new branch] gh/yanbing-j/18/base -> origin/gh/yanbing-j/18/base 2025-12-04T08:27:07.3056020Z * [new branch] gh/yanbing-j/18/head -> origin/gh/yanbing-j/18/head 2025-12-04T08:27:07.3056150Z * [new branch] gh/yanbing-j/18/orig -> origin/gh/yanbing-j/18/orig 2025-12-04T08:27:07.3056304Z * [new branch] gh/yanbing-j/19/base -> origin/gh/yanbing-j/19/base 2025-12-04T08:27:07.3056437Z * [new branch] gh/yanbing-j/19/head -> origin/gh/yanbing-j/19/head 2025-12-04T08:27:07.3056886Z * [new branch] gh/yanbing-j/19/orig -> origin/gh/yanbing-j/19/orig 2025-12-04T08:27:07.3058973Z * [new branch] gh/yanbing-j/20/base -> origin/gh/yanbing-j/20/base 2025-12-04T08:27:07.3059295Z * [new branch] gh/yanbing-j/20/head -> origin/gh/yanbing-j/20/head 2025-12-04T08:27:07.3059479Z * [new branch] gh/yanbing-j/20/orig -> origin/gh/yanbing-j/20/orig 2025-12-04T08:27:07.3061225Z * [new branch] gh/yanbing-j/21/base -> origin/gh/yanbing-j/21/base 2025-12-04T08:27:07.3061544Z * [new branch] gh/yanbing-j/21/head -> origin/gh/yanbing-j/21/head 2025-12-04T08:27:07.3061714Z * [new branch] gh/yanbing-j/22/base -> origin/gh/yanbing-j/22/base 2025-12-04T08:27:07.3064243Z * [new branch] gh/yanbing-j/22/head -> origin/gh/yanbing-j/22/head 2025-12-04T08:27:07.3064415Z * [new branch] gh/yanbing-j/22/orig -> origin/gh/yanbing-j/22/orig 2025-12-04T08:27:07.3064580Z * [new branch] gh/yanbing-j/23/base -> origin/gh/yanbing-j/23/base 2025-12-04T08:27:07.3064720Z * [new branch] gh/yanbing-j/23/head -> origin/gh/yanbing-j/23/head 2025-12-04T08:27:07.3065367Z * [new branch] gh/yanbing-j/23/orig -> origin/gh/yanbing-j/23/orig 2025-12-04T08:27:07.3066718Z * [new branch] gh/yanbing-j/24/base -> origin/gh/yanbing-j/24/base 2025-12-04T08:27:07.3066863Z * [new branch] gh/yanbing-j/24/head -> origin/gh/yanbing-j/24/head 2025-12-04T08:27:07.3067675Z * [new branch] gh/yanbing-j/24/orig -> origin/gh/yanbing-j/24/orig 2025-12-04T08:27:07.3070708Z * [new branch] gh/yanbing-j/25/base -> origin/gh/yanbing-j/25/base 2025-12-04T08:27:07.3070879Z * [new branch] gh/yanbing-j/25/head -> origin/gh/yanbing-j/25/head 2025-12-04T08:27:07.3071021Z * [new branch] gh/yanbing-j/25/orig -> origin/gh/yanbing-j/25/orig 2025-12-04T08:27:07.3071157Z * [new branch] gh/yanbing-j/26/base -> origin/gh/yanbing-j/26/base 2025-12-04T08:27:07.3071293Z * [new branch] gh/yanbing-j/26/head -> origin/gh/yanbing-j/26/head 2025-12-04T08:27:07.3071730Z * [new branch] gh/yanbing-j/26/orig -> origin/gh/yanbing-j/26/orig 2025-12-04T08:27:07.3075971Z * [new branch] gh/yang-yu-hang/1/base -> origin/gh/yang-yu-hang/1/base 2025-12-04T08:27:07.3076150Z * [new branch] gh/yang-yu-hang/1/head -> origin/gh/yang-yu-hang/1/head 2025-12-04T08:27:07.3076294Z * [new branch] gh/yang-yu-hang/1/orig -> origin/gh/yang-yu-hang/1/orig 2025-12-04T08:27:07.3076448Z * [new branch] gh/yang-yu-hang/2/base -> origin/gh/yang-yu-hang/2/base 2025-12-04T08:27:07.3076597Z * [new branch] gh/yang-yu-hang/2/head -> origin/gh/yang-yu-hang/2/head 2025-12-04T08:27:07.3077233Z * [new branch] gh/yang-yu-hang/2/orig -> origin/gh/yang-yu-hang/2/orig 2025-12-04T08:27:07.3081453Z * [new branch] gh/yang-yu-hang/3/base -> origin/gh/yang-yu-hang/3/base 2025-12-04T08:27:07.3081637Z * [new branch] gh/yang-yu-hang/3/head -> origin/gh/yang-yu-hang/3/head 2025-12-04T08:27:07.3081833Z * [new branch] gh/yang-yu-hang/3/orig -> origin/gh/yang-yu-hang/3/orig 2025-12-04T08:27:07.3081983Z * [new branch] gh/yangw-dev/12/base -> origin/gh/yangw-dev/12/base 2025-12-04T08:27:07.3082119Z * [new branch] gh/yangw-dev/12/head -> origin/gh/yangw-dev/12/head 2025-12-04T08:27:07.3082448Z * [new branch] gh/yangw-dev/12/orig -> origin/gh/yangw-dev/12/orig 2025-12-04T08:27:07.3085105Z * [new branch] gh/yangw-dev/13/base -> origin/gh/yangw-dev/13/base 2025-12-04T08:27:07.3085426Z * [new branch] gh/yangw-dev/13/head -> origin/gh/yangw-dev/13/head 2025-12-04T08:27:07.3085858Z * [new branch] gh/yangw-dev/13/orig -> origin/gh/yangw-dev/13/orig 2025-12-04T08:27:07.3089903Z * [new branch] gh/yangw-dev/14/base -> origin/gh/yangw-dev/14/base 2025-12-04T08:27:07.3090081Z * [new branch] gh/yangw-dev/14/head -> origin/gh/yangw-dev/14/head 2025-12-04T08:27:07.3090218Z * [new branch] gh/yangw-dev/14/orig -> origin/gh/yangw-dev/14/orig 2025-12-04T08:27:07.3090351Z * [new branch] gh/yangw-dev/15/base -> origin/gh/yangw-dev/15/base 2025-12-04T08:27:07.3090494Z * [new branch] gh/yangw-dev/15/head -> origin/gh/yangw-dev/15/head 2025-12-04T08:27:07.3091385Z * [new branch] gh/yangw-dev/15/orig -> origin/gh/yangw-dev/15/orig 2025-12-04T08:27:07.3091634Z * [new branch] gh/yangw-dev/19/base -> origin/gh/yangw-dev/19/base 2025-12-04T08:27:07.3092167Z * [new branch] gh/yangw-dev/19/head -> origin/gh/yangw-dev/19/head 2025-12-04T08:27:07.3092352Z * [new branch] gh/yangw-dev/19/orig -> origin/gh/yangw-dev/19/orig 2025-12-04T08:27:07.3092484Z * [new branch] gh/yangw-dev/26/base -> origin/gh/yangw-dev/26/base 2025-12-04T08:27:07.3092613Z * [new branch] gh/yangw-dev/26/head -> origin/gh/yangw-dev/26/head 2025-12-04T08:27:07.3097511Z * [new branch] gh/yangw-dev/26/orig -> origin/gh/yangw-dev/26/orig 2025-12-04T08:27:07.3097677Z * [new branch] gh/yangw-dev/27/base -> origin/gh/yangw-dev/27/base 2025-12-04T08:27:07.3097818Z * [new branch] gh/yangw-dev/27/head -> origin/gh/yangw-dev/27/head 2025-12-04T08:27:07.3098091Z * [new branch] gh/yangw-dev/27/orig -> origin/gh/yangw-dev/27/orig 2025-12-04T08:27:07.3098244Z * [new branch] gh/ydwu4/292/base -> origin/gh/ydwu4/292/base 2025-12-04T08:27:07.3098391Z * [new branch] gh/ydwu4/292/head -> origin/gh/ydwu4/292/head 2025-12-04T08:27:07.3098518Z * [new branch] gh/ydwu4/292/orig -> origin/gh/ydwu4/292/orig 2025-12-04T08:27:07.3102826Z * [new branch] gh/ydwu4/294/base -> origin/gh/ydwu4/294/base 2025-12-04T08:27:07.3103003Z * [new branch] gh/ydwu4/294/head -> origin/gh/ydwu4/294/head 2025-12-04T08:27:07.3103137Z * [new branch] gh/ydwu4/294/orig -> origin/gh/ydwu4/294/orig 2025-12-04T08:27:07.3103261Z * [new branch] gh/ydwu4/295/base -> origin/gh/ydwu4/295/base 2025-12-04T08:27:07.3103396Z * [new branch] gh/ydwu4/295/head -> origin/gh/ydwu4/295/head 2025-12-04T08:27:07.3103559Z * [new branch] gh/ydwu4/295/orig -> origin/gh/ydwu4/295/orig 2025-12-04T08:27:07.3103721Z * [new branch] gh/ydwu4/296/base -> origin/gh/ydwu4/296/base 2025-12-04T08:27:07.3103877Z * [new branch] gh/ydwu4/296/head -> origin/gh/ydwu4/296/head 2025-12-04T08:27:07.3106771Z * [new branch] gh/ydwu4/296/orig -> origin/gh/ydwu4/296/orig 2025-12-04T08:27:07.3107086Z * [new branch] gh/ydwu4/306/base -> origin/gh/ydwu4/306/base 2025-12-04T08:27:07.3107247Z * [new branch] gh/ydwu4/306/head -> origin/gh/ydwu4/306/head 2025-12-04T08:27:07.3107385Z * [new branch] gh/ydwu4/306/orig -> origin/gh/ydwu4/306/orig 2025-12-04T08:27:07.3107539Z * [new branch] gh/ydwu4/312/base -> origin/gh/ydwu4/312/base 2025-12-04T08:27:07.3107668Z * [new branch] gh/ydwu4/312/head -> origin/gh/ydwu4/312/head 2025-12-04T08:27:07.3107823Z * [new branch] gh/ydwu4/312/orig -> origin/gh/ydwu4/312/orig 2025-12-04T08:27:07.3110733Z * [new branch] gh/ydwu4/322/base -> origin/gh/ydwu4/322/base 2025-12-04T08:27:07.3111094Z * [new branch] gh/ydwu4/322/head -> origin/gh/ydwu4/322/head 2025-12-04T08:27:07.3111243Z * [new branch] gh/ydwu4/322/orig -> origin/gh/ydwu4/322/orig 2025-12-04T08:27:07.3111384Z * [new branch] gh/ydwu4/327/base -> origin/gh/ydwu4/327/base 2025-12-04T08:27:07.3111525Z * [new branch] gh/ydwu4/327/head -> origin/gh/ydwu4/327/head 2025-12-04T08:27:07.3111710Z * [new branch] gh/ydwu4/327/orig -> origin/gh/ydwu4/327/orig 2025-12-04T08:27:07.3113586Z * [new branch] gh/ydwu4/328/base -> origin/gh/ydwu4/328/base 2025-12-04T08:27:07.3113942Z * [new branch] gh/ydwu4/328/head -> origin/gh/ydwu4/328/head 2025-12-04T08:27:07.3114098Z * [new branch] gh/ydwu4/328/orig -> origin/gh/ydwu4/328/orig 2025-12-04T08:27:07.3115114Z * [new branch] gh/ydwu4/329/base -> origin/gh/ydwu4/329/base 2025-12-04T08:27:07.3116332Z * [new branch] gh/ydwu4/329/head -> origin/gh/ydwu4/329/head 2025-12-04T08:27:07.3116634Z * [new branch] gh/ydwu4/329/orig -> origin/gh/ydwu4/329/orig 2025-12-04T08:27:07.3117163Z * [new branch] gh/ydwu4/330/base -> origin/gh/ydwu4/330/base 2025-12-04T08:27:07.3117687Z * [new branch] gh/ydwu4/330/head -> origin/gh/ydwu4/330/head 2025-12-04T08:27:07.3118773Z * [new branch] gh/ydwu4/330/orig -> origin/gh/ydwu4/330/orig 2025-12-04T08:27:07.3119703Z * [new branch] gh/ydwu4/331/base -> origin/gh/ydwu4/331/base 2025-12-04T08:27:07.3120275Z * [new branch] gh/ydwu4/331/head -> origin/gh/ydwu4/331/head 2025-12-04T08:27:07.3122053Z * [new branch] gh/ydwu4/331/orig -> origin/gh/ydwu4/331/orig 2025-12-04T08:27:07.3122213Z * [new branch] gh/ydwu4/332/base -> origin/gh/ydwu4/332/base 2025-12-04T08:27:07.3123206Z * [new branch] gh/ydwu4/332/head -> origin/gh/ydwu4/332/head 2025-12-04T08:27:07.3123932Z * [new branch] gh/ydwu4/332/orig -> origin/gh/ydwu4/332/orig 2025-12-04T08:27:07.3124637Z * [new branch] gh/ydwu4/333/base -> origin/gh/ydwu4/333/base 2025-12-04T08:27:07.3125207Z * [new branch] gh/ydwu4/333/head -> origin/gh/ydwu4/333/head 2025-12-04T08:27:07.3125833Z * [new branch] gh/ydwu4/333/orig -> origin/gh/ydwu4/333/orig 2025-12-04T08:27:07.3126938Z * [new branch] gh/ydwu4/334/base -> origin/gh/ydwu4/334/base 2025-12-04T08:27:07.3127228Z * [new branch] gh/ydwu4/334/head -> origin/gh/ydwu4/334/head 2025-12-04T08:27:07.3128177Z * [new branch] gh/ydwu4/334/orig -> origin/gh/ydwu4/334/orig 2025-12-04T08:27:07.3128787Z * [new branch] gh/ydwu4/335/base -> origin/gh/ydwu4/335/base 2025-12-04T08:27:07.3129485Z * [new branch] gh/ydwu4/335/head -> origin/gh/ydwu4/335/head 2025-12-04T08:27:07.3130118Z * [new branch] gh/ydwu4/335/orig -> origin/gh/ydwu4/335/orig 2025-12-04T08:27:07.3131755Z * [new branch] gh/ydwu4/337/base -> origin/gh/ydwu4/337/base 2025-12-04T08:27:07.3131899Z * [new branch] gh/ydwu4/337/head -> origin/gh/ydwu4/337/head 2025-12-04T08:27:07.3132931Z * [new branch] gh/ydwu4/337/orig -> origin/gh/ydwu4/337/orig 2025-12-04T08:27:07.3134320Z * [new branch] gh/ydwu4/339/base -> origin/gh/ydwu4/339/base 2025-12-04T08:27:07.3134594Z * [new branch] gh/ydwu4/339/head -> origin/gh/ydwu4/339/head 2025-12-04T08:27:07.3135522Z * [new branch] gh/ydwu4/339/orig -> origin/gh/ydwu4/339/orig 2025-12-04T08:27:07.3136796Z * [new branch] gh/yf225/133/base -> origin/gh/yf225/133/base 2025-12-04T08:27:07.3137177Z * [new branch] gh/yf225/133/head -> origin/gh/yf225/133/head 2025-12-04T08:27:07.3138574Z * [new branch] gh/yf225/93/base -> origin/gh/yf225/93/base 2025-12-04T08:27:07.3138935Z * [new branch] gh/yf225/93/head -> origin/gh/yf225/93/head 2025-12-04T08:27:07.3143967Z * [new branch] gh/yifuwang/152/base -> origin/gh/yifuwang/152/base 2025-12-04T08:27:07.3144156Z * [new branch] gh/yifuwang/152/head -> origin/gh/yifuwang/152/head 2025-12-04T08:27:07.3144317Z * [new branch] gh/yifuwang/152/orig -> origin/gh/yifuwang/152/orig 2025-12-04T08:27:07.3144472Z * [new branch] gh/yifuwang/195/base -> origin/gh/yifuwang/195/base 2025-12-04T08:27:07.3144646Z * [new branch] gh/yifuwang/195/head -> origin/gh/yifuwang/195/head 2025-12-04T08:27:07.3144981Z * [new branch] gh/yifuwang/195/orig -> origin/gh/yifuwang/195/orig 2025-12-04T08:27:07.3150112Z * [new branch] gh/yiming0416/1/base -> origin/gh/yiming0416/1/base 2025-12-04T08:27:07.3152010Z * [new branch] gh/yiming0416/1/head -> origin/gh/yiming0416/1/head 2025-12-04T08:27:07.3152191Z * [new branch] gh/yiming0416/2/base -> origin/gh/yiming0416/2/base 2025-12-04T08:27:07.3152366Z * [new branch] gh/yiming0416/2/head -> origin/gh/yiming0416/2/head 2025-12-04T08:27:07.3152518Z * [new branch] gh/yushangdi/1/base -> origin/gh/yushangdi/1/base 2025-12-04T08:27:07.3152691Z * [new branch] gh/yushangdi/1/head -> origin/gh/yushangdi/1/head 2025-12-04T08:27:07.3153109Z * [new branch] gh/yushangdi/10/base -> origin/gh/yushangdi/10/base 2025-12-04T08:27:07.3153280Z * [new branch] gh/yushangdi/10/head -> origin/gh/yushangdi/10/head 2025-12-04T08:27:07.3153444Z * [new branch] gh/yushangdi/10/orig -> origin/gh/yushangdi/10/orig 2025-12-04T08:27:07.3153601Z * [new branch] gh/yushangdi/11/base -> origin/gh/yushangdi/11/base 2025-12-04T08:27:07.3153761Z * [new branch] gh/yushangdi/11/head -> origin/gh/yushangdi/11/head 2025-12-04T08:27:07.3154100Z * [new branch] gh/yushangdi/11/orig -> origin/gh/yushangdi/11/orig 2025-12-04T08:27:07.3155393Z * [new branch] gh/yushangdi/2/base -> origin/gh/yushangdi/2/base 2025-12-04T08:27:07.3155749Z * [new branch] gh/yushangdi/2/head -> origin/gh/yushangdi/2/head 2025-12-04T08:27:07.3158240Z * [new branch] gh/yushangdi/7/base -> origin/gh/yushangdi/7/base 2025-12-04T08:27:07.3158458Z * [new branch] gh/yushangdi/7/head -> origin/gh/yushangdi/7/head 2025-12-04T08:27:07.3158617Z * [new branch] gh/yushangdi/7/orig -> origin/gh/yushangdi/7/orig 2025-12-04T08:27:07.3163741Z * [new branch] gh/yushangdi/8/base -> origin/gh/yushangdi/8/base 2025-12-04T08:27:07.3163935Z * [new branch] gh/yushangdi/8/head -> origin/gh/yushangdi/8/head 2025-12-04T08:27:07.3164086Z * [new branch] gh/yushangdi/8/orig -> origin/gh/yushangdi/8/orig 2025-12-04T08:27:07.3164235Z * [new branch] gh/yushangdi/9/base -> origin/gh/yushangdi/9/base 2025-12-04T08:27:07.3164474Z * [new branch] gh/yushangdi/9/head -> origin/gh/yushangdi/9/head 2025-12-04T08:27:07.3173831Z * [new branch] gh/yushangdi/9/orig -> origin/gh/yushangdi/9/orig 2025-12-04T08:27:07.3176321Z * [new branch] gh/zklaus/19/base -> origin/gh/zklaus/19/base 2025-12-04T08:27:07.3176474Z * [new branch] gh/zklaus/19/head -> origin/gh/zklaus/19/head 2025-12-04T08:27:07.3176611Z * [new branch] gh/zklaus/19/orig -> origin/gh/zklaus/19/orig 2025-12-04T08:27:07.3176736Z * [new branch] gh/zklaus/20/base -> origin/gh/zklaus/20/base 2025-12-04T08:27:07.3177086Z * [new branch] gh/zklaus/20/head -> origin/gh/zklaus/20/head 2025-12-04T08:27:07.3182624Z * [new branch] gh/zklaus/20/orig -> origin/gh/zklaus/20/orig 2025-12-04T08:27:07.3187908Z * [new branch] gh/zklaus/21/base -> origin/gh/zklaus/21/base 2025-12-04T08:27:07.3192627Z * [new branch] gh/zklaus/21/head -> origin/gh/zklaus/21/head 2025-12-04T08:27:07.3192944Z * [new branch] gh/zklaus/21/orig -> origin/gh/zklaus/21/orig 2025-12-04T08:27:07.3193100Z * [new branch] gh/zklaus/22/base -> origin/gh/zklaus/22/base 2025-12-04T08:27:07.3193243Z * [new branch] gh/zklaus/22/head -> origin/gh/zklaus/22/head 2025-12-04T08:27:07.3193512Z * [new branch] gh/zklaus/22/orig -> origin/gh/zklaus/22/orig 2025-12-04T08:27:07.3193663Z * [new branch] gh/zklaus/23/base -> origin/gh/zklaus/23/base 2025-12-04T08:27:07.3193799Z * [new branch] gh/zklaus/23/head -> origin/gh/zklaus/23/head 2025-12-04T08:27:07.3193936Z * [new branch] gh/zklaus/23/orig -> origin/gh/zklaus/23/orig 2025-12-04T08:27:07.3194186Z * [new branch] gh/zklaus/24/base -> origin/gh/zklaus/24/base 2025-12-04T08:27:07.3194804Z * [new branch] gh/zklaus/24/head -> origin/gh/zklaus/24/head 2025-12-04T08:27:07.3195146Z * [new branch] gh/zklaus/24/orig -> origin/gh/zklaus/24/orig 2025-12-04T08:27:07.3195334Z * [new branch] gh/zou3519/1197/base -> origin/gh/zou3519/1197/base 2025-12-04T08:27:07.3195639Z * [new branch] gh/zou3519/1197/head -> origin/gh/zou3519/1197/head 2025-12-04T08:27:07.3195920Z * [new branch] gh/zou3519/1197/orig -> origin/gh/zou3519/1197/orig 2025-12-04T08:27:07.3196119Z * [new branch] gh/zou3519/1199/base -> origin/gh/zou3519/1199/base 2025-12-04T08:27:07.3196277Z * [new branch] gh/zou3519/1199/head -> origin/gh/zou3519/1199/head 2025-12-04T08:27:07.3196782Z * [new branch] gh/zou3519/1199/orig -> origin/gh/zou3519/1199/orig 2025-12-04T08:27:07.3196960Z * [new branch] gh/zou3519/1200/base -> origin/gh/zou3519/1200/base 2025-12-04T08:27:07.3197121Z * [new branch] gh/zou3519/1200/head -> origin/gh/zou3519/1200/head 2025-12-04T08:27:07.3197272Z * [new branch] gh/zou3519/1200/orig -> origin/gh/zou3519/1200/orig 2025-12-04T08:27:07.3197419Z * [new branch] gh/zou3519/1201/base -> origin/gh/zou3519/1201/base 2025-12-04T08:27:07.3197616Z * [new branch] gh/zou3519/1201/head -> origin/gh/zou3519/1201/head 2025-12-04T08:27:07.3197760Z * [new branch] gh/zou3519/1201/orig -> origin/gh/zou3519/1201/orig 2025-12-04T08:27:07.3197925Z * [new branch] gh/zou3519/1202/base -> origin/gh/zou3519/1202/base 2025-12-04T08:27:07.3198235Z * [new branch] gh/zou3519/1202/head -> origin/gh/zou3519/1202/head 2025-12-04T08:27:07.3198399Z * [new branch] gh/zou3519/1202/orig -> origin/gh/zou3519/1202/orig 2025-12-04T08:27:07.3198552Z * [new branch] gh/zpcore/1/base -> origin/gh/zpcore/1/base 2025-12-04T08:27:07.3198700Z * [new branch] gh/zpcore/1/head -> origin/gh/zpcore/1/head 2025-12-04T08:27:07.3198847Z * [new branch] gh/zpcore/11/base -> origin/gh/zpcore/11/base 2025-12-04T08:27:07.3198989Z * [new branch] gh/zpcore/11/head -> origin/gh/zpcore/11/head 2025-12-04T08:27:07.3199143Z * [new branch] gh/zpcore/11/orig -> origin/gh/zpcore/11/orig 2025-12-04T08:27:07.3199288Z * [new branch] gh/zpcore/12/base -> origin/gh/zpcore/12/base 2025-12-04T08:27:07.3199655Z * [new branch] gh/zpcore/12/head -> origin/gh/zpcore/12/head 2025-12-04T08:27:07.3199787Z * [new branch] gh/zpcore/12/orig -> origin/gh/zpcore/12/orig 2025-12-04T08:27:07.3199923Z * [new branch] gh/zpcore/13/base -> origin/gh/zpcore/13/base 2025-12-04T08:27:07.3200058Z * [new branch] gh/zpcore/13/head -> origin/gh/zpcore/13/head 2025-12-04T08:27:07.3200185Z * [new branch] gh/zpcore/13/orig -> origin/gh/zpcore/13/orig 2025-12-04T08:27:07.3200462Z * [new branch] gh/zpcore/14/base -> origin/gh/zpcore/14/base 2025-12-04T08:27:07.3200879Z * [new branch] gh/zpcore/14/head -> origin/gh/zpcore/14/head 2025-12-04T08:27:07.3202378Z * [new branch] gh/zpcore/14/orig -> origin/gh/zpcore/14/orig 2025-12-04T08:27:07.3203446Z * [new branch] gh/zpcore/15/base -> origin/gh/zpcore/15/base 2025-12-04T08:27:07.3203670Z * [new branch] gh/zpcore/15/head -> origin/gh/zpcore/15/head 2025-12-04T08:27:07.3205879Z * [new branch] gh/zpcore/15/orig -> origin/gh/zpcore/15/orig 2025-12-04T08:27:07.3206067Z * [new branch] gh/zpcore/2/base -> origin/gh/zpcore/2/base 2025-12-04T08:27:07.3206208Z * [new branch] gh/zpcore/2/head -> origin/gh/zpcore/2/head 2025-12-04T08:27:07.3208951Z * [new branch] gh/zpcore/21/base -> origin/gh/zpcore/21/base 2025-12-04T08:27:07.3209282Z * [new branch] gh/zpcore/21/head -> origin/gh/zpcore/21/head 2025-12-04T08:27:07.3209448Z * [new branch] gh/zpcore/21/orig -> origin/gh/zpcore/21/orig 2025-12-04T08:27:07.3209977Z * [new branch] gh/zpcore/22/base -> origin/gh/zpcore/22/base 2025-12-04T08:27:07.3210498Z * [new branch] gh/zpcore/22/head -> origin/gh/zpcore/22/head 2025-12-04T08:27:07.3213658Z * [new branch] gh/zpcore/22/orig -> origin/gh/zpcore/22/orig 2025-12-04T08:27:07.3213833Z * [new branch] gh/zpcore/23/base -> origin/gh/zpcore/23/base 2025-12-04T08:27:07.3213983Z * [new branch] gh/zpcore/23/head -> origin/gh/zpcore/23/head 2025-12-04T08:27:07.3214120Z * [new branch] gh/zpcore/23/orig -> origin/gh/zpcore/23/orig 2025-12-04T08:27:07.3214712Z * [new branch] gh/zpcore/24/base -> origin/gh/zpcore/24/base 2025-12-04T08:27:07.3215539Z * [new branch] gh/zpcore/24/head -> origin/gh/zpcore/24/head 2025-12-04T08:27:07.3216021Z * [new branch] gh/zpcore/24/orig -> origin/gh/zpcore/24/orig 2025-12-04T08:27:07.3217351Z * [new branch] gh/zpcore/25/base -> origin/gh/zpcore/25/base 2025-12-04T08:27:07.3217643Z * [new branch] gh/zpcore/25/head -> origin/gh/zpcore/25/head 2025-12-04T08:27:07.3218536Z * [new branch] gh/zpcore/25/orig -> origin/gh/zpcore/25/orig 2025-12-04T08:27:07.3221619Z * [new branch] gh/zpcore/26/base -> origin/gh/zpcore/26/base 2025-12-04T08:27:07.3221918Z * [new branch] gh/zpcore/26/head -> origin/gh/zpcore/26/head 2025-12-04T08:27:07.3222059Z * [new branch] gh/zpcore/26/orig -> origin/gh/zpcore/26/orig 2025-12-04T08:27:07.3227157Z * [new branch] gh/zpcore/27/base -> origin/gh/zpcore/27/base 2025-12-04T08:27:07.3227299Z * [new branch] gh/zpcore/27/head -> origin/gh/zpcore/27/head 2025-12-04T08:27:07.3227664Z * [new branch] gh/zpcore/27/orig -> origin/gh/zpcore/27/orig 2025-12-04T08:27:07.3232398Z * [new branch] gh/zpcore/28/base -> origin/gh/zpcore/28/base 2025-12-04T08:27:07.3232566Z * [new branch] gh/zpcore/28/head -> origin/gh/zpcore/28/head 2025-12-04T08:27:07.3232918Z * [new branch] gh/zpcore/28/orig -> origin/gh/zpcore/28/orig 2025-12-04T08:27:07.3233072Z * [new branch] gh/zpcore/3/base -> origin/gh/zpcore/3/base 2025-12-04T08:27:07.3233386Z * [new branch] gh/zpcore/3/head -> origin/gh/zpcore/3/head 2025-12-04T08:27:07.3233524Z * [new branch] gh/zpcore/4/base -> origin/gh/zpcore/4/base 2025-12-04T08:27:07.3235523Z * [new branch] gh/zpcore/4/head -> origin/gh/zpcore/4/head 2025-12-04T08:27:07.3235685Z * [new branch] gh/zpcore/5/base -> origin/gh/zpcore/5/base 2025-12-04T08:27:07.3235836Z * [new branch] gh/zpcore/5/head -> origin/gh/zpcore/5/head 2025-12-04T08:27:07.3236514Z * [new branch] gh/zpcore/6/base -> origin/gh/zpcore/6/base 2025-12-04T08:27:07.3237528Z * [new branch] gh/zpcore/6/head -> origin/gh/zpcore/6/head 2025-12-04T08:27:07.3238580Z * [new branch] gh/zpcore/7/base -> origin/gh/zpcore/7/base 2025-12-04T08:27:07.3238982Z * [new branch] gh/zpcore/7/head -> origin/gh/zpcore/7/head 2025-12-04T08:27:07.3244078Z * [new branch] gh/zpcore/8/base -> origin/gh/zpcore/8/base 2025-12-04T08:27:07.3244394Z * [new branch] gh/zpcore/8/head -> origin/gh/zpcore/8/head 2025-12-04T08:27:07.3244535Z * [new branch] google-main -> origin/google-main 2025-12-04T08:27:07.3244810Z * [new branch] guangyey/external_stream -> origin/guangyey/external_stream 2025-12-04T08:27:07.3251750Z * [new branch] guangyey/test_2025 -> origin/guangyey/test_2025 2025-12-04T08:27:07.3254166Z * [new branch] guilhermeleobas/cherry-pick-55d87d9dfd9 -> origin/guilhermeleobas/cherry-pick-55d87d9dfd9 2025-12-04T08:27:07.3254546Z * [new branch] hameerabbasi/complex_tensor_subclass -> origin/hameerabbasi/complex_tensor_subclass 2025-12-04T08:27:07.3259296Z * [new branch] hameerabbasi/fix-ctensor-gradcheck-tests -> origin/hameerabbasi/fix-ctensor-gradcheck-tests 2025-12-04T08:27:07.3261297Z * [new branch] hameerabbasi/gradcheck-allclose -> origin/hameerabbasi/gradcheck-allclose 2025-12-04T08:27:07.3261588Z * [new branch] hc_baseline -> origin/hc_baseline 2025-12-04T08:27:07.3261756Z * [new branch] hhh_rand -> origin/hhh_rand 2025-12-04T08:27:07.3261916Z * [new branch] huba/f1 -> origin/huba/f1 2025-12-04T08:27:07.3262270Z * [new branch] increase-timeout-linux-jammy-cuda12_8-py3_10-gcc11-test -> origin/increase-timeout-linux-jammy-cuda12_8-py3_10-gcc11-test 2025-12-04T08:27:07.3262387Z * [new branch] inlining -> origin/inlining 2025-12-04T08:27:07.3262643Z * [new branch] inlining-ezyang -> origin/inlining-ezyang 2025-12-04T08:27:07.3262818Z * [new branch] install-torchao-0.13.0 -> origin/install-torchao-0.13.0 2025-12-04T08:27:07.3263241Z * [new branch] instrument-trunk-pull-linux-with-job-test-filters -> origin/instrument-trunk-pull-linux-with-job-test-filters 2025-12-04T08:27:07.3263504Z * [new branch] invoke-subgraph -> origin/invoke-subgraph 2025-12-04T08:27:07.3263667Z * [new branch] issue#58739 -> origin/issue#58739 2025-12-04T08:27:07.3264204Z * [new branch] jainapurva-patch-1 -> origin/jainapurva-patch-1 2025-12-04T08:27:07.3264370Z * [new branch] jathu/o3 -> origin/jathu/o3 2025-12-04T08:27:07.3264502Z * [new branch] jathu/sve -> origin/jathu/sve 2025-12-04T08:27:07.3264727Z * [new branch] jcaip/test-cusparselt-version-0.6.2 -> origin/jcaip/test-cusparselt-version-0.6.2 2025-12-04T08:27:07.3265043Z * [new branch] jcaip/update-cusparselt-0.6.2 -> origin/jcaip/update-cusparselt-0.6.2 2025-12-04T08:27:07.3265233Z * [new branch] jiannanWang/memorysnapshot_filter -> origin/jiannanWang/memorysnapshot_filter 2025-12-04T08:27:07.3265423Z * [new branch] jiannanWang/profilerstepwarning -> origin/jiannanWang/profilerstepwarning 2025-12-04T08:27:07.3265580Z * [new branch] jithunnair-amd-patch-1 -> origin/jithunnair-amd-patch-1 2025-12-04T08:27:07.3265733Z * [new branch] jithunnair-amd-patch-10 -> origin/jithunnair-amd-patch-10 2025-12-04T08:27:07.3265885Z * [new branch] jithunnair-amd-patch-2 -> origin/jithunnair-amd-patch-2 2025-12-04T08:27:07.3266031Z * [new branch] jithunnair-amd-patch-3 -> origin/jithunnair-amd-patch-3 2025-12-04T08:27:07.3266182Z * [new branch] jithunnair-amd-patch-4 -> origin/jithunnair-amd-patch-4 2025-12-04T08:27:07.3266321Z * [new branch] jithunnair-amd-patch-5 -> origin/jithunnair-amd-patch-5 2025-12-04T08:27:07.3266465Z * [new branch] jithunnair-amd-patch-6 -> origin/jithunnair-amd-patch-6 2025-12-04T08:27:07.3266610Z * [new branch] jithunnair-amd-patch-7 -> origin/jithunnair-amd-patch-7 2025-12-04T08:27:07.3266922Z * [new branch] jithunnair-amd-patch-8 -> origin/jithunnair-amd-patch-8 2025-12-04T08:27:07.3267284Z * [new branch] jithunnair-amd-patch-9 -> origin/jithunnair-amd-patch-9 2025-12-04T08:27:07.3268981Z * [new branch] justinchu/native-qdq -> origin/justinchu/native-qdq 2025-12-04T08:27:07.3269264Z * [new branch] kainan666/xlf_debug -> origin/kainan666/xlf_debug 2025-12-04T08:27:07.3271662Z * [new branch] kainan_test -> origin/kainan_test 2025-12-04T08:27:07.3272018Z * [new branch] larryliu0820-patch-1 -> origin/larryliu0820-patch-1 2025-12-04T08:27:07.3272304Z * [new branch] leslie/test_group_gemm_epilogues -> origin/leslie/test_group_gemm_epilogues 2025-12-04T08:27:07.3272500Z * [new branch] lessw2020/fix_cutlass_cache_error -> origin/lessw2020/fix_cutlass_cache_error 2025-12-04T08:27:07.3273829Z * [new branch] liaoxuan/shm_all_reduce -> origin/liaoxuan/shm_all_reduce 2025-12-04T08:27:07.3274463Z * [new branch] liaoxuan/test_fa_disable_softmax -> origin/liaoxuan/test_fa_disable_softmax 2025-12-04T08:27:07.3274673Z * [new branch] liaoxuan/test_int8_sdpa -> origin/liaoxuan/test_int8_sdpa 2025-12-04T08:27:07.3275701Z * [new branch] llama4-stable -> origin/llama4-stable 2025-12-04T08:27:07.3276786Z * [new branch] lts/release/1.8 -> origin/lts/release/1.8 2025-12-04T08:27:07.3277604Z * [new branch] lucaskabela/#94773 -> origin/lucaskabela/#94773 2025-12-04T08:27:07.3278304Z * [new branch] lucaskabela/fix_164876 -> origin/lucaskabela/fix_164876 2025-12-04T08:27:07.3278813Z * [new branch] lucaskabela/flop_counter -> origin/lucaskabela/flop_counter 2025-12-04T08:27:07.3279323Z * [new branch] lucaskabela/func_under_decomp -> origin/lucaskabela/func_under_decomp 2025-12-04T08:27:07.3283327Z * [new branch] lucaskabela/functional_in_dynamo -> origin/lucaskabela/functional_in_dynamo 2025-12-04T08:27:07.3283736Z * [new branch] lucaskabela/install_params_as_graph_attr -> origin/lucaskabela/install_params_as_graph_attr 2025-12-04T08:27:07.3284091Z * [new branch] lucaskabela/parameters_as_graph_attr -> origin/lucaskabela/parameters_as_graph_attr 2025-12-04T08:27:07.3284457Z * [new branch] lucaskabela/remove_aot_dispatcher_metadata -> origin/lucaskabela/remove_aot_dispatcher_metadata 2025-12-04T08:27:07.3285095Z * [new branch] lucaskabela/rnn_decomp -> origin/lucaskabela/rnn_decomp 2025-12-04T08:27:07.3285464Z * [new branch] lucaskabela/typing_backends -> origin/lucaskabela/typing_backends 2025-12-04T08:27:07.3285647Z * [new branch] lucaskabela/typing_ctx_manager -> origin/lucaskabela/typing_ctx_manager 2025-12-04T08:27:07.3285828Z * [new branch] lucaskabela/typing_nn_module -> origin/lucaskabela/typing_nn_module 2025-12-04T08:27:07.3286082Z * [new branch] lucaskabela/typing_user_defined -> origin/lucaskabela/typing_user_defined 2025-12-04T08:27:07.3286556Z * [new branch] lucaskabela/typing_variables -> origin/lucaskabela/typing_variables 2025-12-04T08:27:07.3290007Z * [new branch] lucaskabela/typing_variables_dicts -> origin/lucaskabela/typing_variables_dicts 2025-12-04T08:27:07.3290417Z * [new branch] lucaskabela/typing_variables_functions -> origin/lucaskabela/typing_variables_functions 2025-12-04T08:27:07.3290744Z * [new branch] lucaskabela/typing_variables_lists -> origin/lucaskabela/typing_variables_lists 2025-12-04T08:27:07.3291001Z * [new branch] lw/torch_box_by_ref -> origin/lw/torch_box_by_ref 2025-12-04T08:27:07.3291134Z * [new branch] main -> origin/main 2025-12-04T08:27:07.3291446Z * [new branch] malfet-patch-1 -> origin/malfet-patch-1 2025-12-04T08:27:07.3295954Z * [new branch] malfet-patch-2 -> origin/malfet-patch-2 2025-12-04T08:27:07.3296275Z * [new branch] malfet-patch-3 -> origin/malfet-patch-3 2025-12-04T08:27:07.3296423Z * [new branch] malfet-patch-4 -> origin/malfet-patch-4 2025-12-04T08:27:07.3296662Z * [new branch] malfet-patch-5 -> origin/malfet-patch-5 2025-12-04T08:27:07.3296940Z * [new branch] malfet-patch-6 -> origin/malfet-patch-6 2025-12-04T08:27:07.3297202Z * [new branch] malfet-patch-7 -> origin/malfet-patch-7 2025-12-04T08:27:07.3297357Z * [new branch] malfet-patch-8 -> origin/malfet-patch-8 2025-12-04T08:27:07.3298651Z * [new branch] malfet/add-3.14-ci -> origin/malfet/add-3.14-ci 2025-12-04T08:27:07.3302326Z * [new branch] malfet/be-do-not-make-typos-in-build-artifacts -> origin/malfet/be-do-not-make-typos-in-build-artifacts 2025-12-04T08:27:07.3302658Z * [new branch] malfet/be-move-more-settings-to-checkout-pytorch -> origin/malfet/be-move-more-settings-to-checkout-pytorch 2025-12-04T08:27:07.3302883Z * [new branch] malfet/be-remove-misisng-neon-headers -> origin/malfet/be-remove-misisng-neon-headers 2025-12-04T08:27:07.3303062Z * [new branch] malfet/mps-implement-col2im -> origin/malfet/mps-implement-col2im 2025-12-04T08:27:07.3303289Z * [new branch] manuel/aoti_metal_shimify-thread_safe -> origin/manuel/aoti_metal_shimify-thread_safe 2025-12-04T08:27:07.3303612Z * [new branch] manuel/inductor_link_openmp -> origin/manuel/inductor_link_openmp 2025-12-04T08:27:07.3303820Z * [new branch] masnesral/metaconda -> origin/masnesral/metaconda 2025-12-04T08:27:07.3304051Z * [new branch] mem_profiler_flaky_fix -> origin/mem_profiler_flaky_fix 2025-12-04T08:27:07.3304255Z * [new branch] mem_profiler_stack_trace -> origin/mem_profiler_stack_trace 2025-12-04T08:27:07.3308176Z * [new branch] memory_profiler_stack -> origin/memory_profiler_stack 2025-12-04T08:27:07.3308513Z * [new branch] metascroy-patch-1 -> origin/metascroy-patch-1 2025-12-04T08:27:07.3308738Z * [new branch] mingw_posix -> origin/mingw_posix 2025-12-04T08:27:07.3308916Z * [new branch] mlazos/S429861-debug -> origin/mlazos/S429861-debug 2025-12-04T08:27:07.3309164Z * [new branch] mlazos/aa -> origin/mlazos/aa 2025-12-04T08:27:07.3309301Z * [new branch] mlazos/acts -> origin/mlazos/acts 2025-12-04T08:27:07.3310485Z * [new branch] mlazos/arg-renames -> origin/mlazos/arg-renames 2025-12-04T08:27:07.3310702Z * [new branch] mlazos/bad-cudagraphs -> origin/mlazos/bad-cudagraphs 2025-12-04T08:27:07.3311295Z * [new branch] mlazos/baseline-graph-breaks -> origin/mlazos/baseline-graph-breaks 2025-12-04T08:27:07.3315529Z * [new branch] mlazos/beta-tensor -> origin/mlazos/beta-tensor 2025-12-04T08:27:07.3315842Z * [new branch] mlazos/buffers -> origin/mlazos/buffers 2025-12-04T08:27:07.3316013Z * [new branch] mlazos/buffers2 -> origin/mlazos/buffers2 2025-12-04T08:27:07.3316162Z * [new branch] mlazos/buffers3 -> origin/mlazos/buffers3 2025-12-04T08:27:07.3316284Z * [new branch] mlazos/bwd -> origin/mlazos/bwd 2025-12-04T08:27:07.3316432Z * [new branch] mlazos/combo-test -> origin/mlazos/combo-test 2025-12-04T08:27:07.3316727Z * [new branch] mlazos/ctx-cleanup -> origin/mlazos/ctx-cleanup 2025-12-04T08:27:07.3317190Z * [new branch] mlazos/cuda-cmd-log -> origin/mlazos/cuda-cmd-log 2025-12-04T08:27:07.3317932Z * [new branch] mlazos/cudagraph-tests -> origin/mlazos/cudagraph-tests 2025-12-04T08:27:07.3318576Z * [new branch] mlazos/cudagraphs-measurement -> origin/mlazos/cudagraphs-measurement 2025-12-04T08:27:07.3323179Z * [new branch] mlazos/cutlass-test -> origin/mlazos/cutlass-test 2025-12-04T08:27:07.3323728Z * [new branch] mlazos/cutlass-topo-bug -> origin/mlazos/cutlass-topo-bug 2025-12-04T08:27:07.3324105Z * [new branch] mlazos/dataclass-proxy -> origin/mlazos/dataclass-proxy 2025-12-04T08:27:07.3324244Z * [new branch] mlazos/dc-attrs -> origin/mlazos/dc-attrs 2025-12-04T08:27:07.3324398Z * [new branch] mlazos/dc-helion -> origin/mlazos/dc-helion 2025-12-04T08:27:07.3324530Z * [new branch] mlazos/dict-fix -> origin/mlazos/dict-fix 2025-12-04T08:27:07.3324707Z * [new branch] mlazos/disable-tf -> origin/mlazos/disable-tf 2025-12-04T08:27:07.3325067Z * [new branch] mlazos/dupe-fix -> origin/mlazos/dupe-fix 2025-12-04T08:27:07.3325300Z * [new branch] mlazos/dyn-batch -> origin/mlazos/dyn-batch 2025-12-04T08:27:07.3325451Z * [new branch] mlazos/evt -> origin/mlazos/evt 2025-12-04T08:27:07.3327429Z * [new branch] mlazos/extract-examples -> origin/mlazos/extract-examples 2025-12-04T08:27:07.3327780Z * [new branch] mlazos/foreach-op -> origin/mlazos/foreach-op 2025-12-04T08:27:07.3328032Z * [new branch] mlazos/fp8 -> origin/mlazos/fp8 2025-12-04T08:27:07.3328359Z * [new branch] mlazos/fp8-bias -> origin/mlazos/fp8-bias 2025-12-04T08:27:07.3334088Z * [new branch] mlazos/fp8-bias-fusion -> origin/mlazos/fp8-bias-fusion 2025-12-04T08:27:07.3339546Z * [new branch] mlazos/fp8-fixes -> origin/mlazos/fp8-fixes 2025-12-04T08:27:07.3343978Z * [new branch] mlazos/freezing -> origin/mlazos/freezing 2025-12-04T08:27:07.3348149Z * [new branch] mlazos/h-comp -> origin/mlazos/h-comp 2025-12-04T08:27:07.3353713Z * [new branch] mlazos/h-comp2 -> origin/mlazos/h-comp2 2025-12-04T08:27:07.3355859Z * [new branch] mlazos/hash-hop -> origin/mlazos/hash-hop 2025-12-04T08:27:07.3356024Z * [new branch] mlazos/hc -> origin/mlazos/hc 2025-12-04T08:27:07.3356179Z * [new branch] mlazos/hc-cycles -> origin/mlazos/hc-cycles 2025-12-04T08:27:07.3356324Z * [new branch] mlazos/hc-fixes -> origin/mlazos/hc-fixes 2025-12-04T08:27:07.3356677Z * [new branch] mlazos/hc-fixes3 -> origin/mlazos/hc-fixes3 2025-12-04T08:27:07.3356825Z * [new branch] mlazos/hc-fixes4 -> origin/mlazos/hc-fixes4 2025-12-04T08:27:07.3356976Z * [new branch] mlazos/hc-hf -> origin/mlazos/hc-hf 2025-12-04T08:27:07.3357123Z * [new branch] mlazos/hc-mut -> origin/mlazos/hc-mut 2025-12-04T08:27:07.3357252Z * [new branch] mlazos/hc10 -> origin/mlazos/hc10 2025-12-04T08:27:07.3357387Z * [new branch] mlazos/hc11 -> origin/mlazos/hc11 2025-12-04T08:27:07.3357515Z * [new branch] mlazos/hc12 -> origin/mlazos/hc12 2025-12-04T08:27:07.3357643Z * [new branch] mlazos/hc13 -> origin/mlazos/hc13 2025-12-04T08:27:07.3357764Z * [new branch] mlazos/hc14 -> origin/mlazos/hc14 2025-12-04T08:27:07.3357889Z * [new branch] mlazos/hc15 -> origin/mlazos/hc15 2025-12-04T08:27:07.3358172Z * [new branch] mlazos/hc2 -> origin/mlazos/hc2 2025-12-04T08:27:07.3358306Z * [new branch] mlazos/hc4 -> origin/mlazos/hc4 2025-12-04T08:27:07.3358433Z * [new branch] mlazos/hc5 -> origin/mlazos/hc5 2025-12-04T08:27:07.3358552Z * [new branch] mlazos/hc6 -> origin/mlazos/hc6 2025-12-04T08:27:07.3358669Z * [new branch] mlazos/hc7 -> origin/mlazos/hc7 2025-12-04T08:27:07.3358795Z * [new branch] mlazos/hc8 -> origin/mlazos/hc8 2025-12-04T08:27:07.3358973Z * [new branch] mlazos/hc9 -> origin/mlazos/hc9 2025-12-04T08:27:07.3359127Z * [new branch] mlazos/hc_baseline2 -> origin/mlazos/hc_baseline2 2025-12-04T08:27:07.3359314Z * [new branch] mlazos/inductor-streams -> origin/mlazos/inductor-streams 2025-12-04T08:27:07.3359439Z * [new branch] mlazos/main -> origin/mlazos/main 2025-12-04T08:27:07.3359572Z * [new branch] mlazos/mcg2 -> origin/mlazos/mcg2 2025-12-04T08:27:07.3359722Z * [new branch] mlazos/meta-guards -> origin/mlazos/meta-guards 2025-12-04T08:27:07.3359922Z * [new branch] mlazos/mlazos/foreach-map-adam -> origin/mlazos/mlazos/foreach-map-adam 2025-12-04T08:27:07.3360098Z * [new branch] mlazos/mlazos/tf-mode-backup -> origin/mlazos/mlazos/tf-mode-backup 2025-12-04T08:27:07.3360225Z * [new branch] mlazos/mod-fix -> origin/mlazos/mod-fix 2025-12-04T08:27:07.3360364Z * [new branch] mlazos/mode-fix -> origin/mlazos/mode-fix 2025-12-04T08:27:07.3360499Z * [new branch] mlazos/offsets -> origin/mlazos/offsets 2025-12-04T08:27:07.3360643Z * [new branch] mlazos/overguarding -> origin/mlazos/overguarding 2025-12-04T08:27:07.3360792Z * [new branch] mlazos/proxy-ctors -> origin/mlazos/proxy-ctors 2025-12-04T08:27:07.3360923Z * [new branch] mlazos/quant-fix -> origin/mlazos/quant-fix 2025-12-04T08:27:07.3361065Z * [new branch] mlazos/resnet-fix -> origin/mlazos/resnet-fix 2025-12-04T08:27:07.3361202Z * [new branch] mlazos/rm-buf-names -> origin/mlazos/rm-buf-names 2025-12-04T08:27:07.3361322Z * [new branch] mlazos/rm-code -> origin/mlazos/rm-code 2025-12-04T08:27:07.3361452Z * [new branch] mlazos/rm-spam -> origin/mlazos/rm-spam 2025-12-04T08:27:07.3364304Z * [new branch] mlazos/rtp -> origin/mlazos/rtp 2025-12-04T08:27:07.3364818Z * [new branch] mlazos/static-idx-dbg -> origin/mlazos/static-idx-dbg 2025-12-04T08:27:07.3365214Z * [new branch] mlazos/static-inputs-log -> origin/mlazos/static-inputs-log 2025-12-04T08:27:07.3365357Z * [new branch] mlazos/stests -> origin/mlazos/stests 2025-12-04T08:27:07.3365519Z * [new branch] mlazos/stream-ops -> origin/mlazos/stream-ops 2025-12-04T08:27:07.3365665Z * [new branch] mlazos/td-fix2 -> origin/mlazos/td-fix2 2025-12-04T08:27:07.3365838Z * [new branch] mlazos/tensor-hasattr2 -> origin/mlazos/tensor-hasattr2 2025-12-04T08:27:07.3369474Z * [new branch] mlazos/test -> origin/mlazos/test 2025-12-04T08:27:07.3369600Z * [new branch] mlazos/tf-mode -> origin/mlazos/tf-mode 2025-12-04T08:27:07.3369774Z * [new branch] mlazos/tf-mode-backup2 -> origin/mlazos/tf-mode-backup2 2025-12-04T08:27:07.3369921Z * [new branch] mlazos/tf-mode-reland -> origin/mlazos/tf-mode-reland 2025-12-04T08:27:07.3370075Z * [new branch] mlazos/tf-mode-reland2 -> origin/mlazos/tf-mode-reland2 2025-12-04T08:27:07.3370215Z * [new branch] mlazos/tf-mode-reland3 -> origin/mlazos/tf-mode-reland3 2025-12-04T08:27:07.3370362Z * [new branch] mlazos/triton-no-epi -> origin/mlazos/triton-no-epi 2025-12-04T08:27:07.3370510Z * [new branch] mlazos/tune-proto -> origin/mlazos/tune-proto 2025-12-04T08:27:07.3370649Z * [new branch] mlazos/tuple-fixes -> origin/mlazos/tuple-fixes 2025-12-04T08:27:07.3370797Z * [new branch] mlazos/tuple-fixes2 -> origin/mlazos/tuple-fixes2 2025-12-04T08:27:07.3371028Z * [new branch] mlazos/tuple-handling -> origin/mlazos/tuple-handling 2025-12-04T08:27:07.3373574Z * [new branch] mlazos/user-stream-base -> origin/mlazos/user-stream-base 2025-12-04T08:27:07.3380226Z * [new branch] mlazos/user-streams -> origin/mlazos/user-streams 2025-12-04T08:27:07.3384168Z * [new branch] mlazos/user-streams-backup -> origin/mlazos/user-streams-backup 2025-12-04T08:27:07.3386077Z * [new branch] mlazos/user-streams-backup2 -> origin/mlazos/user-streams-backup2 2025-12-04T08:27:07.3386229Z * [new branch] mlazos/vary-beta -> origin/mlazos/vary-beta 2025-12-04T08:27:07.3386365Z * [new branch] mlazos/vary-beta2 -> origin/mlazos/vary-beta2 2025-12-04T08:27:07.3386516Z * [new branch] mlazos/weird-perf1 -> origin/mlazos/weird-perf1 2025-12-04T08:27:07.3386648Z * [new branch] mm_out_dtype_compile -> origin/mm_out_dtype_compile 2025-12-04T08:27:07.3386814Z * [new branch] module-shim -> origin/module-shim 2025-12-04T08:27:07.3386934Z * [new branch] move_config -> origin/move_config 2025-12-04T08:27:07.3387063Z * [new branch] msaroufim/reduce -> origin/msaroufim/reduce 2025-12-04T08:27:07.3387197Z * [new branch] mtia/basic-cmake -> origin/mtia/basic-cmake 2025-12-04T08:27:07.3387370Z * [new branch] mwizak/fix-triton-block-shape -> origin/mwizak/fix-triton-block-shape 2025-12-04T08:27:07.3387495Z * [new branch] my_varlen_backup -> origin/my_varlen_backup 2025-12-04T08:27:07.3387631Z * [new branch] nativert_num_outputs -> origin/nativert_num_outputs 2025-12-04T08:27:07.3387746Z * [new branch] new-codegen -> origin/new-codegen 2025-12-04T08:27:07.3387871Z * [new branch] newtest-base -> origin/newtest-base 2025-12-04T08:27:07.3388004Z * [new branch] ngimel/addmm_dtype -> origin/ngimel/addmm_dtype 2025-12-04T08:27:07.3388122Z * [new branch] ngimel/div_inv -> origin/ngimel/div_inv 2025-12-04T08:27:07.3388273Z * [new branch] ngimel/error_index_list -> origin/ngimel/error_index_list 2025-12-04T08:27:07.3388541Z * [new branch] ngimel/gather_grid -> origin/ngimel/gather_grid 2025-12-04T08:27:07.3389412Z * [new branch] ngimel/gather_grid_release -> origin/ngimel/gather_grid_release 2025-12-04T08:27:07.3389560Z * [new branch] ngimel/gg_new -> origin/ngimel/gg_new 2025-12-04T08:27:07.3389707Z * [new branch] ngimel/hostalloc -> origin/ngimel/hostalloc 2025-12-04T08:27:07.3389862Z * [new branch] ngimel/storage_id -> origin/ngimel/storage_id 2025-12-04T08:27:07.3389986Z * [new branch] nightly -> origin/nightly 2025-12-04T08:27:07.3390318Z * [new branch] nikitaved/addmm_1_rowcol_lt_path_check -> origin/nikitaved/addmm_1_rowcol_lt_path_check 2025-12-04T08:27:07.3390543Z * [new branch] nikitaved/addmm_epilogue_fusions_2d_bias -> origin/nikitaved/addmm_epilogue_fusions_2d_bias 2025-12-04T08:27:07.3399004Z * [new branch] nikitaved/addmm_epilogue_fusions_inductor -> origin/nikitaved/addmm_epilogue_fusions_inductor 2025-12-04T08:27:07.3399282Z * [new branch] nikitaved/addmm_epilogue_fusions_scratch -> origin/nikitaved/addmm_epilogue_fusions_scratch 2025-12-04T08:27:07.3399507Z * [new branch] nikitaved/grad_addmm_epilogue_fusions -> origin/nikitaved/grad_addmm_epilogue_fusions 2025-12-04T08:27:07.3399717Z * [new branch] nikitaved/simpler_can_use_32bit_index -> origin/nikitaved/simpler_can_use_32bit_index 2025-12-04T08:27:07.3399854Z * [new branch] nikitaved/test -> origin/nikitaved/test 2025-12-04T08:27:07.3400270Z * [new branch] nmacchioni-perf-test-async-autotune -> origin/nmacchioni-perf-test-async-autotune 2025-12-04T08:27:07.3400429Z * [new branch] no_distributed_log_spew -> origin/no_distributed_log_spew 2025-12-04T08:27:07.3400555Z * [new branch] nofun-hack -> origin/nofun-hack 2025-12-04T08:27:07.3400687Z * [new branch] norm_bench -> origin/norm_bench 2025-12-04T08:27:07.3400837Z * [new branch] nullplay/fuse_matmul -> origin/nullplay/fuse_matmul 2025-12-04T08:27:07.3400982Z * [new branch] nullplay_fuse_matmul -> origin/nullplay_fuse_matmul 2025-12-04T08:27:07.3401108Z * [new branch] optimizer_test -> origin/optimizer_test 2025-12-04T08:27:07.3401259Z * [new branch] orig/release/1.10 -> origin/orig/release/1.10 2025-12-04T08:27:07.3401392Z * [new branch] orig/release/1.11 -> origin/orig/release/1.11 2025-12-04T08:27:07.3401522Z * [new branch] orig/release/1.12 -> origin/orig/release/1.12 2025-12-04T08:27:07.3401652Z * [new branch] orig/release/1.13 -> origin/orig/release/1.13 2025-12-04T08:27:07.3401781Z * [new branch] orig/release/1.6 -> origin/orig/release/1.6 2025-12-04T08:27:07.3407180Z * [new branch] orig/release/1.7 -> origin/orig/release/1.7 2025-12-04T08:27:07.3407347Z * [new branch] orig/release/1.8 -> origin/orig/release/1.8 2025-12-04T08:27:07.3407517Z * [new branch] orig/release/1.9 -> origin/orig/release/1.9 2025-12-04T08:27:07.3407687Z * [new branch] orig/release/2.0 -> origin/orig/release/2.0 2025-12-04T08:27:07.3407814Z * [new branch] orig/release/2.1 -> origin/orig/release/2.1 2025-12-04T08:27:07.3407949Z * [new branch] orig/release/2.2 -> origin/orig/release/2.2 2025-12-04T08:27:07.3408112Z * [new branch] orig/release/2.3 -> origin/orig/release/2.3 2025-12-04T08:27:07.3408417Z * [new branch] orig/release/2.4 -> origin/orig/release/2.4 2025-12-04T08:27:07.3408561Z * [new branch] orig/release/2.5 -> origin/orig/release/2.5 2025-12-04T08:27:07.3408980Z * [new branch] orig/release/2.6 -> origin/orig/release/2.6 2025-12-04T08:27:07.3409120Z * [new branch] orig/release/2.7 -> origin/orig/release/2.7 2025-12-04T08:27:07.3411767Z * [new branch] orig/release/2.8 -> origin/orig/release/2.8 2025-12-04T08:27:07.3411923Z * [new branch] orig/release/2.9 -> origin/orig/release/2.9 2025-12-04T08:27:07.3412189Z * [new branch] origin/gh/fxdawnn/1/base -> origin/origin/gh/fxdawnn/1/base 2025-12-04T08:27:07.3412392Z * [new branch] origin/gh/fxdawnn/1/orig -> origin/origin/gh/fxdawnn/1/orig 2025-12-04T08:27:07.3418170Z * [new branch] origin/gh/zpcore/14/orig -> origin/origin/gh/zpcore/14/orig 2025-12-04T08:27:07.3418345Z * [new branch] oulgen-patch-1 -> origin/oulgen-patch-1 2025-12-04T08:27:07.3418481Z * [new branch] oulgen-patch-2 -> origin/oulgen-patch-2 2025-12-04T08:27:07.3418636Z * [new branch] oulgen-patch-3 -> origin/oulgen-patch-3 2025-12-04T08:27:07.3418756Z * [new branch] oulgen-patch-4 -> origin/oulgen-patch-4 2025-12-04T08:27:07.3423380Z * [new branch] padded-tensor -> origin/padded-tensor 2025-12-04T08:27:07.3423524Z * [new branch] pca2 -> origin/pca2 2025-12-04T08:27:07.3423669Z * [new branch] per_channel_backup -> origin/per_channel_backup 2025-12-04T08:27:07.3423801Z * [new branch] perf_ops -> origin/perf_ops 2025-12-04T08:27:07.3423923Z * [new branch] perf_ops_2_9 -> origin/perf_ops_2_9 2025-12-04T08:27:07.3424291Z * [new branch] pianpwk-patch-1 -> origin/pianpwk-patch-1 2025-12-04T08:27:07.3424465Z * [new branch] pianpwk/__draft_debug_mode -> origin/pianpwk/__draft_debug_mode 2025-12-04T08:27:07.3427636Z * [new branch] pianpwk/_debug_mode_for_triton_draft -> origin/pianpwk/_debug_mode_for_triton_draft 2025-12-04T08:27:07.3427932Z * [new branch] pianpwk/_debug_nn_module_compile -> origin/pianpwk/_debug_nn_module_compile 2025-12-04T08:27:07.3428192Z * [new branch] pianpwk/_draft_triton_11_3 -> origin/pianpwk/_draft_triton_11_3 2025-12-04T08:27:07.3428372Z * [new branch] pianpwk/_manual_bucket_draft -> origin/pianpwk/_manual_bucket_draft 2025-12-04T08:27:07.3428633Z * [new branch] pianpwk/_profile_w_dispatch_keys -> origin/pianpwk/_profile_w_dispatch_keys 2025-12-04T08:27:07.3432670Z * [new branch] pianpwk/_super_draft_debug_mode -> origin/pianpwk/_super_draft_debug_mode 2025-12-04T08:27:07.3432944Z * [new branch] pianpwk/_unbacked_local_shard_size -> origin/pianpwk/_unbacked_local_shard_size 2025-12-04T08:27:07.3433111Z * [new branch] pianpwk/anomaly_tb -> origin/pianpwk/anomaly_tb 2025-12-04T08:27:07.3433322Z * [new branch] pianpwk/auto_fx_annotate -> origin/pianpwk/auto_fx_annotate 2025-12-04T08:27:07.3433538Z * [new branch] pianpwk/backed_size_oblivious_export -> origin/pianpwk/backed_size_oblivious_export 2025-12-04T08:27:07.3433780Z * [new branch] pianpwk/bert_dynamic_perf -> origin/pianpwk/bert_dynamic_perf 2025-12-04T08:27:07.3434020Z * [new branch] pianpwk/debug_fwd_stack_traces -> origin/pianpwk/debug_fwd_stack_traces 2025-12-04T08:27:07.3434253Z * [new branch] pianpwk/debug_hash_tensor -> origin/pianpwk/debug_hash_tensor 2025-12-04T08:27:07.3434430Z * [new branch] pianpwk/debug_mode_annotate -> origin/pianpwk/debug_mode_annotate 2025-12-04T08:27:07.3435161Z * [new branch] pianpwk/debug_mode_defaults -> origin/pianpwk/debug_mode_defaults 2025-12-04T08:27:07.3435522Z * [new branch] pianpwk/debug_mode_hacks -> origin/pianpwk/debug_mode_hacks 2025-12-04T08:27:07.3436113Z * [new branch] pianpwk/debug_mode_opcall_refactor -> origin/pianpwk/debug_mode_opcall_refactor 2025-12-04T08:27:07.3436314Z * [new branch] pianpwk/debug_mode_show_ids -> origin/pianpwk/debug_mode_show_ids 2025-12-04T08:27:07.3436482Z * [new branch] pianpwk/debug_mode_triton -> origin/pianpwk/debug_mode_triton 2025-12-04T08:27:07.3436797Z * [new branch] pianpwk/debug_show_stack_trace -> origin/pianpwk/debug_show_stack_trace 2025-12-04T08:27:07.3437351Z * [new branch] pianpwk/debug_wait_on_collective -> origin/pianpwk/debug_wait_on_collective 2025-12-04T08:27:07.3437585Z * [new branch] pianpwk/debugmode_compile_tf -> origin/pianpwk/debugmode_compile_tf 2025-12-04T08:27:07.3437854Z * [new branch] pianpwk/dispatch_key_debugging_for_debug -> origin/pianpwk/dispatch_key_debugging_for_debug 2025-12-04T08:27:07.3438269Z * [new branch] pianpwk/draft_debug_mode_tfcompile -> origin/pianpwk/draft_debug_mode_tfcompile 2025-12-04T08:27:07.3443412Z * [new branch] pianpwk/draft_multikernel_nn -> origin/pianpwk/draft_multikernel_nn 2025-12-04T08:27:07.3445938Z * [new branch] pianpwk/draft_multikernel_status_10_5 -> origin/pianpwk/draft_multikernel_status_10_5 2025-12-04T08:27:07.3450488Z * [new branch] pianpwk/dtensor_custom_chunk -> origin/pianpwk/dtensor_custom_chunk 2025-12-04T08:27:07.3450816Z * [new branch] pianpwk/dtensor_unbacked_keypath -> origin/pianpwk/dtensor_unbacked_keypath 2025-12-04T08:27:07.3455282Z * [new branch] pianpwk/event_list_tree -> origin/pianpwk/event_list_tree 2025-12-04T08:27:07.3455628Z * [new branch] pianpwk/false_numel_refs -> origin/pianpwk/false_numel_refs 2025-12-04T08:27:07.3455879Z * [new branch] pianpwk/maybe_guard_rel -> origin/pianpwk/maybe_guard_rel 2025-12-04T08:27:07.3456099Z * [new branch] pianpwk/multikernel_hints_draft -> origin/pianpwk/multikernel_hints_draft 2025-12-04T08:27:07.3456303Z * [new branch] pianpwk/no_size_oblivious_slice_scat -> origin/pianpwk/no_size_oblivious_slice_scat 2025-12-04T08:27:07.3456582Z * [new branch] pianpwk/oblivious_reshape_view_better -> origin/pianpwk/oblivious_reshape_view_better 2025-12-04T08:27:07.3456736Z * [new branch] pianpwk/pre_forward_hook -> origin/pianpwk/pre_forward_hook 2025-12-04T08:27:07.3461786Z * [new branch] pianpwk/skip_python_keys_alternate -> origin/pianpwk/skip_python_keys_alternate 2025-12-04T08:27:07.3465653Z * [new branch] pianpwk/skip_python_keys_in_guards -> origin/pianpwk/skip_python_keys_in_guards 2025-12-04T08:27:07.3471115Z * [new branch] pianpwk/sym_tokens_draft -> origin/pianpwk/sym_tokens_draft 2025-12-04T08:27:07.3475431Z * [new branch] pianpwk/symint_one_hot -> origin/pianpwk/symint_one_hot 2025-12-04T08:27:07.3475694Z * [new branch] pianpwk/test_pointwise_guard_or_false -> origin/pianpwk/test_pointwise_guard_or_false 2025-12-04T08:27:07.3475915Z * [new branch] pianpwk/totally_draft_sym_wrap -> origin/pianpwk/totally_draft_sym_wrap 2025-12-04T08:27:07.3476080Z * [new branch] pianpwk/try_dumb_stuff -> origin/pianpwk/try_dumb_stuff 2025-12-04T08:27:07.3476249Z * [new branch] pianpwk/try_dumb_stuff_2 -> origin/pianpwk/try_dumb_stuff_2 2025-12-04T08:27:07.3476441Z * [new branch] pianpwk/unbacked_dtensor_mm -> origin/pianpwk/unbacked_dtensor_mm 2025-12-04T08:27:07.3476622Z * [new branch] pianpwk/unbacked_tracing_12_2 -> origin/pianpwk/unbacked_tracing_12_2 2025-12-04T08:27:07.3476794Z * [new branch] pianpwk/user_symints -> origin/pianpwk/user_symints 2025-12-04T08:27:07.3476962Z * [new branch] pianpwk/wan21_reshape -> origin/pianpwk/wan21_reshape 2025-12-04T08:27:07.3477143Z * [new branch] piz/fix_partial_backward_1112 -> origin/piz/fix_partial_backward_1112 2025-12-04T08:27:07.3477444Z * [new branch] piz/prop_cache_clean -> origin/piz/prop_cache_clean 2025-12-04T08:27:07.3477593Z * [new branch] pool-separate -> origin/pool-separate 2025-12-04T08:27:07.3477753Z * [new branch] pr-156087 -> origin/pr-156087 2025-12-04T08:27:07.3477884Z * [new branch] pr/131860 -> origin/pr/131860 2025-12-04T08:27:07.3478183Z * [new branch] predispatch_to -> origin/predispatch_to 2025-12-04T08:27:07.3478340Z * [new branch] protect-c17 -> origin/protect-c17 2025-12-04T08:27:07.3478537Z * [new branch] pt-opt-cuda3 -> origin/pt-opt-cuda3 2025-12-04T08:27:07.3478716Z * [new branch] python_compiled_autograd -> origin/python_compiled_autograd 2025-12-04T08:27:07.3478972Z * [new branch] q1l1/fix_device_moved_constant_type_unknown -> origin/q1l1/fix_device_moved_constant_type_unknown 2025-12-04T08:27:07.3479245Z * [new branch] q1l1/fix_wrong_default_type_for_kernel_call_args -> origin/q1l1/fix_wrong_default_type_for_kernel_call_args 2025-12-04T08:27:07.3479423Z * [new branch] qchip/export-D54134695 -> origin/qchip/export-D54134695 2025-12-04T08:27:07.3479563Z * [new branch] quote-pytest_cache -> origin/quote-pytest_cache 2025-12-04T08:27:07.3479744Z * [new branch] reland-accgrad-stream-warn -> origin/reland-accgrad-stream-warn 2025-12-04T08:27:07.3479866Z * [new branch] release/1.10 -> origin/release/1.10 2025-12-04T08:27:07.3480032Z * [new branch] release/1.11 -> origin/release/1.11 2025-12-04T08:27:07.3480159Z * [new branch] release/1.12 -> origin/release/1.12 2025-12-04T08:27:07.3480276Z * [new branch] release/1.13 -> origin/release/1.13 2025-12-04T08:27:07.3480398Z * [new branch] release/1.4 -> origin/release/1.4 2025-12-04T08:27:07.3480531Z * [new branch] release/1.4.1 -> origin/release/1.4.1 2025-12-04T08:27:07.3480647Z * [new branch] release/1.5 -> origin/release/1.5 2025-12-04T08:27:07.3480770Z * [new branch] release/1.6 -> origin/release/1.6 2025-12-04T08:27:07.3480885Z * [new branch] release/1.7 -> origin/release/1.7 2025-12-04T08:27:07.3480999Z * [new branch] release/1.8 -> origin/release/1.8 2025-12-04T08:27:07.3481122Z * [new branch] release/1.9 -> origin/release/1.9 2025-12-04T08:27:07.3481239Z * [new branch] release/2.0 -> origin/release/2.0 2025-12-04T08:27:07.3481361Z * [new branch] release/2.1 -> origin/release/2.1 2025-12-04T08:27:07.3481480Z * [new branch] release/2.2 -> origin/release/2.2 2025-12-04T08:27:07.3481781Z * [new branch] release/2.3 -> origin/release/2.3 2025-12-04T08:27:07.3482225Z * [new branch] release/2.4 -> origin/release/2.4 2025-12-04T08:27:07.3482371Z * [new branch] release/2.5 -> origin/release/2.5 2025-12-04T08:27:07.3482498Z * [new branch] release/2.6 -> origin/release/2.6 2025-12-04T08:27:07.3482625Z * [new branch] release/2.7 -> origin/release/2.7 2025-12-04T08:27:07.3484270Z * [new branch] release/2.8 -> origin/release/2.8 2025-12-04T08:27:07.3484599Z * [new branch] release/2.9 -> origin/release/2.9 2025-12-04T08:27:07.3484747Z * [new branch] release_notes -> origin/release_notes 2025-12-04T08:27:07.3486668Z * [new branch] remove_pyinterpreter -> origin/remove_pyinterpreter 2025-12-04T08:27:07.3489742Z * [new branch] replace-pytorch-labs-20250812-195836 -> origin/replace-pytorch-labs-20250812-195836 2025-12-04T08:27:07.3489996Z * [new branch] replace-pytorch-labs-20250812-200248 -> origin/replace-pytorch-labs-20250812-200248 2025-12-04T08:27:07.3490221Z * [new branch] replace-pytorch-labs-20250812-200324 -> origin/replace-pytorch-labs-20250812-200324 2025-12-04T08:27:07.3490430Z * [new branch] replace-pytorch-labs-20250812-204020 -> origin/replace-pytorch-labs-20250812-204020 2025-12-04T08:27:07.3490673Z * [new branch] revert-131069-gh/krzysztofjordan/1/head -> origin/revert-131069-gh/krzysztofjordan/1/head 2025-12-04T08:27:07.3496454Z * [new branch] revert-131469-gh/andrewor14/51/head -> origin/revert-131469-gh/andrewor14/51/head 2025-12-04T08:27:07.3496714Z * [new branch] revert-152361-gh/fadara01/1/head -> origin/revert-152361-gh/fadara01/1/head 2025-12-04T08:27:07.3497120Z * [new branch] revert-156870-gh/skarjala/3/head -> origin/revert-156870-gh/skarjala/3/head 2025-12-04T08:27:07.3497534Z * [new branch] revert-157914-cherry-pick-157503-by-pytorch_bot_bot_ -> origin/revert-157914-cherry-pick-157503-by-pytorch_bot_bot_ 2025-12-04T08:27:07.3498106Z * [new branch] revert-hoo-invoke-subgraph -> origin/revert-hoo-invoke-subgraph 2025-12-04T08:27:07.3498316Z * [new branch] revert_always_build_distributed -> origin/revert_always_build_distributed 2025-12-04T08:27:07.3498464Z * [new branch] rms_norm_patch -> origin/rms_norm_patch 2025-12-04T08:27:07.3498641Z * [new branch] ruisi/fix_all_to_all_estimation -> origin/ruisi/fix_all_to_all_estimation 2025-12-04T08:27:07.3498935Z * [new branch] ruisi/fix_comm_estimation -> origin/ruisi/fix_comm_estimation 2025-12-04T08:27:07.3503495Z * [new branch] ruisi/fix_dynamic_shape_estimation -> origin/ruisi/fix_dynamic_shape_estimation 2025-12-04T08:27:07.3503871Z * [new branch] ruisi/fix_llama3_autobucketing -> origin/ruisi/fix_llama3_autobucketing 2025-12-04T08:27:07.3504094Z * [new branch] ruisi/fix_manual_bucketing_ep_pass -> origin/ruisi/fix_manual_bucketing_ep_pass 2025-12-04T08:27:07.3504341Z * [new branch] ruisi/manual_bucket_pass -> origin/ruisi/manual_bucket_pass 2025-12-04T08:27:07.3504771Z * [new branch] ryanguo99/cleanup-dynamo-expected-failures -> origin/ryanguo99/cleanup-dynamo-expected-failures 2025-12-04T08:27:07.3504956Z * [new branch] ryanguo99/fix-closure-var -> origin/ryanguo99/fix-closure-var 2025-12-04T08:27:07.3505119Z * [new branch] rzou/faketensor_bench -> origin/rzou/faketensor_bench 2025-12-04T08:27:07.3505249Z * [new branch] rzou/njt -> origin/rzou/njt 2025-12-04T08:27:07.3505373Z * [new branch] rzou/pca -> origin/rzou/pca 2025-12-04T08:27:07.3505514Z * [new branch] rzou/realprop -> origin/rzou/realprop 2025-12-04T08:27:07.3505644Z * [new branch] samplevllm -> origin/samplevllm 2025-12-04T08:27:07.3510006Z * [new branch] sanchitintel/weird_thing_with_test_cpu_select_algorithm -> origin/sanchitintel/weird_thing_with_test_cpu_select_algorithm 2025-12-04T08:27:07.3510282Z * [new branch] sapling-pr-archive-SS-JIA -> origin/sapling-pr-archive-SS-JIA 2025-12-04T08:27:07.3510558Z * [new branch] sapling-pr-archive-tushar00jain -> origin/sapling-pr-archive-tushar00jain 2025-12-04T08:27:07.3510754Z * [new branch] save -> origin/save 2025-12-04T08:27:07.3510970Z * [new branch] scaled_mm -> origin/scaled_mm 2025-12-04T08:27:07.3511173Z * [new branch] scan_attempt -> origin/scan_attempt 2025-12-04T08:27:07.3511301Z * [new branch] sdym/2.5.1 -> origin/sdym/2.5.1 2025-12-04T08:27:07.3514031Z * [new branch] sekyondaMeta-dynamoconfig-fix -> origin/sekyondaMeta-dynamoconfig-fix 2025-12-04T08:27:07.3514307Z * [new branch] shengf/fx-xform-perf -> origin/shengf/fx-xform-perf 2025-12-04T08:27:07.3514497Z * [new branch] shoumikhin-patch-1 -> origin/shoumikhin-patch-1 2025-12-04T08:27:07.3514751Z * [new branch] solve-accuracy-fix -> origin/solve-accuracy-fix 2025-12-04T08:27:07.3514913Z * [new branch] some_rocm_inductor_skips -> origin/some_rocm_inductor_skips 2025-12-04T08:27:07.3515145Z * [new branch] soulitzer/stash-tls-ac -> origin/soulitzer/stash-tls-ac 2025-12-04T08:27:07.3515312Z * [new branch] sparse-mm-bf16-support -> origin/sparse-mm-bf16-support 2025-12-04T08:27:07.3516346Z * [new branch] starterTaskUpdate -> origin/starterTaskUpdate 2025-12-04T08:27:07.3516774Z * [new branch] suo -> origin/suo 2025-12-04T08:27:07.3517955Z * [new branch] sve-poc -> origin/sve-poc 2025-12-04T08:27:07.3518612Z * [new branch] switch-bn -> origin/switch-bn 2025-12-04T08:27:07.3519486Z * [new branch] sy_annotation_in_autograd_hop -> origin/sy_annotation_in_autograd_hop 2025-12-04T08:27:07.3522667Z * [new branch] sy_aot_eager_record -> origin/sy_aot_eager_record 2025-12-04T08:27:07.3522840Z * [new branch] sy_custom_bucketing -> origin/sy_custom_bucketing 2025-12-04T08:27:07.3522979Z * [new branch] sy_debug_mode_test -> origin/sy_debug_mode_test 2025-12-04T08:27:07.3523351Z * [new branch] sy_deserialize -> origin/sy_deserialize 2025-12-04T08:27:07.3523659Z * [new branch] sy_dump_gm_code -> origin/sy_dump_gm_code 2025-12-04T08:27:07.3523991Z * [new branch] sy_exp -> origin/sy_exp 2025-12-04T08:27:07.3525554Z * [new branch] sy_export_annotation -> origin/sy_export_annotation 2025-12-04T08:27:07.3525877Z * [new branch] sy_invoke_subgraph -> origin/sy_invoke_subgraph 2025-12-04T08:27:07.3526268Z * [new branch] sy_kernel_bw_name -> origin/sy_kernel_bw_name 2025-12-04T08:27:07.3528700Z * [new branch] sy_multi_arch -> origin/sy_multi_arch 2025-12-04T08:27:07.3529036Z * [new branch] sy_nn_module_stack -> origin/sy_nn_module_stack 2025-12-04T08:27:07.3529205Z * [new branch] sy_original_dtensor -> origin/sy_original_dtensor 2025-12-04T08:27:07.3529585Z * [new branch] sy_profiler_cia -> origin/sy_profiler_cia 2025-12-04T08:27:07.3534213Z * [new branch] symm_mem_sync -> origin/symm_mem_sync 2025-12-04T08:27:07.3534564Z * [new branch] sympy-bottleneck-repro -> origin/sympy-bottleneck-repro 2025-12-04T08:27:07.3534756Z * [new branch] tensordict_integration -> origin/tensordict_integration 2025-12-04T08:27:07.3534911Z * [new branch] test-move-conda-builds -> origin/test-move-conda-builds 2025-12-04T08:27:07.3535043Z * [new branch] test-old -> origin/test-old 2025-12-04T08:27:07.3535310Z * [new branch] test/bmm_heur -> origin/test/bmm_heur 2025-12-04T08:27:07.3535851Z * [new branch] tianren/customOp_autotune_fix -> origin/tianren/customOp_autotune_fix 2025-12-04T08:27:07.3536514Z * [new branch] tianren/customOp_enable_max_autotune -> origin/tianren/customOp_enable_max_autotune 2025-12-04T08:27:07.3536966Z * [new branch] tianren/customOp_fusion -> origin/tianren/customOp_fusion 2025-12-04T08:27:07.3540486Z * [new branch] tianren/customop_collectiveop_benchmark -> origin/tianren/customop_collectiveop_benchmark 2025-12-04T08:27:07.3541143Z * [new branch] tianren/customop_collectiveop_benchmark_fix -> origin/tianren/customop_collectiveop_benchmark_fix 2025-12-04T08:27:07.3541436Z * [new branch] tianren/customop_dynamic_config -> origin/tianren/customop_dynamic_config 2025-12-04T08:27:07.3541628Z * [new branch] tianren/dynamic_range_input -> origin/tianren/dynamic_range_input 2025-12-04T08:27:07.3542432Z * [new branch] tianren/dynamic_range_input_fix -> origin/tianren/dynamic_range_input_fix 2025-12-04T08:27:07.3542685Z * [new branch] tianren/dynamic_range_input_merge -> origin/tianren/dynamic_range_input_merge 2025-12-04T08:27:07.3542909Z * [new branch] tianren/flex_paged_attn_fix_temp -> origin/tianren/flex_paged_attn_fix_temp 2025-12-04T08:27:07.3544745Z * [new branch] tianren/fx_codegen_dump -> origin/tianren/fx_codegen_dump 2025-12-04T08:27:07.3545128Z * [new branch] tianren/symmetric_memory -> origin/tianren/symmetric_memory 2025-12-04T08:27:07.3545418Z * [new branch] tianren/test -> origin/tianren/test 2025-12-04T08:27:07.3545629Z * [new branch] tidy_performance_cyy -> origin/tidy_performance_cyy 2025-12-04T08:27:07.3546956Z * [new branch] tmp -> origin/tmp 2025-12-04T08:27:07.3547287Z * [new branch] torchtitan_ep -> origin/torchtitan_ep 2025-12-04T08:27:07.3547735Z * [new branch] torchtitan_integration -> origin/torchtitan_integration 2025-12-04T08:27:07.3550796Z * [new branch] trace_fsdp_torchtune_lora -> origin/trace_fsdp_torchtune_lora 2025-12-04T08:27:07.3551332Z * [new branch] traceable_fsdp_unit_tests -> origin/traceable_fsdp_unit_tests 2025-12-04T08:27:07.3551658Z * [new branch] tree_loop_vec_base -> origin/tree_loop_vec_base 2025-12-04T08:27:07.3551837Z * [new branch] triton_kernel -> origin/triton_kernel 2025-12-04T08:27:07.3552097Z * [new branch] tt_pkg_1908 -> origin/tt_pkg_1908 2025-12-04T08:27:07.3552699Z * [new branch] type_dec -> origin/type_dec 2025-12-04T08:27:07.3553076Z * [new branch] udate-sphinx-dependancies -> origin/udate-sphinx-dependancies 2025-12-04T08:27:07.3554573Z * [new branch] update-audio-commit-hash/17630256502-1803-1 -> origin/update-audio-commit-hash/17630256502-1803-1 2025-12-04T08:27:07.3554848Z * [new branch] update-audio-commit-hash/19087141161-1916-1 -> origin/update-audio-commit-hash/19087141161-1916-1 2025-12-04T08:27:07.3555149Z * [new branch] update-audio-commit-hash/19250643381-1929-1 -> origin/update-audio-commit-hash/19250643381-1929-1 2025-12-04T08:27:07.3555831Z * [new branch] update-audio-commit-hash/19397724337-1935-1 -> origin/update-audio-commit-hash/19397724337-1935-1 2025-12-04T08:27:07.3556464Z * [new branch] update-audio-commit-hash/19555670148-1941-1 -> origin/update-audio-commit-hash/19555670148-1941-1 2025-12-04T08:27:07.3557595Z * [new branch] update-audio-commit-hash/19750627930-1946-1 -> origin/update-audio-commit-hash/19750627930-1946-1 2025-12-04T08:27:07.3558885Z * [new branch] update-triton-commit-hash/13663274526-1487-2 -> origin/update-triton-commit-hash/13663274526-1487-2 2025-12-04T08:27:07.3559251Z * [new branch] update-vision-commit-hash/19087141161-1916-1 -> origin/update-vision-commit-hash/19087141161-1916-1 2025-12-04T08:27:07.3562029Z * [new branch] update-vision-commit-hash/19184897099-1925-1 -> origin/update-vision-commit-hash/19184897099-1925-1 2025-12-04T08:27:07.3562456Z * [new branch] update-vision-commit-hash/19250643381-1929-1 -> origin/update-vision-commit-hash/19250643381-1929-1 2025-12-04T08:27:07.3562818Z * [new branch] update-vision-commit-hash/19381328640-1934-1 -> origin/update-vision-commit-hash/19381328640-1934-1 2025-12-04T08:27:07.3563317Z * [new branch] update-vision-commit-hash/19485237164-1938-1 -> origin/update-vision-commit-hash/19485237164-1938-1 2025-12-04T08:27:07.3564013Z * [new branch] update-vllm-commit-hash/18451675449-1879-1 -> origin/update-vllm-commit-hash/18451675449-1879-1 2025-12-04T08:27:07.3564347Z * [new branch] update-vllm-dockerfile -> origin/update-vllm-dockerfile 2025-12-04T08:27:07.3570976Z * [new branch] update-xla-commit-hash/19224287370-211-1 -> origin/update-xla-commit-hash/19224287370-211-1 2025-12-04T08:27:07.3571398Z * [new branch] update-xla-commit-hash/19422028566-212-1 -> origin/update-xla-commit-hash/19422028566-212-1 2025-12-04T08:27:07.3571718Z * [new branch] update-xla-commit-hash/19626841311-213-1 -> origin/update-xla-commit-hash/19626841311-213-1 2025-12-04T08:27:07.3571971Z * [new branch] update_docs_torch_multinomial_issue#125388 -> origin/update_docs_torch_multinomial_issue#125388 2025-12-04T08:27:07.3572133Z * [new branch] update_operator_readme -> origin/update_operator_readme 2025-12-04T08:27:07.3572420Z * [new branch] update_slow_tests_1722488736 -> origin/update_slow_tests_1722488736 2025-12-04T08:27:07.3572586Z * [new branch] update_slow_tests_1722879173 -> origin/update_slow_tests_1722879173 2025-12-04T08:27:07.3572822Z * [new branch] update_slow_tests_1762155677 -> origin/update_slow_tests_1762155677 2025-12-04T08:27:07.3573517Z * [new branch] update_slow_tests_1763365283 -> origin/update_slow_tests_1763365283 2025-12-04T08:27:07.3573916Z * [new branch] update_submodule_FBGEMM -> origin/update_submodule_FBGEMM 2025-12-04T08:27:07.3574175Z * [new branch] update_submodule_kineto -> origin/update_submodule_kineto 2025-12-04T08:27:07.3574440Z * [new branch] update_submodule_tensorpipe -> origin/update_submodule_tensorpipe 2025-12-04T08:27:07.3574625Z * [new branch] upload-tests-for-autorevert -> origin/upload-tests-for-autorevert 2025-12-04T08:27:07.3574742Z * [new branch] v0.1.2 -> origin/v0.1.2 2025-12-04T08:27:07.3574977Z * [new branch] v1.0.1 -> origin/v1.0.1 2025-12-04T08:27:07.3575150Z * [new branch] v1.0.3 -> origin/v1.0.3 2025-12-04T08:27:07.3575290Z * [new branch] v1.1.0 -> origin/v1.1.0 2025-12-04T08:27:07.3575433Z * [new branch] v1.2.0 -> origin/v1.2.0 2025-12-04T08:27:07.3576635Z * [new branch] v1.3.0 -> origin/v1.3.0 2025-12-04T08:27:07.3577577Z * [new branch] v1.3.1 -> origin/v1.3.1 2025-12-04T08:27:07.3577952Z * [new branch] validate_fn -> origin/validate_fn 2025-12-04T08:27:07.3580485Z * [new branch] validations_2.6 -> origin/validations_2.6 2025-12-04T08:27:07.3580779Z * [new branch] validations_2.8 -> origin/validations_2.8 2025-12-04T08:27:07.3581188Z * [new branch] varlen-api -> origin/varlen-api 2025-12-04T08:27:07.3581418Z * [new branch] varlen-api-backup -> origin/varlen-api-backup 2025-12-04T08:27:07.3581673Z * [new branch] varlen_batch_invariance -> origin/varlen_batch_invariance 2025-12-04T08:27:07.3583015Z * [new branch] viable/strict -> origin/viable/strict 2025-12-04T08:27:07.3586905Z * [new branch] vishal9-team/dtensor_parallelism_toy -> origin/vishal9-team/dtensor_parallelism_toy 2025-12-04T08:27:07.3587242Z * [new branch] vllmbuildci -> origin/vllmbuildci 2025-12-04T08:27:07.3587400Z * [new branch] vllmpin -> origin/vllmpin 2025-12-04T08:27:07.3587705Z * [new branch] vscode-recommend-pyrefly -> origin/vscode-recommend-pyrefly 2025-12-04T08:27:07.3587837Z * [new branch] wdvr-patch-1 -> origin/wdvr-patch-1 2025-12-04T08:27:07.3588120Z * [new branch] wdvr/iss_145259 -> origin/wdvr/iss_145259 2025-12-04T08:27:07.3588974Z * [new branch] whc/pei -> origin/whc/pei 2025-12-04T08:27:07.3589393Z * [new branch] whc/pp_fix -> origin/whc/pp_fix 2025-12-04T08:27:07.3592927Z * [new branch] whc/sharding -> origin/whc/sharding 2025-12-04T08:27:07.3593236Z * [new branch] whc/sharding2 -> origin/whc/sharding2 2025-12-04T08:27:07.3593451Z * [new branch] whc/uneven -> origin/whc/uneven 2025-12-04T08:27:07.3593621Z * [new branch] whc/uneven-merge -> origin/whc/uneven-merge 2025-12-04T08:27:07.3593777Z * [new branch] win_warnings -> origin/win_warnings 2025-12-04T08:27:07.3593946Z * [new branch] windows_libtorch_free -> origin/windows_libtorch_free 2025-12-04T08:27:07.3594338Z * [new branch] xmfan-war -> origin/xmfan-war 2025-12-04T08:27:07.3595674Z * [new branch] xmfan/ca_0516 -> origin/xmfan/ca_0516 2025-12-04T08:27:07.3595915Z * [new branch] xmfan/ca_1051b93192 -> origin/xmfan/ca_1051b93192 2025-12-04T08:27:07.3597108Z * [new branch] xmfan/ca_1a722f62c248391fc4a542e8851a5559aa356ae8 -> origin/xmfan/ca_1a722f62c248391fc4a542e8851a5559aa356ae8 2025-12-04T08:27:07.3597634Z * [new branch] xmfan/ca_5a2be192d1 -> origin/xmfan/ca_5a2be192d1 2025-12-04T08:27:07.3598448Z * [new branch] xmfan/ca_9d59b516e9 -> origin/xmfan/ca_9d59b516e9 2025-12-04T08:27:07.3598750Z * [new branch] xmfan/ca_apr8 -> origin/xmfan/ca_apr8 2025-12-04T08:27:07.3603064Z * [new branch] xmfan/ca_base -> origin/xmfan/ca_base 2025-12-04T08:27:07.3608484Z * [new branch] xmfan/ca_dynamic -> origin/xmfan/ca_dynamic 2025-12-04T08:27:07.3610466Z * [new branch] xmfan/ca_fix_dyn -> origin/xmfan/ca_fix_dyn 2025-12-04T08:27:07.3610754Z * [new branch] xmfan/ca_fix_lowering -> origin/xmfan/ca_fix_lowering 2025-12-04T08:27:07.3614866Z * [new branch] xmfan/ca_fix_polyfills -> origin/xmfan/ca_fix_polyfills 2025-12-04T08:27:07.3615132Z * [new branch] xmfan/ca_jan3 -> origin/xmfan/ca_jan3 2025-12-04T08:27:07.3615467Z * [new branch] xmfan/ca_jun18 -> origin/xmfan/ca_jun18 2025-12-04T08:27:07.3615624Z * [new branch] xmfan/ca_jun24 -> origin/xmfan/ca_jun24 2025-12-04T08:27:07.3615764Z * [new branch] xmfan/ca_nested -> origin/xmfan/ca_nested 2025-12-04T08:27:07.3615921Z * [new branch] xmfan/ca_overhead -> origin/xmfan/ca_overhead 2025-12-04T08:27:07.3616165Z * [new branch] xmfan/ca_overhead_0eba7e5451 -> origin/xmfan/ca_overhead_0eba7e5451 2025-12-04T08:27:07.3616317Z * [new branch] xmfan/cacu_jun18 -> origin/xmfan/cacu_jun18 2025-12-04T08:27:07.3616451Z * [new branch] xmfan/cacu_jun19 -> origin/xmfan/cacu_jun19 2025-12-04T08:27:07.3617043Z * [new branch] xmfan/cacu_jun4 -> origin/xmfan/cacu_jun4 2025-12-04T08:27:07.3622504Z * [new branch] xmfan/disable_duck_shape -> origin/xmfan/disable_duck_shape 2025-12-04T08:27:07.3622864Z * [new branch] xmfan/fca_cpp_node_passthrough -> origin/xmfan/fca_cpp_node_passthrough 2025-12-04T08:27:07.3628012Z * [new branch] xmfan/post_3945954741e2d37023c5d6954f9483008e0892f9 -> origin/xmfan/post_3945954741e2d37023c5d6954f9483008e0892f9 2025-12-04T08:27:07.3630891Z * [new branch] xmfan/pre_3945954741e2d37023c5d6954f9483008e0892f9 -> origin/xmfan/pre_3945954741e2d37023c5d6954f9483008e0892f9 2025-12-04T08:27:07.3631175Z * [new branch] xmfan/single_step -> origin/xmfan/single_step 2025-12-04T08:27:07.3633797Z * [new branch] xmfan/sth_0829 -> origin/xmfan/sth_0829 2025-12-04T08:27:07.3634089Z * [new branch] xmfan/test -> origin/xmfan/test 2025-12-04T08:27:07.3634293Z * [new branch] yguo/debug-0226-constexpr -> origin/yguo/debug-0226-constexpr 2025-12-04T08:27:07.3634452Z * [new branch] yguo/new_latest_changes -> origin/yguo/new_latest_changes 2025-12-04T08:27:07.3634771Z * [new branch] yguo/patch_constexpr_changes -> origin/yguo/patch_constexpr_changes 2025-12-04T08:27:07.3634973Z * [new branch] yiming/bootcamp -> origin/yiming/bootcamp 2025-12-04T08:27:07.3635156Z * [new branch] yiming/run_with_start_end_rng_hop -> origin/yiming/run_with_start_end_rng_hop 2025-12-04T08:27:07.3635306Z * [new branch] yolo-llama3 -> origin/yolo-llama3 2025-12-04T08:27:07.3635456Z * [new branch] zainr/canary-test -> origin/zainr/canary-test 2025-12-04T08:27:07.3635644Z * [new branch] zainr/cleanup-gh-runners -> origin/zainr/cleanup-gh-runners 2025-12-04T08:27:07.3635821Z * [new branch] zainr/pull-migration-c -> origin/zainr/pull-migration-c 2025-12-04T08:27:07.3635947Z * [new branch] zainr/test2 -> origin/zainr/test2 2025-12-04T08:27:07.3636106Z * [new branch] zasdfgbnm-patch-3 -> origin/zasdfgbnm-patch-3 2025-12-04T08:27:07.3636416Z * [new branch] zb2p -> origin/zb2p 2025-12-04T08:27:07.3636596Z * [new branch] zeros-and-scatter-part2 -> origin/zeros-and-scatter-part2 2025-12-04T08:27:07.3636777Z * [new branch] zhxchen17/ci/vllm_lora_oom -> origin/zhxchen17/ci/vllm_lora_oom 2025-12-04T08:27:07.3636981Z * [new branch] zhxchen17/ci/vllm_multimodal_oom -> origin/zhxchen17/ci/vllm_multimodal_oom 2025-12-04T08:27:07.3637146Z * [new branch] zhxchen17/ci/vllm_pin -> origin/zhxchen17/ci/vllm_pin 2025-12-04T08:27:07.3637399Z * [new branch] zhxchen17/dynamo/unsafe_drop_all_guards -> origin/zhxchen17/dynamo/unsafe_drop_all_guards 2025-12-04T08:27:07.3637588Z * [new branch] zhxchen17/export/call_override -> origin/zhxchen17/export/call_override 2025-12-04T08:27:07.3637768Z * [new branch] zhxchen17/export/codemod1 -> origin/zhxchen17/export/codemod1 2025-12-04T08:27:07.3637940Z * [new branch] zhxchen17/export/ctx_return -> origin/zhxchen17/export/ctx_return 2025-12-04T08:27:07.3638440Z * [new branch] zhxchen17/export/disable_side_effect_warn -> origin/zhxchen17/export/disable_side_effect_warn 2025-12-04T08:27:07.3638644Z * [new branch] zhxchen17/export/pytree_check -> origin/zhxchen17/export/pytree_check 2025-12-04T08:27:07.3638815Z * [new branch] zhxchen17/precompile/aoti -> origin/zhxchen17/precompile/aoti 2025-12-04T08:27:07.3639018Z * [new branch] zhxchen17/precompile/globals -> origin/zhxchen17/precompile/globals 2025-12-04T08:27:07.3639244Z * [new branch] zhxchen17/precompile/inductor_guards -> origin/zhxchen17/precompile/inductor_guards 2025-12-04T08:27:07.3639407Z * [new branch] zhxchen17/scratch/0 -> origin/zhxchen17/scratch/0 2025-12-04T08:27:07.3643632Z * [new branch] zhxchen17/torch_export_api_update -> origin/zhxchen17/torch_export_api_update 2025-12-04T08:27:07.3643841Z * [new branch] zhxhcen17/moodycamel -> origin/zhxhcen17/moodycamel 2025-12-04T08:27:07.3643989Z * [new branch] zxiiro/build-times -> origin/zxiiro/build-times 2025-12-04T08:27:07.3644129Z * [new branch] zxiiro/c7i.2xlarge -> origin/zxiiro/c7i.2xlarge 2025-12-04T08:27:07.3644436Z * [new branch] zxiiro/c7i.2xlarge.h100 -> origin/zxiiro/c7i.2xlarge.h100 2025-12-04T08:27:07.3644558Z * [new branch] zxiiro/main -> origin/zxiiro/main 2025-12-04T08:27:07.3648738Z * [new branch] zxiiro/risc64 -> origin/zxiiro/risc64 2025-12-04T08:27:07.3653709Z * [new branch] zxiiro/test-multicloud-arc -> origin/zxiiro/test-multicloud-arc 2025-12-04T08:27:07.3657935Z * [new tag] bc2caa7fdf006894eff7af936babde69ab5a40f8-huydhn-debug -> bc2caa7fdf006894eff7af936babde69ab5a40f8-huydhn-debug 2025-12-04T08:27:07.3661796Z * [new tag] ci/binaries/77164 -> ci/binaries/77164 2025-12-04T08:27:07.3661974Z * [new tag] ciflow/b200/115316 -> ciflow/b200/115316 2025-12-04T08:27:07.3662094Z * [new tag] ciflow/b200/160685 -> ciflow/b200/160685 2025-12-04T08:27:07.3662206Z * [new tag] ciflow/b200/161607 -> ciflow/b200/161607 2025-12-04T08:27:07.3662319Z * [new tag] ciflow/b200/161938 -> ciflow/b200/161938 2025-12-04T08:27:07.3662424Z * [new tag] ciflow/b200/167207 -> ciflow/b200/167207 2025-12-04T08:27:07.3662528Z * [new tag] ciflow/b200/167989 -> ciflow/b200/167989 2025-12-04T08:27:07.3662641Z * [new tag] ciflow/b200/168096 -> ciflow/b200/168096 2025-12-04T08:27:07.3662745Z * [new tag] ciflow/b200/168175 -> ciflow/b200/168175 2025-12-04T08:27:07.3662855Z * [new tag] ciflow/b200/168195 -> ciflow/b200/168195 2025-12-04T08:27:07.3663096Z * [new tag] ciflow/b200/169200 -> ciflow/b200/169200 2025-12-04T08:27:07.3663212Z * [new tag] ciflow/b200/169216 -> ciflow/b200/169216 2025-12-04T08:27:07.3663329Z * [new tag] ciflow/b200/169380 -> ciflow/b200/169380 2025-12-04T08:27:07.3663441Z * [new tag] ciflow/b200/169412 -> ciflow/b200/169412 2025-12-04T08:27:07.3663544Z * [new tag] ciflow/b200/169470 -> ciflow/b200/169470 2025-12-04T08:27:07.3663658Z * [new tag] ciflow/b200/169471 -> ciflow/b200/169471 2025-12-04T08:27:07.3663767Z * [new tag] ciflow/b200/169472 -> ciflow/b200/169472 2025-12-04T08:27:07.3663880Z * [new tag] ciflow/b200/169514 -> ciflow/b200/169514 2025-12-04T08:27:07.3663984Z * [new tag] ciflow/b200/169517 -> ciflow/b200/169517 2025-12-04T08:27:07.3664119Z * [new tag] ciflow/binaries/165922 -> ciflow/binaries/165922 2025-12-04T08:27:07.3664245Z * [new tag] ciflow/binaries/169510 -> ciflow/binaries/169510 2025-12-04T08:27:07.3664386Z * [new tag] ciflow/binaries_wheel/157994 -> ciflow/binaries_wheel/157994 2025-12-04T08:27:07.3664531Z * [new tag] ciflow/binaries_wheel/166829 -> ciflow/binaries_wheel/166829 2025-12-04T08:27:07.3664665Z * [new tag] ciflow/binaries_wheel/167972 -> ciflow/binaries_wheel/167972 2025-12-04T08:27:07.3664796Z * [new tag] ciflow/binaries_wheel/167981 -> ciflow/binaries_wheel/167981 2025-12-04T08:27:07.3664918Z * [new tag] ciflow/dynamo/167695 -> ciflow/dynamo/167695 2025-12-04T08:27:07.3665032Z * [new tag] ciflow/dynamo/168096 -> ciflow/dynamo/168096 2025-12-04T08:27:07.3665216Z * [new tag] ciflow/h100-cutlass-backend/161938 -> ciflow/h100-cutlass-backend/161938 2025-12-04T08:27:07.3665387Z * [new tag] ciflow/h100-cutlass-backend/161940 -> ciflow/h100-cutlass-backend/161940 2025-12-04T08:27:07.3665538Z * [new tag] ciflow/h100-distributed/168923 -> ciflow/h100-distributed/168923 2025-12-04T08:27:07.3665732Z * [new tag] ciflow/h100-symm-mem/167552 -> ciflow/h100-symm-mem/167552 2025-12-04T08:27:07.3665859Z * [new tag] ciflow/h100-symm-mem/168129 -> ciflow/h100-symm-mem/168129 2025-12-04T08:27:07.3665999Z * [new tag] ciflow/h100-symm-mem/168917 -> ciflow/h100-symm-mem/168917 2025-12-04T08:27:07.3666121Z * [new tag] ciflow/h100-symm-mem/169156 -> ciflow/h100-symm-mem/169156 2025-12-04T08:27:07.3666242Z * [new tag] ciflow/h100-symm-mem/169200 -> ciflow/h100-symm-mem/169200 2025-12-04T08:27:07.3666371Z * [new tag] ciflow/h100-symm-mem/169216 -> ciflow/h100-symm-mem/169216 2025-12-04T08:27:07.3666493Z * [new tag] ciflow/h100-symm-mem/169338 -> ciflow/h100-symm-mem/169338 2025-12-04T08:27:07.3666795Z * [new tag] ciflow/h100-symm-mem/169355 -> ciflow/h100-symm-mem/169355 2025-12-04T08:27:07.3666942Z * [new tag] ciflow/h100-symm-mem/169543 -> ciflow/h100-symm-mem/169543 2025-12-04T08:27:07.3667052Z * [new tag] ciflow/h100/115316 -> ciflow/h100/115316 2025-12-04T08:27:07.3667276Z * [new tag] ciflow/h100/160685 -> ciflow/h100/160685 2025-12-04T08:27:07.3667452Z * [new tag] ciflow/h100/160729 -> ciflow/h100/160729 2025-12-04T08:27:07.3668646Z * [new tag] ciflow/h100/161607 -> ciflow/h100/161607 2025-12-04T08:27:07.3672772Z * [new tag] ciflow/h100/161938 -> ciflow/h100/161938 2025-12-04T08:27:07.3672913Z * [new tag] ciflow/h100/167207 -> ciflow/h100/167207 2025-12-04T08:27:07.3673020Z * [new tag] ciflow/h100/167989 -> ciflow/h100/167989 2025-12-04T08:27:07.3673301Z * [new tag] ciflow/h100/168096 -> ciflow/h100/168096 2025-12-04T08:27:07.3673417Z * [new tag] ciflow/h100/168175 -> ciflow/h100/168175 2025-12-04T08:27:07.3673533Z * [new tag] ciflow/h100/168195 -> ciflow/h100/168195 2025-12-04T08:27:07.3673649Z * [new tag] ciflow/h100/168980 -> ciflow/h100/168980 2025-12-04T08:27:07.3673757Z * [new tag] ciflow/h100/169200 -> ciflow/h100/169200 2025-12-04T08:27:07.3673874Z * [new tag] ciflow/h100/169216 -> ciflow/h100/169216 2025-12-04T08:27:07.3673977Z * [new tag] ciflow/h100/169380 -> ciflow/h100/169380 2025-12-04T08:27:07.3674091Z * [new tag] ciflow/h100/169412 -> ciflow/h100/169412 2025-12-04T08:27:07.3674202Z * [new tag] ciflow/h100/169470 -> ciflow/h100/169470 2025-12-04T08:27:07.3674304Z * [new tag] ciflow/h100/169471 -> ciflow/h100/169471 2025-12-04T08:27:07.3674423Z * [new tag] ciflow/h100/169472 -> ciflow/h100/169472 2025-12-04T08:27:07.3674569Z * [new tag] ciflow/h100/169514 -> ciflow/h100/169514 2025-12-04T08:27:07.3676900Z * [new tag] ciflow/inductor-cu126/168096 -> ciflow/inductor-cu126/168096 2025-12-04T08:27:07.3677219Z * [new tag] ciflow/inductor-micro-benchmark-cpu-x86/168096 -> ciflow/inductor-micro-benchmark-cpu-x86/168096 2025-12-04T08:27:07.3677427Z * [new tag] ciflow/inductor-micro-benchmark/166165 -> ciflow/inductor-micro-benchmark/166165 2025-12-04T08:27:07.3677630Z * [new tag] ciflow/inductor-micro-benchmark/168096 -> ciflow/inductor-micro-benchmark/168096 2025-12-04T08:27:07.3678178Z * [new tag] ciflow/inductor-perf-compare/168096 -> ciflow/inductor-perf-compare/168096 2025-12-04T08:27:07.3678714Z * [new tag] ciflow/inductor-perf-test-nightly-rocm-mi300/168073 -> ciflow/inductor-perf-test-nightly-rocm-mi300/168073 2025-12-04T08:27:07.3679026Z * [new tag] ciflow/inductor-perf-test-nightly-rocm-mi300/168096 -> ciflow/inductor-perf-test-nightly-rocm-mi300/168096 2025-12-04T08:27:07.3679461Z * [new tag] ciflow/inductor-perf-test-nightly-rocm-mi300/169024 -> ciflow/inductor-perf-test-nightly-rocm-mi300/169024 2025-12-04T08:27:07.3680075Z * [new tag] ciflow/inductor-perf-test-nightly-rocm-mi355/169024 -> ciflow/inductor-perf-test-nightly-rocm-mi355/169024 2025-12-04T08:27:07.3681609Z * [new tag] ciflow/inductor-perf-test-nightly/168096 -> ciflow/inductor-perf-test-nightly/168096 2025-12-04T08:27:07.3682216Z * [new tag] ciflow/inductor-periodic/168096 -> ciflow/inductor-periodic/168096 2025-12-04T08:27:07.3682412Z * [new tag] ciflow/inductor-periodic/169024 -> ciflow/inductor-periodic/169024 2025-12-04T08:27:07.3682780Z * [new tag] ciflow/inductor-periodic/169425 -> ciflow/inductor-periodic/169425 2025-12-04T08:27:07.3682966Z * [new tag] ciflow/inductor-rocm-mi200/165545 -> ciflow/inductor-rocm-mi200/165545 2025-12-04T08:27:07.3684116Z * [new tag] ciflow/inductor-rocm-mi200/165997 -> ciflow/inductor-rocm-mi200/165997 2025-12-04T08:27:07.3684465Z * [new tag] ciflow/inductor-rocm-mi200/168096 -> ciflow/inductor-rocm-mi200/168096 2025-12-04T08:27:07.3684655Z * [new tag] ciflow/inductor-rocm-mi200/169063 -> ciflow/inductor-rocm-mi200/169063 2025-12-04T08:27:07.3684990Z * [new tag] ciflow/inductor-rocm-mi200/169425 -> ciflow/inductor-rocm-mi200/169425 2025-12-04T08:27:07.3685415Z * [new tag] ciflow/inductor-rocm-mi300/165545 -> ciflow/inductor-rocm-mi300/165545 2025-12-04T08:27:07.3685693Z * [new tag] ciflow/inductor-rocm-mi300/168096 -> ciflow/inductor-rocm-mi300/168096 2025-12-04T08:27:07.3686278Z * [new tag] ciflow/inductor-rocm-mi300/169063 -> ciflow/inductor-rocm-mi300/169063 2025-12-04T08:27:07.3686571Z * [new tag] ciflow/inductor-rocm-mi300/169425 -> ciflow/inductor-rocm-mi300/169425 2025-12-04T08:27:07.3687242Z * [new tag] ciflow/inductor-rocm/162052 -> ciflow/inductor-rocm/162052 2025-12-04T08:27:07.3687540Z * [new tag] ciflow/inductor-rocm/168971 -> ciflow/inductor-rocm/168971 2025-12-04T08:27:07.3688105Z * [new tag] ciflow/inductor-windows/168096 -> ciflow/inductor-windows/168096 2025-12-04T08:27:07.3688565Z * [new tag] ciflow/inductor/144542 -> ciflow/inductor/144542 2025-12-04T08:27:07.3688973Z * [new tag] ciflow/inductor/146506 -> ciflow/inductor/146506 2025-12-04T08:27:07.3689505Z * [new tag] ciflow/inductor/147990 -> ciflow/inductor/147990 2025-12-04T08:27:07.3689967Z * [new tag] ciflow/inductor/148294 -> ciflow/inductor/148294 2025-12-04T08:27:07.3690418Z * [new tag] ciflow/inductor/148492 -> ciflow/inductor/148492 2025-12-04T08:27:07.3690932Z * [new tag] ciflow/inductor/157149 -> ciflow/inductor/157149 2025-12-04T08:27:07.3691396Z * [new tag] ciflow/inductor/157994 -> ciflow/inductor/157994 2025-12-04T08:27:07.3691682Z * [new tag] ciflow/inductor/160174 -> ciflow/inductor/160174 2025-12-04T08:27:07.3692269Z * [new tag] ciflow/inductor/160685 -> ciflow/inductor/160685 2025-12-04T08:27:07.3692501Z * [new tag] ciflow/inductor/160686 -> ciflow/inductor/160686 2025-12-04T08:27:07.3693432Z * [new tag] ciflow/inductor/160687 -> ciflow/inductor/160687 2025-12-04T08:27:07.3693623Z * [new tag] ciflow/inductor/160688 -> ciflow/inductor/160688 2025-12-04T08:27:07.3697408Z * [new tag] ciflow/inductor/160706 -> ciflow/inductor/160706 2025-12-04T08:27:07.3697787Z * [new tag] ciflow/inductor/160729 -> ciflow/inductor/160729 2025-12-04T08:27:07.3697935Z * [new tag] ciflow/inductor/161938 -> ciflow/inductor/161938 2025-12-04T08:27:07.3698057Z * [new tag] ciflow/inductor/161939 -> ciflow/inductor/161939 2025-12-04T08:27:07.3698234Z * [new tag] ciflow/inductor/161940 -> ciflow/inductor/161940 2025-12-04T08:27:07.3698355Z * [new tag] ciflow/inductor/162052 -> ciflow/inductor/162052 2025-12-04T08:27:07.3698475Z * [new tag] ciflow/inductor/162275 -> ciflow/inductor/162275 2025-12-04T08:27:07.3698586Z * [new tag] ciflow/inductor/162795 -> ciflow/inductor/162795 2025-12-04T08:27:07.3698705Z * [new tag] ciflow/inductor/163245 -> ciflow/inductor/163245 2025-12-04T08:27:07.3702012Z * [new tag] ciflow/inductor/163335 -> ciflow/inductor/163335 2025-12-04T08:27:07.3702180Z * [new tag] ciflow/inductor/163503 -> ciflow/inductor/163503 2025-12-04T08:27:07.3702303Z * [new tag] ciflow/inductor/163942 -> ciflow/inductor/163942 2025-12-04T08:27:07.3702428Z * [new tag] ciflow/inductor/165270 -> ciflow/inductor/165270 2025-12-04T08:27:07.3702552Z * [new tag] ciflow/inductor/165274 -> ciflow/inductor/165274 2025-12-04T08:27:07.3702665Z * [new tag] ciflow/inductor/165322 -> ciflow/inductor/165322 2025-12-04T08:27:07.3702777Z * [new tag] ciflow/inductor/165597 -> ciflow/inductor/165597 2025-12-04T08:27:07.3702898Z * [new tag] ciflow/inductor/166063 -> ciflow/inductor/166063 2025-12-04T08:27:07.3703010Z * [new tag] ciflow/inductor/166075 -> ciflow/inductor/166075 2025-12-04T08:27:07.3703132Z * [new tag] ciflow/inductor/166165 -> ciflow/inductor/166165 2025-12-04T08:27:07.3708153Z * [new tag] ciflow/inductor/166254 -> ciflow/inductor/166254 2025-12-04T08:27:07.3708398Z * [new tag] ciflow/inductor/166483 -> ciflow/inductor/166483 2025-12-04T08:27:07.3708584Z * [new tag] ciflow/inductor/166494 -> ciflow/inductor/166494 2025-12-04T08:27:07.3708790Z * [new tag] ciflow/inductor/166545 -> ciflow/inductor/166545 2025-12-04T08:27:07.3708964Z * [new tag] ciflow/inductor/166788 -> ciflow/inductor/166788 2025-12-04T08:27:07.3709095Z * [new tag] ciflow/inductor/166846 -> ciflow/inductor/166846 2025-12-04T08:27:07.3709228Z * [new tag] ciflow/inductor/167300 -> ciflow/inductor/167300 2025-12-04T08:27:07.3709357Z * [new tag] ciflow/inductor/167407 -> ciflow/inductor/167407 2025-12-04T08:27:07.3709486Z * [new tag] ciflow/inductor/167536 -> ciflow/inductor/167536 2025-12-04T08:27:07.3709710Z * [new tag] ciflow/inductor/167552 -> ciflow/inductor/167552 2025-12-04T08:27:07.3709847Z * [new tag] ciflow/inductor/167555 -> ciflow/inductor/167555 2025-12-04T08:27:07.3713841Z * [new tag] ciflow/inductor/167583 -> ciflow/inductor/167583 2025-12-04T08:27:07.3715232Z * [new tag] ciflow/inductor/167599 -> ciflow/inductor/167599 2025-12-04T08:27:07.3715374Z * [new tag] ciflow/inductor/167647 -> ciflow/inductor/167647 2025-12-04T08:27:07.3715500Z * [new tag] ciflow/inductor/167677 -> ciflow/inductor/167677 2025-12-04T08:27:07.3715625Z * [new tag] ciflow/inductor/167680 -> ciflow/inductor/167680 2025-12-04T08:27:07.3715762Z * [new tag] ciflow/inductor/167695 -> ciflow/inductor/167695 2025-12-04T08:27:07.3715889Z * [new tag] ciflow/inductor/167742 -> ciflow/inductor/167742 2025-12-04T08:27:07.3716020Z * [new tag] ciflow/inductor/167768 -> ciflow/inductor/167768 2025-12-04T08:27:07.3716153Z * [new tag] ciflow/inductor/167773 -> ciflow/inductor/167773 2025-12-04T08:27:07.3716276Z * [new tag] ciflow/inductor/167781 -> ciflow/inductor/167781 2025-12-04T08:27:07.3716550Z * [new tag] ciflow/inductor/167880 -> ciflow/inductor/167880 2025-12-04T08:27:07.3716675Z * [new tag] ciflow/inductor/167887 -> ciflow/inductor/167887 2025-12-04T08:27:07.3716808Z * [new tag] ciflow/inductor/167972 -> ciflow/inductor/167972 2025-12-04T08:27:07.3716935Z * [new tag] ciflow/inductor/167989 -> ciflow/inductor/167989 2025-12-04T08:27:07.3717061Z * [new tag] ciflow/inductor/168002 -> ciflow/inductor/168002 2025-12-04T08:27:07.3717201Z * [new tag] ciflow/inductor/168050 -> ciflow/inductor/168050 2025-12-04T08:27:07.3717327Z * [new tag] ciflow/inductor/168051 -> ciflow/inductor/168051 2025-12-04T08:27:07.3717456Z * [new tag] ciflow/inductor/168052 -> ciflow/inductor/168052 2025-12-04T08:27:07.3717591Z * [new tag] ciflow/inductor/168073 -> ciflow/inductor/168073 2025-12-04T08:27:07.3717720Z * [new tag] ciflow/inductor/168096 -> ciflow/inductor/168096 2025-12-04T08:27:07.3717857Z * [new tag] ciflow/inductor/168114 -> ciflow/inductor/168114 2025-12-04T08:27:07.3717982Z * [new tag] ciflow/inductor/168115 -> ciflow/inductor/168115 2025-12-04T08:27:07.3718350Z * [new tag] ciflow/inductor/168127 -> ciflow/inductor/168127 2025-12-04T08:27:07.3718497Z * [new tag] ciflow/inductor/168129 -> ciflow/inductor/168129 2025-12-04T08:27:07.3718619Z * [new tag] ciflow/inductor/168157 -> ciflow/inductor/168157 2025-12-04T08:27:07.3718752Z * [new tag] ciflow/inductor/168175 -> ciflow/inductor/168175 2025-12-04T08:27:07.3718947Z * [new tag] ciflow/inductor/168185 -> ciflow/inductor/168185 2025-12-04T08:27:07.3719083Z * [new tag] ciflow/inductor/168195 -> ciflow/inductor/168195 2025-12-04T08:27:07.3719504Z * [new tag] ciflow/inductor/168209 -> ciflow/inductor/168209 2025-12-04T08:27:07.3719944Z * [new tag] ciflow/inductor/168266 -> ciflow/inductor/168266 2025-12-04T08:27:07.3720474Z * [new tag] ciflow/inductor/168316 -> ciflow/inductor/168316 2025-12-04T08:27:07.3721288Z * [new tag] ciflow/inductor/168326 -> ciflow/inductor/168326 2025-12-04T08:27:07.3721708Z * [new tag] ciflow/inductor/168368 -> ciflow/inductor/168368 2025-12-04T08:27:07.3722102Z * [new tag] ciflow/inductor/168894 -> ciflow/inductor/168894 2025-12-04T08:27:07.3722568Z * [new tag] ciflow/inductor/168934 -> ciflow/inductor/168934 2025-12-04T08:27:07.3725902Z * [new tag] ciflow/inductor/168939 -> ciflow/inductor/168939 2025-12-04T08:27:07.3726070Z * [new tag] ciflow/inductor/168946 -> ciflow/inductor/168946 2025-12-04T08:27:07.3726199Z * [new tag] ciflow/inductor/168950 -> ciflow/inductor/168950 2025-12-04T08:27:07.3726337Z * [new tag] ciflow/inductor/168951 -> ciflow/inductor/168951 2025-12-04T08:27:07.3726460Z * [new tag] ciflow/inductor/168952 -> ciflow/inductor/168952 2025-12-04T08:27:07.3726588Z * [new tag] ciflow/inductor/168955 -> ciflow/inductor/168955 2025-12-04T08:27:07.3726710Z * [new tag] ciflow/inductor/168971 -> ciflow/inductor/168971 2025-12-04T08:27:07.3726837Z * [new tag] ciflow/inductor/168979 -> ciflow/inductor/168979 2025-12-04T08:27:07.3727073Z * [new tag] ciflow/inductor/168980 -> ciflow/inductor/168980 2025-12-04T08:27:07.3729550Z * [new tag] ciflow/inductor/168983 -> ciflow/inductor/168983 2025-12-04T08:27:07.3729880Z * [new tag] ciflow/inductor/169006 -> ciflow/inductor/169006 2025-12-04T08:27:07.3730061Z * [new tag] ciflow/inductor/169023 -> ciflow/inductor/169023 2025-12-04T08:27:07.3730544Z * [new tag] ciflow/inductor/169024 -> ciflow/inductor/169024 2025-12-04T08:27:07.3730959Z * [new tag] ciflow/inductor/169025 -> ciflow/inductor/169025 2025-12-04T08:27:07.3731107Z * [new tag] ciflow/inductor/169066 -> ciflow/inductor/169066 2025-12-04T08:27:07.3731227Z * [new tag] ciflow/inductor/169091 -> ciflow/inductor/169091 2025-12-04T08:27:07.3731344Z * [new tag] ciflow/inductor/169102 -> ciflow/inductor/169102 2025-12-04T08:27:07.3731493Z * [new tag] ciflow/inductor/169103 -> ciflow/inductor/169103 2025-12-04T08:27:07.3732183Z * [new tag] ciflow/inductor/169121 -> ciflow/inductor/169121 2025-12-04T08:27:07.3732476Z * [new tag] ciflow/inductor/169134 -> ciflow/inductor/169134 2025-12-04T08:27:07.3732790Z * [new tag] ciflow/inductor/169135 -> ciflow/inductor/169135 2025-12-04T08:27:07.3733565Z * [new tag] ciflow/inductor/169141 -> ciflow/inductor/169141 2025-12-04T08:27:07.3733741Z * [new tag] ciflow/inductor/169151 -> ciflow/inductor/169151 2025-12-04T08:27:07.3738111Z * [new tag] ciflow/inductor/169161 -> ciflow/inductor/169161 2025-12-04T08:27:07.3738410Z * [new tag] ciflow/inductor/169167 -> ciflow/inductor/169167 2025-12-04T08:27:07.3738707Z * [new tag] ciflow/inductor/169177 -> ciflow/inductor/169177 2025-12-04T08:27:07.3738849Z * [new tag] ciflow/inductor/169185 -> ciflow/inductor/169185 2025-12-04T08:27:07.3739062Z * [new tag] ciflow/inductor/169196 -> ciflow/inductor/169196 2025-12-04T08:27:07.3739459Z * [new tag] ciflow/inductor/169200 -> ciflow/inductor/169200 2025-12-04T08:27:07.3739599Z * [new tag] ciflow/inductor/169204 -> ciflow/inductor/169204 2025-12-04T08:27:07.3739736Z * [new tag] ciflow/inductor/169216 -> ciflow/inductor/169216 2025-12-04T08:27:07.3739855Z * [new tag] ciflow/inductor/169219 -> ciflow/inductor/169219 2025-12-04T08:27:07.3739984Z * [new tag] ciflow/inductor/169220 -> ciflow/inductor/169220 2025-12-04T08:27:07.3740102Z * [new tag] ciflow/inductor/169230 -> ciflow/inductor/169230 2025-12-04T08:27:07.3740237Z * [new tag] ciflow/inductor/169242 -> ciflow/inductor/169242 2025-12-04T08:27:07.3740505Z * [new tag] ciflow/inductor/169245 -> ciflow/inductor/169245 2025-12-04T08:27:07.3744902Z * [new tag] ciflow/inductor/169260 -> ciflow/inductor/169260 2025-12-04T08:27:07.3750204Z * [new tag] ciflow/inductor/169282 -> ciflow/inductor/169282 2025-12-04T08:27:07.3750596Z * [new tag] ciflow/inductor/169286 -> ciflow/inductor/169286 2025-12-04T08:27:07.3750728Z * [new tag] ciflow/inductor/169299 -> ciflow/inductor/169299 2025-12-04T08:27:07.3750862Z * [new tag] ciflow/inductor/169304 -> ciflow/inductor/169304 2025-12-04T08:27:07.3751377Z * [new tag] ciflow/inductor/169305 -> ciflow/inductor/169305 2025-12-04T08:27:07.3751531Z * [new tag] ciflow/inductor/169308 -> ciflow/inductor/169308 2025-12-04T08:27:07.3751647Z * [new tag] ciflow/inductor/169319 -> ciflow/inductor/169319 2025-12-04T08:27:07.3751772Z * [new tag] ciflow/inductor/169326 -> ciflow/inductor/169326 2025-12-04T08:27:07.3751887Z * [new tag] ciflow/inductor/169332 -> ciflow/inductor/169332 2025-12-04T08:27:07.3752024Z * [new tag] ciflow/inductor/169333 -> ciflow/inductor/169333 2025-12-04T08:27:07.3752155Z * [new tag] ciflow/inductor/169336 -> ciflow/inductor/169336 2025-12-04T08:27:07.3752422Z * [new tag] ciflow/inductor/169340 -> ciflow/inductor/169340 2025-12-04T08:27:07.3752559Z * [new tag] ciflow/inductor/169341 -> ciflow/inductor/169341 2025-12-04T08:27:07.3752714Z * [new tag] ciflow/inductor/169343 -> ciflow/inductor/169343 2025-12-04T08:27:07.3752850Z * [new tag] ciflow/inductor/169346 -> ciflow/inductor/169346 2025-12-04T08:27:07.3752979Z * [new tag] ciflow/inductor/169348 -> ciflow/inductor/169348 2025-12-04T08:27:07.3753104Z * [new tag] ciflow/inductor/169350 -> ciflow/inductor/169350 2025-12-04T08:27:07.3753237Z * [new tag] ciflow/inductor/169355 -> ciflow/inductor/169355 2025-12-04T08:27:07.3753377Z * [new tag] ciflow/inductor/169370 -> ciflow/inductor/169370 2025-12-04T08:27:07.3753506Z * [new tag] ciflow/inductor/169375 -> ciflow/inductor/169375 2025-12-04T08:27:07.3753640Z * [new tag] ciflow/inductor/169389 -> ciflow/inductor/169389 2025-12-04T08:27:07.3753774Z * [new tag] ciflow/inductor/169391 -> ciflow/inductor/169391 2025-12-04T08:27:07.3753909Z * [new tag] ciflow/inductor/169393 -> ciflow/inductor/169393 2025-12-04T08:27:07.3754035Z * [new tag] ciflow/inductor/169399 -> ciflow/inductor/169399 2025-12-04T08:27:07.3754202Z * [new tag] ciflow/inductor/169400 -> ciflow/inductor/169400 2025-12-04T08:27:07.3754519Z * [new tag] ciflow/inductor/169415 -> ciflow/inductor/169415 2025-12-04T08:27:07.3761512Z * [new tag] ciflow/inductor/169417 -> ciflow/inductor/169417 2025-12-04T08:27:07.3764266Z * [new tag] ciflow/inductor/169418 -> ciflow/inductor/169418 2025-12-04T08:27:07.3764517Z * [new tag] ciflow/inductor/169430 -> ciflow/inductor/169430 2025-12-04T08:27:07.3764752Z * [new tag] ciflow/inductor/169432 -> ciflow/inductor/169432 2025-12-04T08:27:07.3765021Z * [new tag] ciflow/inductor/169436 -> ciflow/inductor/169436 2025-12-04T08:27:07.3765265Z * [new tag] ciflow/inductor/169437 -> ciflow/inductor/169437 2025-12-04T08:27:07.3765479Z * [new tag] ciflow/inductor/169438 -> ciflow/inductor/169438 2025-12-04T08:27:07.3765719Z * [new tag] ciflow/inductor/169441 -> ciflow/inductor/169441 2025-12-04T08:27:07.3765961Z * [new tag] ciflow/inductor/169446 -> ciflow/inductor/169446 2025-12-04T08:27:07.3766188Z * [new tag] ciflow/inductor/169447 -> ciflow/inductor/169447 2025-12-04T08:27:07.3766415Z * [new tag] ciflow/inductor/169452 -> ciflow/inductor/169452 2025-12-04T08:27:07.3766655Z * [new tag] ciflow/inductor/169455 -> ciflow/inductor/169455 2025-12-04T08:27:07.3766901Z * [new tag] ciflow/inductor/169459 -> ciflow/inductor/169459 2025-12-04T08:27:07.3768434Z * [new tag] ciflow/inductor/169463 -> ciflow/inductor/169463 2025-12-04T08:27:07.3768716Z * [new tag] ciflow/inductor/169476 -> ciflow/inductor/169476 2025-12-04T08:27:07.3772220Z * [new tag] ciflow/inductor/169485 -> ciflow/inductor/169485 2025-12-04T08:27:07.3772575Z * [new tag] ciflow/inductor/169493 -> ciflow/inductor/169493 2025-12-04T08:27:07.3778737Z * [new tag] ciflow/inductor/169496 -> ciflow/inductor/169496 2025-12-04T08:27:07.3783609Z * [new tag] ciflow/inductor/169497 -> ciflow/inductor/169497 2025-12-04T08:27:07.3788639Z * [new tag] ciflow/inductor/169503 -> ciflow/inductor/169503 2025-12-04T08:27:07.3792802Z * [new tag] ciflow/inductor/169504 -> ciflow/inductor/169504 2025-12-04T08:27:07.3794898Z * [new tag] ciflow/inductor/169505 -> ciflow/inductor/169505 2025-12-04T08:27:07.3795193Z * [new tag] ciflow/inductor/169508 -> ciflow/inductor/169508 2025-12-04T08:27:07.3795412Z * [new tag] ciflow/inductor/169509 -> ciflow/inductor/169509 2025-12-04T08:27:07.3795548Z * [new tag] ciflow/inductor/169513 -> ciflow/inductor/169513 2025-12-04T08:27:07.3795692Z * [new tag] ciflow/inductor/169514 -> ciflow/inductor/169514 2025-12-04T08:27:07.3795826Z * [new tag] ciflow/inductor/169515 -> ciflow/inductor/169515 2025-12-04T08:27:07.3795951Z * [new tag] ciflow/inductor/169517 -> ciflow/inductor/169517 2025-12-04T08:27:07.3796076Z * [new tag] ciflow/inductor/169519 -> ciflow/inductor/169519 2025-12-04T08:27:07.3796215Z * [new tag] ciflow/inductor/169520 -> ciflow/inductor/169520 2025-12-04T08:27:07.3796341Z * [new tag] ciflow/inductor/169521 -> ciflow/inductor/169521 2025-12-04T08:27:07.3796477Z * [new tag] ciflow/inductor/169524 -> ciflow/inductor/169524 2025-12-04T08:27:07.3796604Z * [new tag] ciflow/inductor/169527 -> ciflow/inductor/169527 2025-12-04T08:27:07.3796732Z * [new tag] ciflow/inductor/169528 -> ciflow/inductor/169528 2025-12-04T08:27:07.3796866Z * [new tag] ciflow/inductor/169532 -> ciflow/inductor/169532 2025-12-04T08:27:07.3796992Z * [new tag] ciflow/inductor/169535 -> ciflow/inductor/169535 2025-12-04T08:27:07.3797127Z * [new tag] ciflow/inductor/169536 -> ciflow/inductor/169536 2025-12-04T08:27:07.3797251Z * [new tag] ciflow/inductor/169547 -> ciflow/inductor/169547 2025-12-04T08:27:07.3797421Z * [new tag] ciflow/inductor/169548 -> ciflow/inductor/169548 2025-12-04T08:27:07.3797555Z * [new tag] ciflow/inductor/169549 -> ciflow/inductor/169549 2025-12-04T08:27:07.3797682Z * [new tag] ciflow/inductor/169551 -> ciflow/inductor/169551 2025-12-04T08:27:07.3797806Z * [new tag] ciflow/inductor/169552 -> ciflow/inductor/169552 2025-12-04T08:27:07.3797938Z * [new tag] ciflow/inductor/169553 -> ciflow/inductor/169553 2025-12-04T08:27:07.3798346Z * [new tag] ciflow/inductor/3b9a386 -> ciflow/inductor/3b9a386 2025-12-04T08:27:07.3798498Z * [new tag] ciflow/inductor/3d4b92b -> ciflow/inductor/3d4b92b 2025-12-04T08:27:07.3798630Z * [new tag] ciflow/inductor/d224ac7 -> ciflow/inductor/d224ac7 2025-12-04T08:27:07.3798786Z * [new tag] ciflow/linux-aarch64/157994 -> ciflow/linux-aarch64/157994 2025-12-04T08:27:07.3798941Z * [new tag] ciflow/linux-aarch64/166075 -> ciflow/linux-aarch64/166075 2025-12-04T08:27:07.3799078Z * [new tag] ciflow/linux-aarch64/166876 -> ciflow/linux-aarch64/166876 2025-12-04T08:27:07.3799222Z * [new tag] ciflow/linux-aarch64/167981 -> ciflow/linux-aarch64/167981 2025-12-04T08:27:07.3799343Z * [new tag] ciflow/mps/166254 -> ciflow/mps/166254 2025-12-04T08:27:07.3799461Z * [new tag] ciflow/mps/169017 -> ciflow/mps/169017 2025-12-04T08:27:07.3799581Z * [new tag] ciflow/mps/169372 -> ciflow/mps/169372 2025-12-04T08:27:07.3799695Z * [new tag] ciflow/mps/169478 -> ciflow/mps/169478 2025-12-04T08:27:07.3799840Z * [new tag] ciflow/op-benchmark/157994 -> ciflow/op-benchmark/157994 2025-12-04T08:27:07.3799989Z * [new tag] ciflow/op-benchmark/166075 -> ciflow/op-benchmark/166075 2025-12-04T08:27:07.3800130Z * [new tag] ciflow/op-benchmark/169544 -> ciflow/op-benchmark/169544 2025-12-04T08:27:07.3800337Z * [new tag] ciflow/periodic-rocm-mi200/165997 -> ciflow/periodic-rocm-mi200/165997 2025-12-04T08:27:07.3800552Z * [new tag] ciflow/periodic-rocm-mi200/166517 -> ciflow/periodic-rocm-mi200/166517 2025-12-04T08:27:07.3800722Z * [new tag] ciflow/periodic-rocm-mi200/169063 -> ciflow/periodic-rocm-mi200/169063 2025-12-04T08:27:07.3800896Z * [new tag] ciflow/periodic-rocm-mi200/169425 -> ciflow/periodic-rocm-mi200/169425 2025-12-04T08:27:07.3805374Z * [new tag] ciflow/periodic-rocm-mi300/166517 -> ciflow/periodic-rocm-mi300/166517 2025-12-04T08:27:07.3809525Z * [new tag] ciflow/periodic-rocm-mi300/169063 -> ciflow/periodic-rocm-mi300/169063 2025-12-04T08:27:07.3811562Z * [new tag] ciflow/periodic-rocm-mi300/169425 -> ciflow/periodic-rocm-mi300/169425 2025-12-04T08:27:07.3811740Z * [new tag] ciflow/periodic/054a2fd -> ciflow/periodic/054a2fd 2025-12-04T08:27:07.3812279Z * [new tag] ciflow/periodic/167207 -> ciflow/periodic/167207 2025-12-04T08:27:07.3812518Z * [new tag] ciflow/periodic/167978 -> ciflow/periodic/167978 2025-12-04T08:27:07.3812768Z * [new tag] ciflow/periodic/168096 -> ciflow/periodic/168096 2025-12-04T08:27:07.3812989Z * [new tag] ciflow/periodic/169286 -> ciflow/periodic/169286 2025-12-04T08:27:07.3813232Z * [new tag] ciflow/periodic/2a6d37d -> ciflow/periodic/2a6d37d 2025-12-04T08:27:07.3813442Z * [new tag] ciflow/periodic/317eeb8 -> ciflow/periodic/317eeb8 2025-12-04T08:27:07.3813654Z * [new tag] ciflow/periodic/3c32 -> ciflow/periodic/3c32 2025-12-04T08:27:07.3813889Z * [new tag] ciflow/periodic/3e98831 -> ciflow/periodic/3e98831 2025-12-04T08:27:07.3814552Z * [new tag] ciflow/periodic/7c648509a7470ace9fb2bae960dd4790f7e943e9 -> ciflow/periodic/7c648509a7470ace9fb2bae960dd4790f7e943e9 2025-12-04T08:27:07.3814816Z * [new tag] ciflow/periodic/94512-point -> ciflow/periodic/94512-point 2025-12-04T08:27:07.3815104Z * [new tag] ciflow/periodic/csl/test87519 -> ciflow/periodic/csl/test87519 2025-12-04T08:27:07.3815391Z * [new tag] ciflow/periodic/csltest88275 -> ciflow/periodic/csltest88275 2025-12-04T08:27:07.3815651Z * [new tag] ciflow/periodic/csltest88761 -> ciflow/periodic/csltest88761 2025-12-04T08:27:07.3815922Z * [new tag] ciflow/periodic/release_1.12 -> ciflow/periodic/release_1.12 2025-12-04T08:27:07.3816207Z * [new tag] ciflow/periodic/release_1.12.0 -> ciflow/periodic/release_1.12.0 2025-12-04T08:27:07.3816459Z * [new tag] ciflow/periodic/sha-ec5b83 -> ciflow/periodic/sha-ec5b83 2025-12-04T08:27:07.3816655Z * [new tag] ciflow/pull/167207 -> ciflow/pull/167207 2025-12-04T08:27:07.3817006Z * [new tag] ciflow/quantization-periodic/169207 -> ciflow/quantization-periodic/169207 2025-12-04T08:27:07.3817227Z * [new tag] ciflow/rocm-mi200/165545 -> ciflow/rocm-mi200/165545 2025-12-04T08:27:07.3817462Z * [new tag] ciflow/rocm-mi200/165997 -> ciflow/rocm-mi200/165997 2025-12-04T08:27:07.3817693Z * [new tag] ciflow/rocm-mi200/168096 -> ciflow/rocm-mi200/168096 2025-12-04T08:27:07.3821229Z * [new tag] ciflow/rocm-mi200/168275 -> ciflow/rocm-mi200/168275 2025-12-04T08:27:07.3821473Z * [new tag] ciflow/rocm-mi200/169063 -> ciflow/rocm-mi200/169063 2025-12-04T08:27:07.3821918Z * [new tag] ciflow/rocm-mi200/169356 -> ciflow/rocm-mi200/169356 2025-12-04T08:27:07.3822070Z * [new tag] ciflow/rocm-mi200/169425 -> ciflow/rocm-mi200/169425 2025-12-04T08:27:07.3822214Z * [new tag] ciflow/rocm-mi300/165545 -> ciflow/rocm-mi300/165545 2025-12-04T08:27:07.3825747Z * [new tag] ciflow/rocm-mi300/167157 -> ciflow/rocm-mi300/167157 2025-12-04T08:27:07.3825901Z * [new tag] ciflow/rocm-mi300/168096 -> ciflow/rocm-mi300/168096 2025-12-04T08:27:07.3828963Z * [new tag] ciflow/rocm-mi300/169063 -> ciflow/rocm-mi300/169063 2025-12-04T08:27:07.3829094Z * [new tag] ciflow/rocm-mi300/169425 -> ciflow/rocm-mi300/169425 2025-12-04T08:27:07.3829221Z * [new tag] ciflow/rocm-mi355/167157 -> ciflow/rocm-mi355/167157 2025-12-04T08:27:07.3829341Z * [new tag] ciflow/rocm-mi355/168275 -> ciflow/rocm-mi355/168275 2025-12-04T08:27:07.3829469Z * [new tag] ciflow/rocm-mi355/169425 -> ciflow/rocm-mi355/169425 2025-12-04T08:27:07.3829664Z * [new tag] ciflow/rocm-navi31/168275 -> ciflow/rocm-navi31/168275 2025-12-04T08:27:07.3829803Z * [new tag] ciflow/rocm-navi31/169425 -> ciflow/rocm-navi31/169425 2025-12-04T08:27:07.3829925Z * [new tag] ciflow/rocm/115316 -> ciflow/rocm/115316 2025-12-04T08:27:07.3830036Z * [new tag] ciflow/rocm/148492 -> ciflow/rocm/148492 2025-12-04T08:27:07.3830152Z * [new tag] ciflow/rocm/160685 -> ciflow/rocm/160685 2025-12-04T08:27:07.3830259Z * [new tag] ciflow/rocm/161607 -> ciflow/rocm/161607 2025-12-04T08:27:07.3830364Z * [new tag] ciflow/rocm/162052 -> ciflow/rocm/162052 2025-12-04T08:27:07.3830478Z * [new tag] ciflow/rocm/165997 -> ciflow/rocm/165997 2025-12-04T08:27:07.3830584Z * [new tag] ciflow/rocm/166165 -> ciflow/rocm/166165 2025-12-04T08:27:07.3830696Z * [new tag] ciflow/rocm/166517 -> ciflow/rocm/166517 2025-12-04T08:27:07.3830802Z * [new tag] ciflow/rocm/167207 -> ciflow/rocm/167207 2025-12-04T08:27:07.3830979Z * [new tag] ciflow/rocm/167536 -> ciflow/rocm/167536 2025-12-04T08:27:07.3831094Z * [new tag] ciflow/rocm/167781 -> ciflow/rocm/167781 2025-12-04T08:27:07.3831204Z * [new tag] ciflow/rocm/167989 -> ciflow/rocm/167989 2025-12-04T08:27:07.3835263Z * [new tag] ciflow/rocm/168073 -> ciflow/rocm/168073 2025-12-04T08:27:07.3835544Z * [new tag] ciflow/rocm/168195 -> ciflow/rocm/168195 2025-12-04T08:27:07.3835689Z * [new tag] ciflow/rocm/168939 -> ciflow/rocm/168939 2025-12-04T08:27:07.3835859Z * [new tag] ciflow/rocm/168971 -> ciflow/rocm/168971 2025-12-04T08:27:07.3836119Z * [new tag] ciflow/rocm/169024 -> ciflow/rocm/169024 2025-12-04T08:27:07.3836242Z * [new tag] ciflow/rocm/169200 -> ciflow/rocm/169200 2025-12-04T08:27:07.3836791Z * [new tag] ciflow/rocm/169216 -> ciflow/rocm/169216 2025-12-04T08:27:07.3836939Z * [new tag] ciflow/rocm/169312 -> ciflow/rocm/169312 2025-12-04T08:27:07.3837074Z * [new tag] ciflow/rocm/169380 -> ciflow/rocm/169380 2025-12-04T08:27:07.3837202Z * [new tag] ciflow/rocm/169427 -> ciflow/rocm/169427 2025-12-04T08:27:07.3837327Z * [new tag] ciflow/rocm/169455 -> ciflow/rocm/169455 2025-12-04T08:27:07.3837456Z * [new tag] ciflow/rocm/169470 -> ciflow/rocm/169470 2025-12-04T08:27:07.3837579Z * [new tag] ciflow/rocm/169471 -> ciflow/rocm/169471 2025-12-04T08:27:07.3837708Z * [new tag] ciflow/rocm/169472 -> ciflow/rocm/169472 2025-12-04T08:27:07.3837831Z * [new tag] ciflow/rocm/169514 -> ciflow/rocm/169514 2025-12-04T08:27:07.3837966Z * [new tag] ciflow/slow/01c7106 -> ciflow/slow/01c7106 2025-12-04T08:27:07.3838326Z * [new tag] ciflow/slow/0577043 -> ciflow/slow/0577043 2025-12-04T08:27:07.3838690Z * [new tag] ciflow/slow/0d5b74da0cab798fbfdb9caa53fad816999c8386-sdym -> ciflow/slow/0d5b74da0cab798fbfdb9caa53fad816999c8386-sdym 2025-12-04T08:27:07.3838980Z * [new tag] ciflow/slow/0e81104 -> ciflow/slow/0e81104 2025-12-04T08:27:07.3839103Z * [new tag] ciflow/slow/167207 -> ciflow/slow/167207 2025-12-04T08:27:07.3839829Z * [new tag] ciflow/slow/168050 -> ciflow/slow/168050 2025-12-04T08:27:07.3840537Z * [new tag] ciflow/slow/1732077 -> ciflow/slow/1732077 2025-12-04T08:27:07.3841382Z * [new tag] ciflow/slow/187eb7c -> ciflow/slow/187eb7c 2025-12-04T08:27:07.3842927Z * [new tag] ciflow/slow/1faef89 -> ciflow/slow/1faef89 2025-12-04T08:27:07.3843970Z * [new tag] ciflow/slow/3920ec1 -> ciflow/slow/3920ec1 2025-12-04T08:27:07.3844938Z * [new tag] ciflow/slow/3b7c6b2 -> ciflow/slow/3b7c6b2 2025-12-04T08:27:07.3845855Z * [new tag] ciflow/slow/59a3759 -> ciflow/slow/59a3759 2025-12-04T08:27:07.3846765Z * [new tag] ciflow/slow/70ef0bb -> ciflow/slow/70ef0bb 2025-12-04T08:27:07.3847640Z * [new tag] ciflow/slow/788ff06 -> ciflow/slow/788ff06 2025-12-04T08:27:07.3849050Z * [new tag] ciflow/slow/8751002215790a3a88750faa8f4366933e296693-sdym -> ciflow/slow/8751002215790a3a88750faa8f4366933e296693-sdym 2025-12-04T08:27:07.3849534Z * [new tag] ciflow/slow/9d85864 -> ciflow/slow/9d85864 2025-12-04T08:27:07.3850422Z * [new tag] ciflow/slow/9ffad5b -> ciflow/slow/9ffad5b 2025-12-04T08:27:07.3851246Z * [new tag] ciflow/slow/a206e8b -> ciflow/slow/a206e8b 2025-12-04T08:27:07.3852112Z * [new tag] ciflow/slow/a837609 -> ciflow/slow/a837609 2025-12-04T08:27:07.3852954Z * [new tag] ciflow/slow/af841f3 -> ciflow/slow/af841f3 2025-12-04T08:27:07.3854434Z * [new tag] ciflow/slow/da3aba1e46157c4df504b067477cdf2b3c96b194-sdym -> ciflow/slow/da3aba1e46157c4df504b067477cdf2b3c96b194-sdym 2025-12-04T08:27:07.3854902Z * [new tag] ciflow/torchbench/168175 -> ciflow/torchbench/168175 2025-12-04T08:27:07.3855628Z * [new tag] ciflow/trunk/148492 -> ciflow/trunk/148492 2025-12-04T08:27:07.3856256Z * [new tag] ciflow/trunk/157149 -> ciflow/trunk/157149 2025-12-04T08:27:07.3856879Z * [new tag] ciflow/trunk/157994 -> ciflow/trunk/157994 2025-12-04T08:27:07.3857491Z * [new tag] ciflow/trunk/159718 -> ciflow/trunk/159718 2025-12-04T08:27:07.3858210Z * [new tag] ciflow/trunk/160174 -> ciflow/trunk/160174 2025-12-04T08:27:07.3858789Z * [new tag] ciflow/trunk/160685 -> ciflow/trunk/160685 2025-12-04T08:27:07.3859978Z * [new tag] ciflow/trunk/160729 -> ciflow/trunk/160729 2025-12-04T08:27:07.3860191Z * [new tag] ciflow/trunk/162275 -> ciflow/trunk/162275 2025-12-04T08:27:07.3864480Z * [new tag] ciflow/trunk/162795 -> ciflow/trunk/162795 2025-12-04T08:27:07.3864778Z * [new tag] ciflow/trunk/163245 -> ciflow/trunk/163245 2025-12-04T08:27:07.3865117Z * [new tag] ciflow/trunk/163942 -> ciflow/trunk/163942 2025-12-04T08:27:07.3865287Z * [new tag] ciflow/trunk/165274 -> ciflow/trunk/165274 2025-12-04T08:27:07.3865497Z * [new tag] ciflow/trunk/165483 -> ciflow/trunk/165483 2025-12-04T08:27:07.3865778Z * [new tag] ciflow/trunk/165728 -> ciflow/trunk/165728 2025-12-04T08:27:07.3871852Z * [new tag] ciflow/trunk/165922 -> ciflow/trunk/165922 2025-12-04T08:27:07.3872002Z * [new tag] ciflow/trunk/166075 -> ciflow/trunk/166075 2025-12-04T08:27:07.3872112Z * [new tag] ciflow/trunk/166165 -> ciflow/trunk/166165 2025-12-04T08:27:07.3872375Z * [new tag] ciflow/trunk/166829 -> ciflow/trunk/166829 2025-12-04T08:27:07.3872484Z * [new tag] ciflow/trunk/166843 -> ciflow/trunk/166843 2025-12-04T08:27:07.3872598Z * [new tag] ciflow/trunk/166876 -> ciflow/trunk/166876 2025-12-04T08:27:07.3872704Z * [new tag] ciflow/trunk/167207 -> ciflow/trunk/167207 2025-12-04T08:27:07.3872808Z * [new tag] ciflow/trunk/167536 -> ciflow/trunk/167536 2025-12-04T08:27:07.3872924Z * [new tag] ciflow/trunk/167552 -> ciflow/trunk/167552 2025-12-04T08:27:07.3873034Z * [new tag] ciflow/trunk/167555 -> ciflow/trunk/167555 2025-12-04T08:27:07.3873154Z * [new tag] ciflow/trunk/167599 -> ciflow/trunk/167599 2025-12-04T08:27:07.3873267Z * [new tag] ciflow/trunk/167659 -> ciflow/trunk/167659 2025-12-04T08:27:07.3873384Z * [new tag] ciflow/trunk/167672 -> ciflow/trunk/167672 2025-12-04T08:27:07.3873499Z * [new tag] ciflow/trunk/167742 -> ciflow/trunk/167742 2025-12-04T08:27:07.3873608Z * [new tag] ciflow/trunk/167781 -> ciflow/trunk/167781 2025-12-04T08:27:07.3873722Z * [new tag] ciflow/trunk/167837 -> ciflow/trunk/167837 2025-12-04T08:27:07.3873838Z * [new tag] ciflow/trunk/167887 -> ciflow/trunk/167887 2025-12-04T08:27:07.3873946Z * [new tag] ciflow/trunk/167978 -> ciflow/trunk/167978 2025-12-04T08:27:07.3874070Z * [new tag] ciflow/trunk/168050 -> ciflow/trunk/168050 2025-12-04T08:27:07.3874553Z * [new tag] ciflow/trunk/168051 -> ciflow/trunk/168051 2025-12-04T08:27:07.3874772Z * [new tag] ciflow/trunk/168096 -> ciflow/trunk/168096 2025-12-04T08:27:07.3875436Z * [new tag] ciflow/trunk/168127 -> ciflow/trunk/168127 2025-12-04T08:27:07.3875758Z * [new tag] ciflow/trunk/168157 -> ciflow/trunk/168157 2025-12-04T08:27:07.3878625Z * [new tag] ciflow/trunk/168175 -> ciflow/trunk/168175 2025-12-04T08:27:07.3878776Z * [new tag] ciflow/trunk/168209 -> ciflow/trunk/168209 2025-12-04T08:27:07.3878910Z * [new tag] ciflow/trunk/168213 -> ciflow/trunk/168213 2025-12-04T08:27:07.3879053Z * [new tag] ciflow/trunk/168226 -> ciflow/trunk/168226 2025-12-04T08:27:07.3879180Z * [new tag] ciflow/trunk/168262 -> ciflow/trunk/168262 2025-12-04T08:27:07.3879319Z * [new tag] ciflow/trunk/168275 -> ciflow/trunk/168275 2025-12-04T08:27:07.3879628Z * [new tag] ciflow/trunk/168328 -> ciflow/trunk/168328 2025-12-04T08:27:07.3880392Z * [new tag] ciflow/trunk/168368 -> ciflow/trunk/168368 2025-12-04T08:27:07.3880534Z * [new tag] ciflow/trunk/168917 -> ciflow/trunk/168917 2025-12-04T08:27:07.3883630Z * [new tag] ciflow/trunk/168933 -> ciflow/trunk/168933 2025-12-04T08:27:07.3883955Z * [new tag] ciflow/trunk/168941 -> ciflow/trunk/168941 2025-12-04T08:27:07.3884131Z * [new tag] ciflow/trunk/168955 -> ciflow/trunk/168955 2025-12-04T08:27:07.3884318Z * [new tag] ciflow/trunk/168980 -> ciflow/trunk/168980 2025-12-04T08:27:07.3884459Z * [new tag] ciflow/trunk/169004 -> ciflow/trunk/169004 2025-12-04T08:27:07.3884572Z * [new tag] ciflow/trunk/169006 -> ciflow/trunk/169006 2025-12-04T08:27:07.3884839Z * [new tag] ciflow/trunk/169023 -> ciflow/trunk/169023 2025-12-04T08:27:07.3884960Z * [new tag] ciflow/trunk/169025 -> ciflow/trunk/169025 2025-12-04T08:27:07.3885217Z * [new tag] ciflow/trunk/169066 -> ciflow/trunk/169066 2025-12-04T08:27:07.3885339Z * [new tag] ciflow/trunk/169091 -> ciflow/trunk/169091 2025-12-04T08:27:07.3887376Z * [new tag] ciflow/trunk/169102 -> ciflow/trunk/169102 2025-12-04T08:27:07.3887665Z * [new tag] ciflow/trunk/169103 -> ciflow/trunk/169103 2025-12-04T08:27:07.3887803Z * [new tag] ciflow/trunk/169125 -> ciflow/trunk/169125 2025-12-04T08:27:07.3888165Z * [new tag] ciflow/trunk/169139 -> ciflow/trunk/169139 2025-12-04T08:27:07.3889372Z * [new tag] ciflow/trunk/169148 -> ciflow/trunk/169148 2025-12-04T08:27:07.3889684Z * [new tag] ciflow/trunk/169151 -> ciflow/trunk/169151 2025-12-04T08:27:07.3889821Z * [new tag] ciflow/trunk/169156 -> ciflow/trunk/169156 2025-12-04T08:27:07.3892770Z * [new tag] ciflow/trunk/169176 -> ciflow/trunk/169176 2025-12-04T08:27:07.3893058Z * [new tag] ciflow/trunk/169204 -> ciflow/trunk/169204 2025-12-04T08:27:07.3893192Z * [new tag] ciflow/trunk/169207 -> ciflow/trunk/169207 2025-12-04T08:27:07.3893466Z * [new tag] ciflow/trunk/169211 -> ciflow/trunk/169211 2025-12-04T08:27:07.3893708Z * [new tag] ciflow/trunk/169229 -> ciflow/trunk/169229 2025-12-04T08:27:07.3894186Z * [new tag] ciflow/trunk/169231 -> ciflow/trunk/169231 2025-12-04T08:27:07.3894326Z * [new tag] ciflow/trunk/169260 -> ciflow/trunk/169260 2025-12-04T08:27:07.3894439Z * [new tag] ciflow/trunk/169271 -> ciflow/trunk/169271 2025-12-04T08:27:07.3900940Z * [new tag] ciflow/trunk/169280 -> ciflow/trunk/169280 2025-12-04T08:27:07.3905487Z * [new tag] ciflow/trunk/169281 -> ciflow/trunk/169281 2025-12-04T08:27:07.3907898Z * [new tag] ciflow/trunk/169286 -> ciflow/trunk/169286 2025-12-04T08:27:07.3908228Z * [new tag] ciflow/trunk/169293 -> ciflow/trunk/169293 2025-12-04T08:27:07.3908349Z * [new tag] ciflow/trunk/169296 -> ciflow/trunk/169296 2025-12-04T08:27:07.3908569Z * [new tag] ciflow/trunk/169304 -> ciflow/trunk/169304 2025-12-04T08:27:07.3908834Z * [new tag] ciflow/trunk/169305 -> ciflow/trunk/169305 2025-12-04T08:27:07.3908964Z * [new tag] ciflow/trunk/169312 -> ciflow/trunk/169312 2025-12-04T08:27:07.3909121Z * [new tag] ciflow/trunk/169328 -> ciflow/trunk/169328 2025-12-04T08:27:07.3909367Z * [new tag] ciflow/trunk/169343 -> ciflow/trunk/169343 2025-12-04T08:27:07.3909493Z * [new tag] ciflow/trunk/169355 -> ciflow/trunk/169355 2025-12-04T08:27:07.3909640Z * [new tag] ciflow/trunk/169370 -> ciflow/trunk/169370 2025-12-04T08:27:07.3909826Z * [new tag] ciflow/trunk/169379 -> ciflow/trunk/169379 2025-12-04T08:27:07.3909956Z * [new tag] ciflow/trunk/169380 -> ciflow/trunk/169380 2025-12-04T08:27:07.3910065Z * [new tag] ciflow/trunk/169385 -> ciflow/trunk/169385 2025-12-04T08:27:07.3910173Z * [new tag] ciflow/trunk/169387 -> ciflow/trunk/169387 2025-12-04T08:27:07.3910464Z * [new tag] ciflow/trunk/169410 -> ciflow/trunk/169410 2025-12-04T08:27:07.3910627Z * [new tag] ciflow/trunk/169412 -> ciflow/trunk/169412 2025-12-04T08:27:07.3910749Z * [new tag] ciflow/trunk/169418 -> ciflow/trunk/169418 2025-12-04T08:27:07.3910913Z * [new tag] ciflow/trunk/169423 -> ciflow/trunk/169423 2025-12-04T08:27:07.3911032Z * [new tag] ciflow/trunk/169427 -> ciflow/trunk/169427 2025-12-04T08:27:07.3911264Z * [new tag] ciflow/trunk/169430 -> ciflow/trunk/169430 2025-12-04T08:27:07.3911375Z * [new tag] ciflow/trunk/169437 -> ciflow/trunk/169437 2025-12-04T08:27:07.3911492Z * [new tag] ciflow/trunk/169442 -> ciflow/trunk/169442 2025-12-04T08:27:07.3911601Z * [new tag] ciflow/trunk/169452 -> ciflow/trunk/169452 2025-12-04T08:27:07.3911711Z * [new tag] ciflow/trunk/169454 -> ciflow/trunk/169454 2025-12-04T08:27:07.3911833Z * [new tag] ciflow/trunk/169459 -> ciflow/trunk/169459 2025-12-04T08:27:07.3911947Z * [new tag] ciflow/trunk/169474 -> ciflow/trunk/169474 2025-12-04T08:27:07.3912072Z * [new tag] ciflow/trunk/169475 -> ciflow/trunk/169475 2025-12-04T08:27:07.3912189Z * [new tag] ciflow/trunk/169476 -> ciflow/trunk/169476 2025-12-04T08:27:07.3912308Z * [new tag] ciflow/trunk/169487 -> ciflow/trunk/169487 2025-12-04T08:27:07.3912429Z * [new tag] ciflow/trunk/169497 -> ciflow/trunk/169497 2025-12-04T08:27:07.3912544Z * [new tag] ciflow/trunk/169503 -> ciflow/trunk/169503 2025-12-04T08:27:07.3912672Z * [new tag] ciflow/trunk/169505 -> ciflow/trunk/169505 2025-12-04T08:27:07.3912787Z * [new tag] ciflow/trunk/169507 -> ciflow/trunk/169507 2025-12-04T08:27:07.3912902Z * [new tag] ciflow/trunk/169514 -> ciflow/trunk/169514 2025-12-04T08:27:07.3913027Z * [new tag] ciflow/trunk/169517 -> ciflow/trunk/169517 2025-12-04T08:27:07.3913191Z * [new tag] ciflow/trunk/169519 -> ciflow/trunk/169519 2025-12-04T08:27:07.3913527Z * [new tag] ciflow/trunk/169528 -> ciflow/trunk/169528 2025-12-04T08:27:07.3914969Z * [new tag] ciflow/trunk/169541 -> ciflow/trunk/169541 2025-12-04T08:27:07.3915153Z * [new tag] ciflow/unstable/123 -> ciflow/unstable/123 2025-12-04T08:27:07.3915564Z * [new tag] ciflow/vllm/165270 -> ciflow/vllm/165270 2025-12-04T08:27:07.3916020Z * [new tag] ciflow/vllm/165274 -> ciflow/vllm/165274 2025-12-04T08:27:07.3916476Z * [new tag] ciflow/vllm/166494 -> ciflow/vllm/166494 2025-12-04T08:27:07.3916924Z * [new tag] ciflow/vllm/169219 -> ciflow/vllm/169219 2025-12-04T08:27:07.3917756Z * [new tag] ciflow/vllm/169220 -> ciflow/vllm/169220 2025-12-04T08:27:07.3918287Z * [new tag] ciflow/xpu/157994 -> ciflow/xpu/157994 2025-12-04T08:27:07.3919281Z * [new tag] ciflow/xpu/159718 -> ciflow/xpu/159718 2025-12-04T08:27:07.3919494Z * [new tag] ciflow/xpu/161940 -> ciflow/xpu/161940 2025-12-04T08:27:07.3919734Z * [new tag] ciflow/xpu/163251 -> ciflow/xpu/163251 2025-12-04T08:27:07.3921965Z * [new tag] ciflow/xpu/166829 -> ciflow/xpu/166829 2025-12-04T08:27:07.3922187Z * [new tag] ciflow/xpu/166843 -> ciflow/xpu/166843 2025-12-04T08:27:07.3922420Z * [new tag] ciflow/xpu/167972 -> ciflow/xpu/167972 2025-12-04T08:27:07.3922604Z * [new tag] ciflow/xpu/167981 -> ciflow/xpu/167981 2025-12-04T08:27:07.3922743Z * [new tag] ciflow/xpu/168213 -> ciflow/xpu/168213 2025-12-04T08:27:07.3922882Z * [new tag] ciflow/xpu/168262 -> ciflow/xpu/168262 2025-12-04T08:27:07.3923312Z * [new tag] ciflow/xpu/168328 -> ciflow/xpu/168328 2025-12-04T08:27:07.3927092Z * [new tag] ciflow/xpu/168950 -> ciflow/xpu/168950 2025-12-04T08:27:07.3929352Z * [new tag] ciflow/xpu/169039 -> ciflow/xpu/169039 2025-12-04T08:27:07.3929692Z * [new tag] ciflow/xpu/169200 -> ciflow/xpu/169200 2025-12-04T08:27:07.3929808Z * [new tag] ciflow/xpu/169203 -> ciflow/xpu/169203 2025-12-04T08:27:07.3929920Z * [new tag] ciflow/xpu/169229 -> ciflow/xpu/169229 2025-12-04T08:27:07.3930043Z * [new tag] ciflow/xpu/169230 -> ciflow/xpu/169230 2025-12-04T08:27:07.3930155Z * [new tag] ciflow/xpu/169231 -> ciflow/xpu/169231 2025-12-04T08:27:07.3930268Z * [new tag] ciflow/xpu/169241 -> ciflow/xpu/169241 2025-12-04T08:27:07.3930387Z * [new tag] ciflow/xpu/169280 -> ciflow/xpu/169280 2025-12-04T08:27:07.3930608Z * [new tag] ciflow/xpu/169296 -> ciflow/xpu/169296 2025-12-04T08:27:07.3935427Z * [new tag] ciflow/xpu/169353 -> ciflow/xpu/169353 2025-12-04T08:27:07.3937455Z * [new tag] ciflow/xpu/169410 -> ciflow/xpu/169410 2025-12-04T08:27:07.3937693Z * [new tag] ciflow/xpu/169442 -> ciflow/xpu/169442 2025-12-04T08:27:07.3943478Z * [new tag] cslpull75 -> cslpull75 2025-12-04T08:27:07.3943626Z * [new tag] cslpull76 -> cslpull76 2025-12-04T08:27:07.3943726Z * [new tag] cslpull77 -> cslpull77 2025-12-04T08:27:07.3943834Z * [new tag] cslpull78 -> cslpull78 2025-12-04T08:27:07.3943930Z * [new tag] cslpull79 -> cslpull79 2025-12-04T08:27:07.3944033Z * [new tag] cslpull80 -> cslpull80 2025-12-04T08:27:07.3944324Z * [new tag] cslpull81 -> cslpull81 2025-12-04T08:27:07.3944433Z * [new tag] cslpull82 -> cslpull82 2025-12-04T08:27:07.3944543Z * [new tag] cslpull83 -> cslpull83 2025-12-04T08:27:07.3944639Z * [new tag] cslpull84 -> cslpull84 2025-12-04T08:27:07.3944734Z * [new tag] cslpull85 -> cslpull85 2025-12-04T08:27:07.3944837Z * [new tag] cslpull86 -> cslpull86 2025-12-04T08:27:07.3944932Z * [new tag] cslpull87 -> cslpull87 2025-12-04T08:27:07.3945036Z * [new tag] cslpull88 -> cslpull88 2025-12-04T08:27:07.3945131Z * [new tag] cslpull89 -> cslpull89 2025-12-04T08:27:07.3945228Z * [new tag] cslpull90 -> cslpull90 2025-12-04T08:27:07.3945334Z * [new tag] cslpull91 -> cslpull91 2025-12-04T08:27:07.3945430Z * [new tag] cslpull92 -> cslpull92 2025-12-04T08:27:07.3945536Z * [new tag] flight_5 -> flight_5 2025-12-04T08:27:07.3945651Z * [new tag] flight_5.1 -> flight_5.1 2025-12-04T08:27:07.3945919Z * [new tag] flight_5.2 -> flight_5.2 2025-12-04T08:27:07.3946182Z * [new tag] flight_5.3 -> flight_5.3 2025-12-04T08:27:07.3946310Z * [new tag] forpull1 -> forpull1 2025-12-04T08:27:07.3946436Z * [new tag] malfet/tag-2ef5611 -> malfet/tag-2ef5611 2025-12-04T08:27:07.3946563Z * [new tag] malfet/tag-317b1a0 -> malfet/tag-317b1a0 2025-12-04T08:27:07.3946805Z * [new tag] malfet/tag-ec6f767 -> malfet/tag-ec6f767 2025-12-04T08:27:07.3952070Z * [new tag] nightly-binary -> nightly-binary 2025-12-04T08:27:07.3952383Z * [new tag] sqzhang_flight4_plus -> sqzhang_flight4_plus 2025-12-04T08:27:07.3952525Z * [new tag] sqzhang_flight_3 -> sqzhang_flight_3 2025-12-04T08:27:07.3952960Z * [new tag] trunk/02d8bd6974cf84b721680d773dbdb1b6f40ce272 -> trunk/02d8bd6974cf84b721680d773dbdb1b6f40ce272 2025-12-04T08:27:07.3953251Z * [new tag] trunk/066997fb38ade71e00d78e9d572e380b5f02bd3e -> trunk/066997fb38ade71e00d78e9d572e380b5f02bd3e 2025-12-04T08:27:07.3953635Z * [new tag] trunk/076e7b19fa1d481ad778d06d2b49ba57d3ce8c88 -> trunk/076e7b19fa1d481ad778d06d2b49ba57d3ce8c88 2025-12-04T08:27:07.3954633Z * [new tag] trunk/07dcc0b83db3211653a38565a24e15acdba75654 -> trunk/07dcc0b83db3211653a38565a24e15acdba75654 2025-12-04T08:27:07.3955326Z * [new tag] trunk/082e96b68dfcd16cab7cfafc4d3d055767dab3eb -> trunk/082e96b68dfcd16cab7cfafc4d3d055767dab3eb 2025-12-04T08:27:07.3955606Z * [new tag] trunk/088048f2fea28ff7d450f65c72419ca45780d30b -> trunk/088048f2fea28ff7d450f65c72419ca45780d30b 2025-12-04T08:27:07.3955853Z * [new tag] trunk/09076941a95c76f4d9ad189d064dfd8baa39e672 -> trunk/09076941a95c76f4d9ad189d064dfd8baa39e672 2025-12-04T08:27:07.3956132Z * [new tag] trunk/0b80a4c62b94402844bf221791c096b0035c6d75 -> trunk/0b80a4c62b94402844bf221791c096b0035c6d75 2025-12-04T08:27:07.3956370Z * [new tag] trunk/0bbbdf1750567a980634ad907a325357ba8ba8f2 -> trunk/0bbbdf1750567a980634ad907a325357ba8ba8f2 2025-12-04T08:27:07.3956623Z * [new tag] trunk/0c281dd78773b2bc17c58ead0e4cd4ac46e775c5 -> trunk/0c281dd78773b2bc17c58ead0e4cd4ac46e775c5 2025-12-04T08:27:07.3956852Z * [new tag] trunk/135f3753c418a6879b1954904184937b67e61688 -> trunk/135f3753c418a6879b1954904184937b67e61688 2025-12-04T08:27:07.3957274Z * [new tag] trunk/15da21026cb13cd20257dc9e96830db108743c10 -> trunk/15da21026cb13cd20257dc9e96830db108743c10 2025-12-04T08:27:07.3957543Z * [new tag] trunk/166efdad2ac827f30fb02504c6017520257f88ec -> trunk/166efdad2ac827f30fb02504c6017520257f88ec 2025-12-04T08:27:07.3957798Z * [new tag] trunk/174272c15fae553d8488140af931f7d8050a313f -> trunk/174272c15fae553d8488140af931f7d8050a313f 2025-12-04T08:27:07.3958175Z * [new tag] trunk/18f3ca08f13b8de61307f5e8cd7d4cccb67e9d11 -> trunk/18f3ca08f13b8de61307f5e8cd7d4cccb67e9d11 2025-12-04T08:27:07.3958507Z * [new tag] trunk/1902eddfe655a15ebcf2c72bd81ade110fdeef63 -> trunk/1902eddfe655a15ebcf2c72bd81ade110fdeef63 2025-12-04T08:27:07.3959113Z * [new tag] trunk/195f92e98d3d66738577f11f22c4b5c8a1c76dd5 -> trunk/195f92e98d3d66738577f11f22c4b5c8a1c76dd5 2025-12-04T08:27:07.3962064Z * [new tag] trunk/1aa13e17de39e3c768ea7aebaad166ce72a06676 -> trunk/1aa13e17de39e3c768ea7aebaad166ce72a06676 2025-12-04T08:27:07.3962513Z * [new tag] trunk/1afe2832f58e24e54a5bfda5a5afa9b96fdea40e -> trunk/1afe2832f58e24e54a5bfda5a5afa9b96fdea40e 2025-12-04T08:27:07.3962874Z * [new tag] trunk/1c87554d74140eaee964ca8b1832cede67f5f520 -> trunk/1c87554d74140eaee964ca8b1832cede67f5f520 2025-12-04T08:27:07.3963222Z * [new tag] trunk/1ccb743b7b5be955f49736c162c4f5004b8a0dd8 -> trunk/1ccb743b7b5be955f49736c162c4f5004b8a0dd8 2025-12-04T08:27:07.3963488Z * [new tag] trunk/1cee47d6ce0a02227185b566593f002dd639ca0c -> trunk/1cee47d6ce0a02227185b566593f002dd639ca0c 2025-12-04T08:27:07.3963758Z * [new tag] trunk/1d21b4df2babe322e5d085ceb6de884eb260a62d -> trunk/1d21b4df2babe322e5d085ceb6de884eb260a62d 2025-12-04T08:27:07.3964020Z * [new tag] trunk/1e34fb2550e4aa650314f7a6d9f6daf4da7478a8 -> trunk/1e34fb2550e4aa650314f7a6d9f6daf4da7478a8 2025-12-04T08:27:07.3964537Z * [new tag] trunk/1e526fb5b1d93bfc70691c5c3955fdffc1b7b7de -> trunk/1e526fb5b1d93bfc70691c5c3955fdffc1b7b7de 2025-12-04T08:27:07.3965763Z * [new tag] trunk/1ee32a8b1f554a312d79bad01ded24f38cd95543 -> trunk/1ee32a8b1f554a312d79bad01ded24f38cd95543 2025-12-04T08:27:07.3966200Z * [new tag] trunk/201e2c4117eb9744594dad6a5c18213d7b4705d7 -> trunk/201e2c4117eb9744594dad6a5c18213d7b4705d7 2025-12-04T08:27:07.3966581Z * [new tag] trunk/2353a0f60eb4b4cb6675907a7fa9fbedc1c02e7f -> trunk/2353a0f60eb4b4cb6675907a7fa9fbedc1c02e7f 2025-12-04T08:27:07.3968341Z * [new tag] trunk/285779b1621cf9f073a062b0889a642d200308d9 -> trunk/285779b1621cf9f073a062b0889a642d200308d9 2025-12-04T08:27:07.3968825Z * [new tag] trunk/2887faaec6295d081580d09fce161201826c6d87 -> trunk/2887faaec6295d081580d09fce161201826c6d87 2025-12-04T08:27:07.3969205Z * [new tag] trunk/296e67c92635443c67b11c0ae1bd045f03ebb7bc -> trunk/296e67c92635443c67b11c0ae1bd045f03ebb7bc 2025-12-04T08:27:07.3969491Z * [new tag] trunk/29856679769b3dede478767e2fe6cfb51197cb25 -> trunk/29856679769b3dede478767e2fe6cfb51197cb25 2025-12-04T08:27:07.3970026Z * [new tag] trunk/29e5455a4740c326ab187c7aa7b5ef98034ea563 -> trunk/29e5455a4740c326ab187c7aa7b5ef98034ea563 2025-12-04T08:27:07.3971203Z * [new tag] trunk/2ac3ef882afb23136adc188975f0a8802fc68adf -> trunk/2ac3ef882afb23136adc188975f0a8802fc68adf 2025-12-04T08:27:07.3971833Z * [new tag] trunk/2bec68e73b64715354af076ad309335f943e36cd -> trunk/2bec68e73b64715354af076ad309335f943e36cd 2025-12-04T08:27:07.3972404Z * [new tag] trunk/2c87367e6f88662cd5cedbd1537748b7948c38e1 -> trunk/2c87367e6f88662cd5cedbd1537748b7948c38e1 2025-12-04T08:27:07.3975078Z * [new tag] trunk/2d1f78fe3ec13820f136a2e0336da12a25f41708 -> trunk/2d1f78fe3ec13820f136a2e0336da12a25f41708 2025-12-04T08:27:07.3975675Z * [new tag] trunk/2df6058f116a65722a0e03073402feb242572d35 -> trunk/2df6058f116a65722a0e03073402feb242572d35 2025-12-04T08:27:07.3976074Z * [new tag] trunk/2e0c2e170fe658c440775c8e5c44228aafcc47ec -> trunk/2e0c2e170fe658c440775c8e5c44228aafcc47ec 2025-12-04T08:27:07.3976461Z * [new tag] trunk/2f9b7dad7b5419b063bd0f2e204de192720ebb94 -> trunk/2f9b7dad7b5419b063bd0f2e204de192720ebb94 2025-12-04T08:27:07.3977110Z * [new tag] trunk/305168768a95d69c444df5cd334bb774edfe06f1 -> trunk/305168768a95d69c444df5cd334bb774edfe06f1 2025-12-04T08:27:07.3977420Z * [new tag] trunk/31fc12773026e8e00f054dd79ad9b2491e693b48 -> trunk/31fc12773026e8e00f054dd79ad9b2491e693b48 2025-12-04T08:27:07.3977734Z * [new tag] trunk/320de0c6b0a3e7c6d2693ea5c28d5d0156ba7991 -> trunk/320de0c6b0a3e7c6d2693ea5c28d5d0156ba7991 2025-12-04T08:27:07.3978173Z * [new tag] trunk/3418bd29475dff06695045fcdf93e7d0dac67da8 -> trunk/3418bd29475dff06695045fcdf93e7d0dac67da8 2025-12-04T08:27:07.3978468Z * [new tag] trunk/34a98608afa0cb5b48f0d6d30432fdd0a2614ddf -> trunk/34a98608afa0cb5b48f0d6d30432fdd0a2614ddf 2025-12-04T08:27:07.3979881Z * [new tag] trunk/35b7a9a26c5923d98aebaa41a031dae21788a9ee -> trunk/35b7a9a26c5923d98aebaa41a031dae21788a9ee 2025-12-04T08:27:07.3980349Z * [new tag] trunk/39d07dbf03a911bdd45d1af78d8638dc92074938 -> trunk/39d07dbf03a911bdd45d1af78d8638dc92074938 2025-12-04T08:27:07.3980628Z * [new tag] trunk/3cd98b4205ada151042cc7ff097a82d4a4b18725 -> trunk/3cd98b4205ada151042cc7ff097a82d4a4b18725 2025-12-04T08:27:07.3981203Z * [new tag] trunk/3d35fd20a78ff4d016fa80f4e5fad37191d7bcae -> trunk/3d35fd20a78ff4d016fa80f4e5fad37191d7bcae 2025-12-04T08:27:07.3982751Z * [new tag] trunk/409a5fee945c46a3edaf5df162812f201bfd7b2f -> trunk/409a5fee945c46a3edaf5df162812f201bfd7b2f 2025-12-04T08:27:07.3983170Z * [new tag] trunk/42e9005cda22da3f1c559c3649218cebd671027c -> trunk/42e9005cda22da3f1c559c3649218cebd671027c 2025-12-04T08:27:07.3983444Z * [new tag] trunk/43b94713bbf340d3c124fde02d0f73add4021247 -> trunk/43b94713bbf340d3c124fde02d0f73add4021247 2025-12-04T08:27:07.3983805Z * [new tag] trunk/44ac69388a4a5eb463dbd2a13f00d1e3b924566c -> trunk/44ac69388a4a5eb463dbd2a13f00d1e3b924566c 2025-12-04T08:27:07.3984268Z * [new tag] trunk/45d14e2497292be06ad36eaa1aaaf7c630a2586a -> trunk/45d14e2497292be06ad36eaa1aaaf7c630a2586a 2025-12-04T08:27:07.3985739Z * [new tag] trunk/45d310ad84854dff730c0b12e577d7998d978686 -> trunk/45d310ad84854dff730c0b12e577d7998d978686 2025-12-04T08:27:07.3986157Z * [new tag] trunk/47b28ddf7bd74b50fa93b307a7d3b183a6d77f54 -> trunk/47b28ddf7bd74b50fa93b307a7d3b183a6d77f54 2025-12-04T08:27:07.3986431Z * [new tag] trunk/481e5ab336275bd3acd5fa8a611b05b4469012af -> trunk/481e5ab336275bd3acd5fa8a611b05b4469012af 2025-12-04T08:27:07.3986948Z * [new tag] trunk/491731647f6b8a9345dcfb3bc9416aea254a7d96 -> trunk/491731647f6b8a9345dcfb3bc9416aea254a7d96 2025-12-04T08:27:07.3988789Z * [new tag] trunk/49a04d26088acc17d948ddd66920f3e16371e873 -> trunk/49a04d26088acc17d948ddd66920f3e16371e873 2025-12-04T08:27:07.3989231Z * [new tag] trunk/4bebc827c47d2f1f0fa1a417a5201a97aef3d985 -> trunk/4bebc827c47d2f1f0fa1a417a5201a97aef3d985 2025-12-04T08:27:07.3989605Z * [new tag] trunk/4c246677784c6a14bc2dbb9ff8773ef0a3a3222f -> trunk/4c246677784c6a14bc2dbb9ff8773ef0a3a3222f 2025-12-04T08:27:07.3989847Z * [new tag] trunk/4cfb47ff548b6d996641058cf04a70e311a4c3aa -> trunk/4cfb47ff548b6d996641058cf04a70e311a4c3aa 2025-12-04T08:27:07.3990123Z * [new tag] trunk/4e0061c1aa52f606dda8cfab0bd7591e588faf2c -> trunk/4e0061c1aa52f606dda8cfab0bd7591e588faf2c 2025-12-04T08:27:07.3991810Z * [new tag] trunk/4fefb8e7e942386ffac764a41b232241f82bea3a -> trunk/4fefb8e7e942386ffac764a41b232241f82bea3a 2025-12-04T08:27:07.3992369Z * [new tag] trunk/503b2640023521f5a35cd9a52fc8033d73a95d0d -> trunk/503b2640023521f5a35cd9a52fc8033d73a95d0d 2025-12-04T08:27:07.3992649Z * [new tag] trunk/518c2b1b3dab9a2ef2849e04b3bc2f20c1c41db9 -> trunk/518c2b1b3dab9a2ef2849e04b3bc2f20c1c41db9 2025-12-04T08:27:07.3994080Z * [new tag] trunk/5191b2fa68ba19960912bfd7fd721c79d76bb1f3 -> trunk/5191b2fa68ba19960912bfd7fd721c79d76bb1f3 2025-12-04T08:27:07.3994367Z * [new tag] trunk/52ac0f0dc4acacd219f1317fbc28ec631c01e07a -> trunk/52ac0f0dc4acacd219f1317fbc28ec631c01e07a 2025-12-04T08:27:07.3994715Z * [new tag] trunk/539ba711b029de9f191070f4f0d12f18f5b7f292 -> trunk/539ba711b029de9f191070f4f0d12f18f5b7f292 2025-12-04T08:27:07.3995270Z * [new tag] trunk/556375b55deebebbc56cb7aef81f4d52f031ba28 -> trunk/556375b55deebebbc56cb7aef81f4d52f031ba28 2025-12-04T08:27:07.3996185Z * [new tag] trunk/55c4ab554845481d0a69a3811937575fe8bb1a66 -> trunk/55c4ab554845481d0a69a3811937575fe8bb1a66 2025-12-04T08:27:07.3996878Z * [new tag] trunk/5634469fda9e5d98869c82c7d03bb08914245f96 -> trunk/5634469fda9e5d98869c82c7d03bb08914245f96 2025-12-04T08:27:07.3997475Z * [new tag] trunk/5778f6ff894686a975a9a23645178ae4c87ad5dc -> trunk/5778f6ff894686a975a9a23645178ae4c87ad5dc 2025-12-04T08:27:07.3998261Z * [new tag] trunk/587d63a3e07de5dc91065f9ef70bcacda9989068 -> trunk/587d63a3e07de5dc91065f9ef70bcacda9989068 2025-12-04T08:27:07.3998836Z * [new tag] trunk/597930f6b568852356ca9795dac76f9e4653adbd -> trunk/597930f6b568852356ca9795dac76f9e4653adbd 2025-12-04T08:27:07.3999346Z * [new tag] trunk/597df3a4e2a67b9fdbe1a89b2f4d74f822274db6 -> trunk/597df3a4e2a67b9fdbe1a89b2f4d74f822274db6 2025-12-04T08:27:07.4003725Z * [new tag] trunk/59abd50e931f4efb21b053f7a2911f5d8a49d883 -> trunk/59abd50e931f4efb21b053f7a2911f5d8a49d883 2025-12-04T08:27:07.4004172Z * [new tag] trunk/5a607febc04c3a2b5824c75f3f60307867439a2c -> trunk/5a607febc04c3a2b5824c75f3f60307867439a2c 2025-12-04T08:27:07.4004525Z * [new tag] trunk/5bf1cdf4755c54ef462b44cb8041b0a57311556b -> trunk/5bf1cdf4755c54ef462b44cb8041b0a57311556b 2025-12-04T08:27:07.4005142Z * [new tag] trunk/5f0030ba63d334d7e8c93a09e41403b89e4c573c -> trunk/5f0030ba63d334d7e8c93a09e41403b89e4c573c 2025-12-04T08:27:07.4005399Z * [new tag] trunk/5f21d27e71268464d362a96c9ac09ea475f7f202 -> trunk/5f21d27e71268464d362a96c9ac09ea475f7f202 2025-12-04T08:27:07.4005946Z * [new tag] trunk/5fafc13038c9988d9ac21fa793fbd5890604b447 -> trunk/5fafc13038c9988d9ac21fa793fbd5890604b447 2025-12-04T08:27:07.4006230Z * [new tag] trunk/61be54a31dc09b59d99b62176fb935aee0b924ef -> trunk/61be54a31dc09b59d99b62176fb935aee0b924ef 2025-12-04T08:27:07.4006474Z * [new tag] trunk/62d3ccd71484ed6a760d909b41487101bbc65719 -> trunk/62d3ccd71484ed6a760d909b41487101bbc65719 2025-12-04T08:27:07.4006743Z * [new tag] trunk/641cdb68ae27668eb441d0e49c87a0602c120c2b -> trunk/641cdb68ae27668eb441d0e49c87a0602c120c2b 2025-12-04T08:27:07.4006981Z * [new tag] trunk/65c4620d6bb0c6029f69762c22b91dda2294da9a -> trunk/65c4620d6bb0c6029f69762c22b91dda2294da9a 2025-12-04T08:27:07.4007231Z * [new tag] trunk/66004b993744b4106bf8afaba71f3c228a804206 -> trunk/66004b993744b4106bf8afaba71f3c228a804206 2025-12-04T08:27:07.4007482Z * [new tag] trunk/6658a04c7ca67acb64512341342e7b3ee13ee386 -> trunk/6658a04c7ca67acb64512341342e7b3ee13ee386 2025-12-04T08:27:07.4014274Z * [new tag] trunk/6864e309092a71f8ab0ca6a4dc7f8a4073fd31c4 -> trunk/6864e309092a71f8ab0ca6a4dc7f8a4073fd31c4 2025-12-04T08:27:07.4019070Z * [new tag] trunk/6c261c6cb07892c90ca19ed51c9705b1659a3f7d -> trunk/6c261c6cb07892c90ca19ed51c9705b1659a3f7d 2025-12-04T08:27:07.4022761Z * [new tag] trunk/6c8b6a043f1628188b6396b3a2a6e000ca68362b -> trunk/6c8b6a043f1628188b6396b3a2a6e000ca68362b 2025-12-04T08:27:07.4027336Z * [new tag] trunk/6ceb4a32f92ae67ce5d7d97931d17401ebf5ffa5 -> trunk/6ceb4a32f92ae67ce5d7d97931d17401ebf5ffa5 2025-12-04T08:27:07.4030852Z * [new tag] trunk/6e404e9b7d6f5fb0de86aa73888c3038248c17f8 -> trunk/6e404e9b7d6f5fb0de86aa73888c3038248c17f8 2025-12-04T08:27:07.4034306Z * [new tag] trunk/6ec30b490aee1db6bcdc7340abddef25784f08ec -> trunk/6ec30b490aee1db6bcdc7340abddef25784f08ec 2025-12-04T08:27:07.4034585Z * [new tag] trunk/6f2783a6c08e1db34275ff25176ffe9aebc30a71 -> trunk/6f2783a6c08e1db34275ff25176ffe9aebc30a71 2025-12-04T08:27:07.4035020Z * [new tag] trunk/6f53fefeb90ad3281119b5cfc4aa9ffd8a066e3d -> trunk/6f53fefeb90ad3281119b5cfc4aa9ffd8a066e3d 2025-12-04T08:27:07.4035325Z * [new tag] trunk/6f7dcf51e46d0c880db1a2f5c70de57adb576f4a -> trunk/6f7dcf51e46d0c880db1a2f5c70de57adb576f4a 2025-12-04T08:27:07.4035678Z * [new tag] trunk/6ff831180d2fa436c7f1c1af3adac641fce9d60e -> trunk/6ff831180d2fa436c7f1c1af3adac641fce9d60e 2025-12-04T08:27:07.4036433Z * [new tag] trunk/70076464a63ab218a7ceefb0e76ccd7131deb8f8 -> trunk/70076464a63ab218a7ceefb0e76ccd7131deb8f8 2025-12-04T08:27:07.4036742Z * [new tag] trunk/70d797a5fc109b20a517646fcaa819477cd0d485 -> trunk/70d797a5fc109b20a517646fcaa819477cd0d485 2025-12-04T08:27:07.4037045Z * [new tag] trunk/7348cb355ff0a6f79cd4871215aea72185748734 -> trunk/7348cb355ff0a6f79cd4871215aea72185748734 2025-12-04T08:27:07.4037296Z * [new tag] trunk/74fe26a1ebe32931783569f2e762e3c2c974901f -> trunk/74fe26a1ebe32931783569f2e762e3c2c974901f 2025-12-04T08:27:07.4037565Z * [new tag] trunk/76aeb8c7e0f795b3fddca134cbea9a69da3ee696 -> trunk/76aeb8c7e0f795b3fddca134cbea9a69da3ee696 2025-12-04T08:27:07.4037811Z * [new tag] trunk/7741edd4ed665f3988052e260863efb508d61a03 -> trunk/7741edd4ed665f3988052e260863efb508d61a03 2025-12-04T08:27:07.4038137Z * [new tag] trunk/78adb3b3df41b45d2368b67226d2f864b78939a6 -> trunk/78adb3b3df41b45d2368b67226d2f864b78939a6 2025-12-04T08:27:07.4038411Z * [new tag] trunk/79d7b178225e5ed24d4e1db74e5abbff848f5fb7 -> trunk/79d7b178225e5ed24d4e1db74e5abbff848f5fb7 2025-12-04T08:27:07.4038859Z * [new tag] trunk/7a1e316115fc6996b3f2336822ba5d5f6179f0c3 -> trunk/7a1e316115fc6996b3f2336822ba5d5f6179f0c3 2025-12-04T08:27:07.4039133Z * [new tag] trunk/7a41b66367c38d0af3e8a90f7be48d6b281e7bca -> trunk/7a41b66367c38d0af3e8a90f7be48d6b281e7bca 2025-12-04T08:27:07.4039443Z * [new tag] trunk/7b7af390ea8541c611d1ce2018a6934188fc197b -> trunk/7b7af390ea8541c611d1ce2018a6934188fc197b 2025-12-04T08:27:07.4039692Z * [new tag] trunk/7ba4680f3755a560af81aa0f688791e367aa3609 -> trunk/7ba4680f3755a560af81aa0f688791e367aa3609 2025-12-04T08:27:07.4039956Z * [new tag] trunk/7bc2a66ded06a0b2549aa51d807edc5dc3e73d1b -> trunk/7bc2a66ded06a0b2549aa51d807edc5dc3e73d1b 2025-12-04T08:27:07.4040218Z * [new tag] trunk/7c648509a7470ace9fb2bae960dd4790f7e943e9 -> trunk/7c648509a7470ace9fb2bae960dd4790f7e943e9 2025-12-04T08:27:07.4040485Z * [new tag] trunk/7cbc2d034cecd21ab5c9707d0a9c525c17143fb8 -> trunk/7cbc2d034cecd21ab5c9707d0a9c525c17143fb8 2025-12-04T08:27:07.4040746Z * [new tag] trunk/7d1bbaf4ba301ea3fba6f3c7bc02d58f6417aaed -> trunk/7d1bbaf4ba301ea3fba6f3c7bc02d58f6417aaed 2025-12-04T08:27:07.4041001Z * [new tag] trunk/7d2a33e4ebf60b217a3cd77feae19231eb996fc8 -> trunk/7d2a33e4ebf60b217a3cd77feae19231eb996fc8 2025-12-04T08:27:07.4041244Z * [new tag] trunk/7eb625920054b1126a7d2d99818aaa188c6ba95e -> trunk/7eb625920054b1126a7d2d99818aaa188c6ba95e 2025-12-04T08:27:07.4041514Z * [new tag] trunk/7f55ba19c456a3d6cc443dd9edb6bb7cca677ead -> trunk/7f55ba19c456a3d6cc443dd9edb6bb7cca677ead 2025-12-04T08:27:07.4041816Z * [new tag] trunk/81af382128efa094d8702e18f2c133760904c718 -> trunk/81af382128efa094d8702e18f2c133760904c718 2025-12-04T08:27:07.4042075Z * [new tag] trunk/84149583d483e9c973c9a0feda70e4f3964947b0 -> trunk/84149583d483e9c973c9a0feda70e4f3964947b0 2025-12-04T08:27:07.4042342Z * [new tag] trunk/85a315917efe82c24306be805c584ec044951c75 -> trunk/85a315917efe82c24306be805c584ec044951c75 2025-12-04T08:27:07.4042602Z * [new tag] trunk/87329491c82a5f8c1cc4ec11d8f55a5de2551ece -> trunk/87329491c82a5f8c1cc4ec11d8f55a5de2551ece 2025-12-04T08:27:07.4042846Z * [new tag] trunk/892640e25aeefa8007c5af837214b4502b6b62a6 -> trunk/892640e25aeefa8007c5af837214b4502b6b62a6 2025-12-04T08:27:07.4043101Z * [new tag] trunk/89e3bbcb5b5321dc8b9520b4d5a8ee60cea1d0b4 -> trunk/89e3bbcb5b5321dc8b9520b4d5a8ee60cea1d0b4 2025-12-04T08:27:07.4043351Z * [new tag] trunk/8c73bbbb02159223c0c97d268a0a74cb78158a1c -> trunk/8c73bbbb02159223c0c97d268a0a74cb78158a1c 2025-12-04T08:27:07.4043612Z * [new tag] trunk/8d56e98c8db988a22cb2dfaeefb30bc7d2a3cc43 -> trunk/8d56e98c8db988a22cb2dfaeefb30bc7d2a3cc43 2025-12-04T08:27:07.4043861Z * [new tag] trunk/8d9dd9603e5ee26c01007f0cd4f018e584840922 -> trunk/8d9dd9603e5ee26c01007f0cd4f018e584840922 2025-12-04T08:27:07.4044112Z * [new tag] trunk/8ef0c0b02b062d75e7c9be2594914a3e784d23ca -> trunk/8ef0c0b02b062d75e7c9be2594914a3e784d23ca 2025-12-04T08:27:07.4044358Z * [new tag] trunk/90b27e7e8352cde97d32ddad24740ef819633f38 -> trunk/90b27e7e8352cde97d32ddad24740ef819633f38 2025-12-04T08:27:07.4044607Z * [new tag] trunk/90f0139e64b2951815d524b6a373bed20c4fbf90 -> trunk/90f0139e64b2951815d524b6a373bed20c4fbf90 2025-12-04T08:27:07.4044857Z * [new tag] trunk/93d0d6838c56af59b0dba794e6aa08f0c1c7799c -> trunk/93d0d6838c56af59b0dba794e6aa08f0c1c7799c 2025-12-04T08:27:07.4045111Z * [new tag] trunk/94ca8d5f1e81fea3ae488650a0fb6795049a9f87 -> trunk/94ca8d5f1e81fea3ae488650a0fb6795049a9f87 2025-12-04T08:27:07.4045361Z * [new tag] trunk/9844fbeadd5cebdf1281d6fbf79164139c352693 -> trunk/9844fbeadd5cebdf1281d6fbf79164139c352693 2025-12-04T08:27:07.4045637Z * [new tag] trunk/99024dec888ec1e50b546822a32b6fb2f35e5eaa -> trunk/99024dec888ec1e50b546822a32b6fb2f35e5eaa 2025-12-04T08:27:07.4045876Z * [new tag] trunk/9a296e640fc88aa44d275b48cd9cc30c573b169d -> trunk/9a296e640fc88aa44d275b48cd9cc30c573b169d 2025-12-04T08:27:07.4046106Z * [new tag] trunk/9b3e34d8589b29f7b4e7fab6f78711b7ca6e4639 -> trunk/9b3e34d8589b29f7b4e7fab6f78711b7ca6e4639 2025-12-04T08:27:07.4046337Z * [new tag] trunk/9cd055e547e9b67a5f9827f8999c38d7eda1bcb8 -> trunk/9cd055e547e9b67a5f9827f8999c38d7eda1bcb8 2025-12-04T08:27:07.4046581Z * [new tag] trunk/9f0df5686cb4ada94f94620acba2e3c3f363b11d -> trunk/9f0df5686cb4ada94f94620acba2e3c3f363b11d 2025-12-04T08:27:07.4046818Z * [new tag] trunk/9f7fceb887d0cfa0326a59b887821c63ff11340a -> trunk/9f7fceb887d0cfa0326a59b887821c63ff11340a 2025-12-04T08:27:07.4047063Z * [new tag] trunk/9f8ef8855d3078d70f7b782540ff2aaf158d6742 -> trunk/9f8ef8855d3078d70f7b782540ff2aaf158d6742 2025-12-04T08:27:07.4047307Z * [new tag] trunk/9fb52efc797b47a1f425a03aa5e47b866d8b1098 -> trunk/9fb52efc797b47a1f425a03aa5e47b866d8b1098 2025-12-04T08:27:07.4047542Z * [new tag] trunk/9ff4a2ebc5762d46c73e46b1b523d7ff349fedfa -> trunk/9ff4a2ebc5762d46c73e46b1b523d7ff349fedfa 2025-12-04T08:27:07.4047775Z * [new tag] trunk/a0f3937b94422354538ebbd47202d5b0e8a3fd0d -> trunk/a0f3937b94422354538ebbd47202d5b0e8a3fd0d 2025-12-04T08:27:07.4048004Z * [new tag] trunk/a15066c28b3145e6edbfc88359d0411d14cfc70c -> trunk/a15066c28b3145e6edbfc88359d0411d14cfc70c 2025-12-04T08:27:07.4048285Z * [new tag] trunk/a20f775e82564d2a9979221ed7f3b8d7cf54ce90 -> trunk/a20f775e82564d2a9979221ed7f3b8d7cf54ce90 2025-12-04T08:27:07.4048686Z * [new tag] trunk/a2973fb00ec002dd4b6bbf07385f066efb259b8c -> trunk/a2973fb00ec002dd4b6bbf07385f066efb259b8c 2025-12-04T08:27:07.4048938Z * [new tag] trunk/a7dc6dab9ad911259d4801c502907e531594db45 -> trunk/a7dc6dab9ad911259d4801c502907e531594db45 2025-12-04T08:27:07.4049177Z * [new tag] trunk/a951a9cee65c01660bbc6e6fded90ecb10fa6109 -> trunk/a951a9cee65c01660bbc6e6fded90ecb10fa6109 2025-12-04T08:27:07.4049540Z * [new tag] trunk/abfa1a6d65c7c159e35c72c25979b9da4971689e -> trunk/abfa1a6d65c7c159e35c72c25979b9da4971689e 2025-12-04T08:27:07.4049963Z * [new tag] trunk/ae3a2395bf66151078e2d201716f7d63ce1c6f3e -> trunk/ae3a2395bf66151078e2d201716f7d63ce1c6f3e 2025-12-04T08:27:07.4050622Z * [new tag] trunk/afdff7f0325080dedac44d080cb5a3b0e65e6c5e -> trunk/afdff7f0325080dedac44d080cb5a3b0e65e6c5e 2025-12-04T08:27:07.4051062Z * [new tag] trunk/b1aed4e7a72c03a38f44543aaea0dae2e9b76d48 -> trunk/b1aed4e7a72c03a38f44543aaea0dae2e9b76d48 2025-12-04T08:27:07.4051951Z * [new tag] trunk/b1decff555cd50e2123c8c6e25cc0d447c411f62 -> trunk/b1decff555cd50e2123c8c6e25cc0d447c411f62 2025-12-04T08:27:07.4052310Z * [new tag] trunk/b2b6b034c9fd08672c40e63ef243556ad4c49bd2 -> trunk/b2b6b034c9fd08672c40e63ef243556ad4c49bd2 2025-12-04T08:27:07.4055151Z * [new tag] trunk/b39813b4a04931682b0491adba2138d01d716d99 -> trunk/b39813b4a04931682b0491adba2138d01d716d99 2025-12-04T08:27:07.4055494Z * [new tag] trunk/b3a7edb2311367974cc7cd764cfb11a5d6758b24 -> trunk/b3a7edb2311367974cc7cd764cfb11a5d6758b24 2025-12-04T08:27:07.4055758Z * [new tag] trunk/b4cc1329c86acaef6d42c1fac7169b8d870ab0d7 -> trunk/b4cc1329c86acaef6d42c1fac7169b8d870ab0d7 2025-12-04T08:27:07.4056033Z * [new tag] trunk/b555c39217f765759954a4f9f9bd1e9b87bed11a -> trunk/b555c39217f765759954a4f9f9bd1e9b87bed11a 2025-12-04T08:27:07.4056289Z * [new tag] trunk/b6b6c80379388b7f9932c3e6a0f9907bf430e417 -> trunk/b6b6c80379388b7f9932c3e6a0f9907bf430e417 2025-12-04T08:27:07.4062551Z * [new tag] trunk/b6b6d912df0b6f4082f8e50b18bd1de1dd7325f4 -> trunk/b6b6d912df0b6f4082f8e50b18bd1de1dd7325f4 2025-12-04T08:27:07.4063029Z * [new tag] trunk/b7d60685f8cbc939b68a20871e90db67e729329b -> trunk/b7d60685f8cbc939b68a20871e90db67e729329b 2025-12-04T08:27:07.4063452Z * [new tag] trunk/b7f6b9a4fc6259f7af068f31868b3119bb1bac3e -> trunk/b7f6b9a4fc6259f7af068f31868b3119bb1bac3e 2025-12-04T08:27:07.4063838Z * [new tag] trunk/b8c4ba3593761e7b2a3ebd86f040fb07b47c02cf -> trunk/b8c4ba3593761e7b2a3ebd86f040fb07b47c02cf 2025-12-04T08:27:07.4064469Z * [new tag] trunk/b9c8f3a4884befb965ff42620ce44a71b04887f5 -> trunk/b9c8f3a4884befb965ff42620ce44a71b04887f5 2025-12-04T08:27:07.4064761Z * [new tag] trunk/ba1412546f3082c0958c077acc2025e4dbc33f1f -> trunk/ba1412546f3082c0958c077acc2025e4dbc33f1f 2025-12-04T08:27:07.4065009Z * [new tag] trunk/bac403c0b38c63bdbcc0c31f1c2b0bc0260f610f -> trunk/bac403c0b38c63bdbcc0c31f1c2b0bc0260f610f 2025-12-04T08:27:07.4065248Z * [new tag] trunk/bb3034198b459401fabeab254e1b99f0115046e2 -> trunk/bb3034198b459401fabeab254e1b99f0115046e2 2025-12-04T08:27:07.4065476Z * [new tag] trunk/bc39b2b3bc7a6e19a42e62bd576974035086fe55 -> trunk/bc39b2b3bc7a6e19a42e62bd576974035086fe55 2025-12-04T08:27:07.4066023Z * [new tag] trunk/bc43d5b297f207a11d83d77ddf0152bdaabe15a8 -> trunk/bc43d5b297f207a11d83d77ddf0152bdaabe15a8 2025-12-04T08:27:07.4066391Z * [new tag] trunk/bc6a4863c7246a6493d16d4ea6eee71ec07c6a09 -> trunk/bc6a4863c7246a6493d16d4ea6eee71ec07c6a09 2025-12-04T08:27:07.4066761Z * [new tag] trunk/bea4912944defdbcb8b061800caab6cbbbd01df5 -> trunk/bea4912944defdbcb8b061800caab6cbbbd01df5 2025-12-04T08:27:07.4067281Z * [new tag] trunk/c04e2c656f48d82d1521b867bbbf03967b9b7564 -> trunk/c04e2c656f48d82d1521b867bbbf03967b9b7564 2025-12-04T08:27:07.4067644Z * [new tag] trunk/c0660bcee27e7d7731634e274576a7081882bede -> trunk/c0660bcee27e7d7731634e274576a7081882bede 2025-12-04T08:27:07.4067978Z * [new tag] trunk/c178ed43d3d99cbefe84fbfb21d6f282b20d62ac -> trunk/c178ed43d3d99cbefe84fbfb21d6f282b20d62ac 2025-12-04T08:27:07.4068286Z * [new tag] trunk/c55b1e8f61d041ee436d697449eb028931d574fb -> trunk/c55b1e8f61d041ee436d697449eb028931d574fb 2025-12-04T08:27:07.4073933Z * [new tag] trunk/c6ae7579fe12fe75f1a8f7043a494c90567273f1 -> trunk/c6ae7579fe12fe75f1a8f7043a494c90567273f1 2025-12-04T08:27:07.4074442Z * [new tag] trunk/c8210e7d94bad5ae21ac389fa4ba8a463c76c4d0 -> trunk/c8210e7d94bad5ae21ac389fa4ba8a463c76c4d0 2025-12-04T08:27:07.4074861Z * [new tag] trunk/cc0853af42122f8185321f542616f4474e717f09 -> trunk/cc0853af42122f8185321f542616f4474e717f09 2025-12-04T08:27:07.4075215Z * [new tag] trunk/cddec6562eabfa390d014fa3741a5659cf9c94c9 -> trunk/cddec6562eabfa390d014fa3741a5659cf9c94c9 2025-12-04T08:27:07.4075953Z * [new tag] trunk/ce5e7e3bf1f4b69a4f4f93d288ba75b906df492a -> trunk/ce5e7e3bf1f4b69a4f4f93d288ba75b906df492a 2025-12-04T08:27:07.4076251Z * [new tag] trunk/d038b0130ec7c20ebcac219301292fd8e98a1ace -> trunk/d038b0130ec7c20ebcac219301292fd8e98a1ace 2025-12-04T08:27:07.4076516Z * [new tag] trunk/d16447dacaf2420ea175f0c275c75da951f57d39 -> trunk/d16447dacaf2420ea175f0c275c75da951f57d39 2025-12-04T08:27:07.4076784Z * [new tag] trunk/d19f1e8cab6810bb2e99141f9976665954c67a50 -> trunk/d19f1e8cab6810bb2e99141f9976665954c67a50 2025-12-04T08:27:07.4077039Z * [new tag] trunk/d1c9f03b2a5af4104721712f8cdffe9b4f340c01 -> trunk/d1c9f03b2a5af4104721712f8cdffe9b4f340c01 2025-12-04T08:27:07.4077317Z * [new tag] trunk/d40f4950f2b7f7aa380a22fe0f6166e71680fbcf -> trunk/d40f4950f2b7f7aa380a22fe0f6166e71680fbcf 2025-12-04T08:27:07.4077571Z * [new tag] trunk/d5038950bacfe36bbf24a47a455fe76901deb8e8 -> trunk/d5038950bacfe36bbf24a47a455fe76901deb8e8 2025-12-04T08:27:07.4078004Z * [new tag] trunk/d54ff42903c2ae0533931ff11d23b35f875bdb3d -> trunk/d54ff42903c2ae0533931ff11d23b35f875bdb3d 2025-12-04T08:27:07.4078427Z * [new tag] trunk/d76697633a2d2b9cced1ae21161849b33bfe7e47 -> trunk/d76697633a2d2b9cced1ae21161849b33bfe7e47 2025-12-04T08:27:07.4078677Z * [new tag] trunk/d78f52b199c547106d4cd9d2856dd0805c118bf1 -> trunk/d78f52b199c547106d4cd9d2856dd0805c118bf1 2025-12-04T08:27:07.4078931Z * [new tag] trunk/d8fd5c6eed28e5004150691d048a3f6785e19a8e -> trunk/d8fd5c6eed28e5004150691d048a3f6785e19a8e 2025-12-04T08:27:07.4079188Z * [new tag] trunk/d900f5e86745dec76713f4b0ef07005ef36b2f5a -> trunk/d900f5e86745dec76713f4b0ef07005ef36b2f5a 2025-12-04T08:27:07.4079435Z * [new tag] trunk/d973dc6b87d763859fe1c5bd1287e3b6b1c49d1b -> trunk/d973dc6b87d763859fe1c5bd1287e3b6b1c49d1b 2025-12-04T08:27:07.4079696Z * [new tag] trunk/d998c03304cb6ede76e1ed535b4ddeb6c2bf40ec -> trunk/d998c03304cb6ede76e1ed535b4ddeb6c2bf40ec 2025-12-04T08:27:07.4079958Z * [new tag] trunk/d9cb8a70833101dbbe16b99520cfbdd70d0a87bf -> trunk/d9cb8a70833101dbbe16b99520cfbdd70d0a87bf 2025-12-04T08:27:07.4080210Z * [new tag] trunk/d9d5e91b43f70eb8637af55db6856d49be391ffd -> trunk/d9d5e91b43f70eb8637af55db6856d49be391ffd 2025-12-04T08:27:07.4080461Z * [new tag] trunk/dd18a75336a4fbd7497955cc5665904724fce889 -> trunk/dd18a75336a4fbd7497955cc5665904724fce889 2025-12-04T08:27:07.4085251Z * [new tag] trunk/ded9bcd61a059bf723e6e84689552962b480ea77 -> trunk/ded9bcd61a059bf723e6e84689552962b480ea77 2025-12-04T08:27:07.4089607Z * [new tag] trunk/dfbd3714d15c37a7b83b322a6b60f997fc00f50c -> trunk/dfbd3714d15c37a7b83b322a6b60f997fc00f50c 2025-12-04T08:27:07.4093594Z * [new tag] trunk/e115f9f4e4b039f8e9a642aaa2bd8254a920541b -> trunk/e115f9f4e4b039f8e9a642aaa2bd8254a920541b 2025-12-04T08:27:07.4096708Z * [new tag] trunk/e3f24fd73ad74c6e7176687986436956c7c18235 -> trunk/e3f24fd73ad74c6e7176687986436956c7c18235 2025-12-04T08:27:07.4102209Z * [new tag] trunk/e7d24d3ff93d1503ba63860b7057438ad93f918e -> trunk/e7d24d3ff93d1503ba63860b7057438ad93f918e 2025-12-04T08:27:07.4102500Z * [new tag] trunk/ea7035f462a0d2830865ee86c832bd101e1427fc -> trunk/ea7035f462a0d2830865ee86c832bd101e1427fc 2025-12-04T08:27:07.4102772Z * [new tag] trunk/eb5c63652a33da42e7018c23df5f20a3eb4c6ccf -> trunk/eb5c63652a33da42e7018c23df5f20a3eb4c6ccf 2025-12-04T08:27:07.4103011Z * [new tag] trunk/ec2c71f5c85021b8938cdafadce24c15a36fd93e -> trunk/ec2c71f5c85021b8938cdafadce24c15a36fd93e 2025-12-04T08:27:07.4103290Z * [new tag] trunk/ecbcc3f6bf327856b435b259ac63cc2f328c4b4e -> trunk/ecbcc3f6bf327856b435b259ac63cc2f328c4b4e 2025-12-04T08:27:07.4103515Z * [new tag] trunk/ee87bbe876c42575e961b32a0827d76bc9782ca2 -> trunk/ee87bbe876c42575e961b32a0827d76bc9782ca2 2025-12-04T08:27:07.4103753Z * [new tag] trunk/ef019d1d431c4c5a95b594cb90d40a50cd00f5e4 -> trunk/ef019d1d431c4c5a95b594cb90d40a50cd00f5e4 2025-12-04T08:27:07.4103987Z * [new tag] trunk/ef8ecc13830a86c4b231f1aad9aba7851db61b53 -> trunk/ef8ecc13830a86c4b231f1aad9aba7851db61b53 2025-12-04T08:27:07.4104202Z * [new tag] trunk/f1076f5510920044912247b1abb8760cb820f598 -> trunk/f1076f5510920044912247b1abb8760cb820f598 2025-12-04T08:27:07.4104442Z * [new tag] trunk/f2d6a75a00a1d648ca9a0abc6a33e14c3dea6c40 -> trunk/f2d6a75a00a1d648ca9a0abc6a33e14c3dea6c40 2025-12-04T08:27:07.4104679Z * [new tag] trunk/f47dd0ddef1359e5b43e4b962412f67b30ecde56 -> trunk/f47dd0ddef1359e5b43e4b962412f67b30ecde56 2025-12-04T08:27:07.4104915Z * [new tag] trunk/f49d32dfa4730dcfb1b60eeeb369b5889da983c8 -> trunk/f49d32dfa4730dcfb1b60eeeb369b5889da983c8 2025-12-04T08:27:07.4105288Z * [new tag] trunk/f4dedf78fc30fd4b93975787ca6074ee89db9467 -> trunk/f4dedf78fc30fd4b93975787ca6074ee89db9467 2025-12-04T08:27:07.4105524Z * [new tag] trunk/f7c0d03819ebed05c4038f095d66d1b8c54aca17 -> trunk/f7c0d03819ebed05c4038f095d66d1b8c54aca17 2025-12-04T08:27:07.4105748Z * [new tag] trunk/f7e1bd80a063e17453c361837ba6ea2570920a73 -> trunk/f7e1bd80a063e17453c361837ba6ea2570920a73 2025-12-04T08:27:07.4105974Z * [new tag] trunk/f9bd6c53624c7c0ea3772de78498326e84c2f0e7 -> trunk/f9bd6c53624c7c0ea3772de78498326e84c2f0e7 2025-12-04T08:27:07.4106213Z * [new tag] trunk/fb5be221a46b51bfc9509013b0d85bc5a9d4f15b -> trunk/fb5be221a46b51bfc9509013b0d85bc5a9d4f15b 2025-12-04T08:27:07.4106450Z * [new tag] trunk/fdf863d5e1de3b2688c9511e96876e34581dbfd7 -> trunk/fdf863d5e1de3b2688c9511e96876e34581dbfd7 2025-12-04T08:27:07.4106729Z * [new tag] trunk/fe0e65adfc0e7ca6e5f57e6ea8b16bd5cc967307 -> trunk/fe0e65adfc0e7ca6e5f57e6ea8b16bd5cc967307 2025-12-04T08:27:07.4106964Z * [new tag] trunk/fec710bf89173f5355468a7ce1afe9157c3d9009 -> trunk/fec710bf89173f5355468a7ce1afe9157c3d9009 2025-12-04T08:27:07.4107206Z * [new tag] trunk/ffd9b0fb4355e97af82fc42cf185c3ffa0fc0a32 -> trunk/ffd9b0fb4355e97af82fc42cf185c3ffa0fc0a32 2025-12-04T08:27:07.4107326Z * [new tag] v0.1.1 -> v0.1.1 2025-12-04T08:27:07.4107437Z * [new tag] v0.1.10 -> v0.1.10 2025-12-04T08:27:07.4107548Z * [new tag] v0.1.11 -> v0.1.11 2025-12-04T08:27:07.4107644Z * [new tag] v0.1.12 -> v0.1.12 2025-12-04T08:27:07.4107788Z * [new tag] v0.1.2 -> v0.1.2 2025-12-04T08:27:07.4107891Z * [new tag] v0.1.3 -> v0.1.3 2025-12-04T08:27:07.4107984Z * [new tag] v0.1.4 -> v0.1.4 2025-12-04T08:27:07.4108088Z * [new tag] v0.1.5 -> v0.1.5 2025-12-04T08:27:07.4108186Z * [new tag] v0.1.6 -> v0.1.6 2025-12-04T08:27:07.4108274Z * [new tag] v0.1.7 -> v0.1.7 2025-12-04T08:27:07.4108368Z * [new tag] v0.1.8 -> v0.1.8 2025-12-04T08:27:07.4108457Z * [new tag] v0.1.9 -> v0.1.9 2025-12-04T08:27:07.4108545Z * [new tag] v0.2.0 -> v0.2.0 2025-12-04T08:27:07.4108638Z * [new tag] v0.3.0 -> v0.3.0 2025-12-04T08:27:07.4108726Z * [new tag] v0.3.1 -> v0.3.1 2025-12-04T08:27:07.4108815Z * [new tag] v0.4.0 -> v0.4.0 2025-12-04T08:27:07.4108910Z * [new tag] v0.4.1 -> v0.4.1 2025-12-04T08:27:07.4109000Z * [new tag] v1.0.0 -> v1.0.0 2025-12-04T08:27:07.4109113Z * [new tag] v1.0.0a0 -> v1.0.0a0 2025-12-04T08:27:07.4109203Z * [new tag] v1.0.1 -> v1.0.1 2025-12-04T08:27:07.4109296Z * [new tag] v1.0rc0 -> v1.0rc0 2025-12-04T08:27:07.4109399Z * [new tag] v1.0rc1 -> v1.0rc1 2025-12-04T08:27:07.4109498Z * [new tag] v1.1.0 -> v1.1.0 2025-12-04T08:27:07.4109603Z * [new tag] v1.1.0a0 -> v1.1.0a0 2025-12-04T08:27:07.4109692Z * [new tag] v1.10.0 -> v1.10.0 2025-12-04T08:27:07.4109801Z * [new tag] v1.10.0-rc1 -> v1.10.0-rc1 2025-12-04T08:27:07.4109907Z * [new tag] v1.10.0-rc2 -> v1.10.0-rc2 2025-12-04T08:27:07.4110004Z * [new tag] v1.10.0-rc3 -> v1.10.0-rc3 2025-12-04T08:27:07.4110131Z * [new tag] v1.10.1 -> v1.10.1 2025-12-04T08:27:07.4110235Z * [new tag] v1.10.1-rc1 -> v1.10.1-rc1 2025-12-04T08:27:07.4110326Z * [new tag] v1.10.2 -> v1.10.2 2025-12-04T08:27:07.4110433Z * [new tag] v1.10.2-rc1 -> v1.10.2-rc1 2025-12-04T08:27:07.4110541Z * [new tag] v1.11.0 -> v1.11.0 2025-12-04T08:27:07.4111848Z * [new tag] v1.11.0-rc1 -> v1.11.0-rc1 2025-12-04T08:27:07.4112143Z * [new tag] v1.11.0-rc2 -> v1.11.0-rc2 2025-12-04T08:27:07.4114869Z * [new tag] v1.11.0-rc3 -> v1.11.0-rc3 2025-12-04T08:27:07.4115035Z * [new tag] v1.11.0-rc4 -> v1.11.0-rc4 2025-12-04T08:27:07.4115145Z * [new tag] v1.11.0-rc5 -> v1.11.0-rc5 2025-12-04T08:27:07.4115263Z * [new tag] v1.11.0-rc6 -> v1.11.0-rc6 2025-12-04T08:27:07.4115365Z * [new tag] v1.11.0-rc7 -> v1.11.0-rc7 2025-12-04T08:27:07.4116387Z * [new tag] v1.12.0 -> v1.12.0 2025-12-04T08:27:07.4116845Z * [new tag] v1.12.0-rc1 -> v1.12.0-rc1 2025-12-04T08:27:07.4118185Z * [new tag] v1.12.0-rc2 -> v1.12.0-rc2 2025-12-04T08:27:07.4118781Z * [new tag] v1.12.0-rc3 -> v1.12.0-rc3 2025-12-04T08:27:07.4121656Z * [new tag] v1.12.0-rc4 -> v1.12.0-rc4 2025-12-04T08:27:07.4121797Z * [new tag] v1.12.0-rc5 -> v1.12.0-rc5 2025-12-04T08:27:07.4122099Z * [new tag] v1.12.0-rc6 -> v1.12.0-rc6 2025-12-04T08:27:07.4122205Z * [new tag] v1.12.0-rc7 -> v1.12.0-rc7 2025-12-04T08:27:07.4122305Z * [new tag] v1.12.0-rc8 -> v1.12.0-rc8 2025-12-04T08:27:07.4123066Z * [new tag] v1.12.1 -> v1.12.1 2025-12-04T08:27:07.4124215Z * [new tag] v1.12.1-rc1 -> v1.12.1-rc1 2025-12-04T08:27:07.4124647Z * [new tag] v1.12.1-rc2 -> v1.12.1-rc2 2025-12-04T08:27:07.4128622Z * [new tag] v1.12.1-rc3 -> v1.12.1-rc3 2025-12-04T08:27:07.4128762Z * [new tag] v1.12.1-rc4 -> v1.12.1-rc4 2025-12-04T08:27:07.4128862Z * [new tag] v1.12.1-rc5 -> v1.12.1-rc5 2025-12-04T08:27:07.4128965Z * [new tag] v1.13.0 -> v1.13.0 2025-12-04T08:27:07.4129089Z * [new tag] v1.13.0-rc1 -> v1.13.0-rc1 2025-12-04T08:27:07.4129185Z * [new tag] v1.13.0-rc2 -> v1.13.0-rc2 2025-12-04T08:27:07.4129290Z * [new tag] v1.13.0-rc3 -> v1.13.0-rc3 2025-12-04T08:27:07.4131036Z * [new tag] v1.13.0-rc4 -> v1.13.0-rc4 2025-12-04T08:27:07.4131180Z * [new tag] v1.13.0-rc5 -> v1.13.0-rc5 2025-12-04T08:27:07.4131283Z * [new tag] v1.13.0-rc6 -> v1.13.0-rc6 2025-12-04T08:27:07.4134064Z * [new tag] v1.13.1 -> v1.13.1 2025-12-04T08:27:07.4134348Z * [new tag] v1.13.1-rc1 -> v1.13.1-rc1 2025-12-04T08:27:07.4134469Z * [new tag] v1.2.0 -> v1.2.0 2025-12-04T08:27:07.4134654Z * [new tag] v1.2.0a0 -> v1.2.0a0 2025-12-04T08:27:07.4134774Z * [new tag] v1.3.0 -> v1.3.0 2025-12-04T08:27:07.4135000Z * [new tag] v1.3.0a0 -> v1.3.0a0 2025-12-04T08:27:07.4135116Z * [new tag] v1.3.1 -> v1.3.1 2025-12-04T08:27:07.4135792Z * [new tag] v1.4.0 -> v1.4.0 2025-12-04T08:27:07.4136437Z * [new tag] v1.4.0a0 -> v1.4.0a0 2025-12-04T08:27:07.4136608Z * [new tag] v1.4.1 -> v1.4.1 2025-12-04T08:27:07.4140457Z * [new tag] v1.5.0 -> v1.5.0 2025-12-04T08:27:07.4140616Z * [new tag] v1.5.0-rc1 -> v1.5.0-rc1 2025-12-04T08:27:07.4140726Z * [new tag] v1.5.0-rc2 -> v1.5.0-rc2 2025-12-04T08:27:07.4140821Z * [new tag] v1.5.0-rc3 -> v1.5.0-rc3 2025-12-04T08:27:07.4140925Z * [new tag] v1.5.0-rc4 -> v1.5.0-rc4 2025-12-04T08:27:07.4141037Z * [new tag] v1.5.0-rc5 -> v1.5.0-rc5 2025-12-04T08:27:07.4141138Z * [new tag] v1.5.1 -> v1.5.1 2025-12-04T08:27:07.4141277Z * [new tag] v1.5.1-rc1 -> v1.5.1-rc1 2025-12-04T08:27:07.4142812Z * [new tag] v1.6.0 -> v1.6.0 2025-12-04T08:27:07.4143113Z * [new tag] v1.6.0-rc1 -> v1.6.0-rc1 2025-12-04T08:27:07.4143243Z * [new tag] v1.6.0-rc2 -> v1.6.0-rc2 2025-12-04T08:27:07.4146409Z * [new tag] v1.6.0-rc3 -> v1.6.0-rc3 2025-12-04T08:27:07.4146709Z * [new tag] v1.6.0-rc4 -> v1.6.0-rc4 2025-12-04T08:27:07.4147004Z * [new tag] v1.6.0-rc5 -> v1.6.0-rc5 2025-12-04T08:27:07.4147153Z * [new tag] v1.6.0-rc6 -> v1.6.0-rc6 2025-12-04T08:27:07.4147407Z * [new tag] v1.6.0-rc7 -> v1.6.0-rc7 2025-12-04T08:27:07.4147534Z * [new tag] v1.7.0 -> v1.7.0 2025-12-04T08:27:07.4147657Z * [new tag] v1.7.0-rc1 -> v1.7.0-rc1 2025-12-04T08:27:07.4148135Z * [new tag] v1.7.0-rc2 -> v1.7.0-rc2 2025-12-04T08:27:07.4149006Z * [new tag] v1.7.0-rc3 -> v1.7.0-rc3 2025-12-04T08:27:07.4149497Z * [new tag] v1.7.0-rc4 -> v1.7.0-rc4 2025-12-04T08:27:07.4149918Z * [new tag] v1.7.1 -> v1.7.1 2025-12-04T08:27:07.4150875Z * [new tag] v1.7.1-rc1 -> v1.7.1-rc1 2025-12-04T08:27:07.4151134Z * [new tag] v1.7.1-rc2 -> v1.7.1-rc2 2025-12-04T08:27:07.4151951Z * [new tag] v1.7.1-rc3 -> v1.7.1-rc3 2025-12-04T08:27:07.4152328Z * [new tag] v1.8.0 -> v1.8.0 2025-12-04T08:27:07.4152852Z * [new tag] v1.8.0-rc1 -> v1.8.0-rc1 2025-12-04T08:27:07.4153642Z * [new tag] v1.8.0-rc2 -> v1.8.0-rc2 2025-12-04T08:27:07.4154030Z * [new tag] v1.8.0-rc3 -> v1.8.0-rc3 2025-12-04T08:27:07.4154842Z * [new tag] v1.8.0-rc4 -> v1.8.0-rc4 2025-12-04T08:27:07.4155126Z * [new tag] v1.8.0-rc5 -> v1.8.0-rc5 2025-12-04T08:27:07.4155440Z * [new tag] v1.8.1 -> v1.8.1 2025-12-04T08:27:07.4156342Z * [new tag] v1.8.1-rc1 -> v1.8.1-rc1 2025-12-04T08:27:07.4156454Z * [new tag] v1.8.1-rc2 -> v1.8.1-rc2 2025-12-04T08:27:07.4157062Z * [new tag] v1.8.1-rc3 -> v1.8.1-rc3 2025-12-04T08:27:07.4158474Z * [new tag] v1.8.2 -> v1.8.2 2025-12-04T08:27:07.4158702Z * [new tag] v1.8.2-rc1 -> v1.8.2-rc1 2025-12-04T08:27:07.4159280Z * [new tag] v1.9.0 -> v1.9.0 2025-12-04T08:27:07.4160139Z * [new tag] v1.9.0-rc1 -> v1.9.0-rc1 2025-12-04T08:27:07.4160391Z * [new tag] v1.9.0-rc2 -> v1.9.0-rc2 2025-12-04T08:27:07.4162604Z * [new tag] v1.9.0-rc3 -> v1.9.0-rc3 2025-12-04T08:27:07.4162744Z * [new tag] v1.9.0-rc4 -> v1.9.0-rc4 2025-12-04T08:27:07.4162856Z * [new tag] v1.9.1 -> v1.9.1 2025-12-04T08:27:07.4162975Z * [new tag] v1.9.1-rc1 -> v1.9.1-rc1 2025-12-04T08:27:07.4163503Z * [new tag] v1.9.1-rc2 -> v1.9.1-rc2 2025-12-04T08:27:07.4164419Z * [new tag] v2.0.0 -> v2.0.0 2025-12-04T08:27:07.4164694Z * [new tag] v2.0.0-rc1 -> v2.0.0-rc1 2025-12-04T08:27:07.4165443Z * [new tag] v2.0.0-rc2 -> v2.0.0-rc2 2025-12-04T08:27:07.4165807Z * [new tag] v2.0.0-rc3 -> v2.0.0-rc3 2025-12-04T08:27:07.4168638Z * [new tag] v2.0.0-rc4 -> v2.0.0-rc4 2025-12-04T08:27:07.4168773Z * [new tag] v2.0.0-rc5 -> v2.0.0-rc5 2025-12-04T08:27:07.4168882Z * [new tag] v2.0.0-rc6 -> v2.0.0-rc6 2025-12-04T08:27:07.4168989Z * [new tag] v2.0.1 -> v2.0.1 2025-12-04T08:27:07.4169213Z * [new tag] v2.0.1-rc1 -> v2.0.1-rc1 2025-12-04T08:27:07.4169625Z * [new tag] v2.0.1-rc2 -> v2.0.1-rc2 2025-12-04T08:27:07.4170061Z * [new tag] v2.0.1-rc3 -> v2.0.1-rc3 2025-12-04T08:27:07.4171225Z * [new tag] v2.0.1-rc4 -> v2.0.1-rc4 2025-12-04T08:27:07.4172076Z * [new tag] v2.1.0 -> v2.1.0 2025-12-04T08:27:07.4172197Z * [new tag] v2.1.0-rc1 -> v2.1.0-rc1 2025-12-04T08:27:07.4176440Z * [new tag] v2.1.0-rc2 -> v2.1.0-rc2 2025-12-04T08:27:07.4176576Z * [new tag] v2.1.0-rc3 -> v2.1.0-rc3 2025-12-04T08:27:07.4176688Z * [new tag] v2.1.0-rc4 -> v2.1.0-rc4 2025-12-04T08:27:07.4176788Z * [new tag] v2.1.0-rc5 -> v2.1.0-rc5 2025-12-04T08:27:07.4176887Z * [new tag] v2.1.0-rc6 -> v2.1.0-rc6 2025-12-04T08:27:07.4177006Z * [new tag] v2.1.1 -> v2.1.1 2025-12-04T08:27:07.4177109Z * [new tag] v2.1.1-rc1 -> v2.1.1-rc1 2025-12-04T08:27:07.4177214Z * [new tag] v2.1.1-rc2 -> v2.1.1-rc2 2025-12-04T08:27:07.4178043Z * [new tag] v2.1.1-rc3 -> v2.1.1-rc3 2025-12-04T08:27:07.4178444Z * [new tag] v2.1.1-rc4 -> v2.1.1-rc4 2025-12-04T08:27:07.4179290Z * [new tag] v2.1.1-rc5 -> v2.1.1-rc5 2025-12-04T08:27:07.4179564Z * [new tag] v2.1.1-rc6 -> v2.1.1-rc6 2025-12-04T08:27:07.4179978Z * [new tag] v2.1.2 -> v2.1.2 2025-12-04T08:27:07.4183531Z * [new tag] v2.1.2-rc1 -> v2.1.2-rc1 2025-12-04T08:27:07.4183674Z * [new tag] v2.1.2-rc2 -> v2.1.2-rc2 2025-12-04T08:27:07.4183776Z * [new tag] v2.1.2-rc3 -> v2.1.2-rc3 2025-12-04T08:27:07.4183886Z * [new tag] v2.2.0 -> v2.2.0 2025-12-04T08:27:07.4183984Z * [new tag] v2.2.0-rc1 -> v2.2.0-rc1 2025-12-04T08:27:07.4184093Z * [new tag] v2.2.0-rc2 -> v2.2.0-rc2 2025-12-04T08:27:07.4184200Z * [new tag] v2.2.0-rc3 -> v2.2.0-rc3 2025-12-04T08:27:07.4189654Z * [new tag] v2.2.0-rc4 -> v2.2.0-rc4 2025-12-04T08:27:07.4189950Z * [new tag] v2.2.0-rc5 -> v2.2.0-rc5 2025-12-04T08:27:07.4190050Z * [new tag] v2.2.0-rc6 -> v2.2.0-rc6 2025-12-04T08:27:07.4190144Z * [new tag] v2.2.0-rc7 -> v2.2.0-rc7 2025-12-04T08:27:07.4190247Z * [new tag] v2.2.0-rc8 -> v2.2.0-rc8 2025-12-04T08:27:07.4190348Z * [new tag] v2.2.1 -> v2.2.1 2025-12-04T08:27:07.4190448Z * [new tag] v2.2.1-rc1 -> v2.2.1-rc1 2025-12-04T08:27:07.4190552Z * [new tag] v2.2.1-rc2 -> v2.2.1-rc2 2025-12-04T08:27:07.4190654Z * [new tag] v2.2.1-rc3 -> v2.2.1-rc3 2025-12-04T08:27:07.4195411Z * [new tag] v2.2.2 -> v2.2.2 2025-12-04T08:27:07.4195555Z * [new tag] v2.2.2-rc1 -> v2.2.2-rc1 2025-12-04T08:27:07.4195856Z * [new tag] v2.2.2-rc2 -> v2.2.2-rc2 2025-12-04T08:27:07.4195963Z * [new tag] v2.2.2-rc3 -> v2.2.2-rc3 2025-12-04T08:27:07.4196188Z * [new tag] v2.3.0 -> v2.3.0 2025-12-04T08:27:07.4196318Z * [new tag] v2.3.0-rc1 -> v2.3.0-rc1 2025-12-04T08:27:07.4196437Z * [new tag] v2.3.0-rc10 -> v2.3.0-rc10 2025-12-04T08:27:07.4196552Z * [new tag] v2.3.0-rc11 -> v2.3.0-rc11 2025-12-04T08:27:07.4196662Z * [new tag] v2.3.0-rc12 -> v2.3.0-rc12 2025-12-04T08:27:07.4196767Z * [new tag] v2.3.0-rc2 -> v2.3.0-rc2 2025-12-04T08:27:07.4197376Z * [new tag] v2.3.0-rc3 -> v2.3.0-rc3 2025-12-04T08:27:07.4197495Z * [new tag] v2.3.0-rc4 -> v2.3.0-rc4 2025-12-04T08:27:07.4197748Z * [new tag] v2.3.0-rc5 -> v2.3.0-rc5 2025-12-04T08:27:07.4197856Z * [new tag] v2.3.0-rc6 -> v2.3.0-rc6 2025-12-04T08:27:07.4197974Z * [new tag] v2.3.0-rc7 -> v2.3.0-rc7 2025-12-04T08:27:07.4198307Z * [new tag] v2.3.0-rc8 -> v2.3.0-rc8 2025-12-04T08:27:07.4198420Z * [new tag] v2.3.0-rc9 -> v2.3.0-rc9 2025-12-04T08:27:07.4198536Z * [new tag] v2.3.1 -> v2.3.1 2025-12-04T08:27:07.4198638Z * [new tag] v2.3.1-rc1 -> v2.3.1-rc1 2025-12-04T08:27:07.4198741Z * [new tag] v2.3.1-rc2 -> v2.3.1-rc2 2025-12-04T08:27:07.4201753Z * [new tag] v2.3.1-rc3 -> v2.3.1-rc3 2025-12-04T08:27:07.4201849Z * [new tag] v2.4.0 -> v2.4.0 2025-12-04T08:27:07.4205871Z * [new tag] v2.4.0-rc1 -> v2.4.0-rc1 2025-12-04T08:27:07.4206079Z * [new tag] v2.4.0-rc2 -> v2.4.0-rc2 2025-12-04T08:27:07.4206265Z * [new tag] v2.4.0-rc3 -> v2.4.0-rc3 2025-12-04T08:27:07.4206457Z * [new tag] v2.4.0-rc4 -> v2.4.0-rc4 2025-12-04T08:27:07.4206649Z * [new tag] v2.4.0-rc5 -> v2.4.0-rc5 2025-12-04T08:27:07.4208762Z * [new tag] v2.4.0-rc6 -> v2.4.0-rc6 2025-12-04T08:27:07.4208957Z * [new tag] v2.4.0-rc7 -> v2.4.0-rc7 2025-12-04T08:27:07.4209149Z * [new tag] v2.4.0-rc8 -> v2.4.0-rc8 2025-12-04T08:27:07.4209337Z * [new tag] v2.4.0-rc9 -> v2.4.0-rc9 2025-12-04T08:27:07.4209512Z * [new tag] v2.4.1 -> v2.4.1 2025-12-04T08:27:07.4209700Z * [new tag] v2.4.1-rc1 -> v2.4.1-rc1 2025-12-04T08:27:07.4213092Z * [new tag] v2.4.1-rc2 -> v2.4.1-rc2 2025-12-04T08:27:07.4213299Z * [new tag] v2.4.1-rc3 -> v2.4.1-rc3 2025-12-04T08:27:07.4213483Z * [new tag] v2.5.0 -> v2.5.0 2025-12-04T08:27:07.4213661Z * [new tag] v2.5.0-rc1 -> v2.5.0-rc1 2025-12-04T08:27:07.4213829Z * [new tag] v2.5.0-rc10 -> v2.5.0-rc10 2025-12-04T08:27:07.4218825Z * [new tag] v2.5.0-rc2 -> v2.5.0-rc2 2025-12-04T08:27:07.4219132Z * [new tag] v2.5.0-rc3 -> v2.5.0-rc3 2025-12-04T08:27:07.4219266Z * [new tag] v2.5.0-rc4 -> v2.5.0-rc4 2025-12-04T08:27:07.4219383Z * [new tag] v2.5.0-rc5 -> v2.5.0-rc5 2025-12-04T08:27:07.4219494Z * [new tag] v2.5.0-rc6 -> v2.5.0-rc6 2025-12-04T08:27:07.4219601Z * [new tag] v2.5.0-rc7 -> v2.5.0-rc7 2025-12-04T08:27:07.4219707Z * [new tag] v2.5.0-rc8 -> v2.5.0-rc8 2025-12-04T08:27:07.4219806Z * [new tag] v2.5.0-rc9 -> v2.5.0-rc9 2025-12-04T08:27:07.4219906Z * [new tag] v2.5.1 -> v2.5.1 2025-12-04T08:27:07.4220009Z * [new tag] v2.5.1-rc1 -> v2.5.1-rc1 2025-12-04T08:27:07.4220107Z * [new tag] v2.6.0 -> v2.6.0 2025-12-04T08:27:07.4220210Z * [new tag] v2.6.0-rc1 -> v2.6.0-rc1 2025-12-04T08:27:07.4220306Z * [new tag] v2.6.0-rc2 -> v2.6.0-rc2 2025-12-04T08:27:07.4220491Z * [new tag] v2.6.0-rc3 -> v2.6.0-rc3 2025-12-04T08:27:07.4220600Z * [new tag] v2.6.0-rc4 -> v2.6.0-rc4 2025-12-04T08:27:07.4226188Z * [new tag] v2.6.0-rc5 -> v2.6.0-rc5 2025-12-04T08:27:07.4226354Z * [new tag] v2.6.0-rc6 -> v2.6.0-rc6 2025-12-04T08:27:07.4226469Z * [new tag] v2.6.0-rc7 -> v2.6.0-rc7 2025-12-04T08:27:07.4226575Z * [new tag] v2.6.0-rc8 -> v2.6.0-rc8 2025-12-04T08:27:07.4226681Z * [new tag] v2.6.0-rc9 -> v2.6.0-rc9 2025-12-04T08:27:07.4226799Z * [new tag] v2.7.0 -> v2.7.0 2025-12-04T08:27:07.4226905Z * [new tag] v2.7.0-rc1 -> v2.7.0-rc1 2025-12-04T08:27:07.4227017Z * [new tag] v2.7.0-rc10 -> v2.7.0-rc10 2025-12-04T08:27:07.4227139Z * [new tag] v2.7.0-rc2 -> v2.7.0-rc2 2025-12-04T08:27:07.4227238Z * [new tag] v2.7.0-rc3 -> v2.7.0-rc3 2025-12-04T08:27:07.4227347Z * [new tag] v2.7.0-rc4 -> v2.7.0-rc4 2025-12-04T08:27:07.4227486Z * [new tag] v2.7.0-rc5 -> v2.7.0-rc5 2025-12-04T08:27:07.4228498Z * [new tag] v2.7.0-rc6 -> v2.7.0-rc6 2025-12-04T08:27:07.4228766Z * [new tag] v2.7.0-rc7 -> v2.7.0-rc7 2025-12-04T08:27:07.4229796Z * [new tag] v2.7.0-rc8 -> v2.7.0-rc8 2025-12-04T08:27:07.4229981Z * [new tag] v2.7.0-rc9 -> v2.7.0-rc9 2025-12-04T08:27:07.4233987Z * [new tag] v2.7.1 -> v2.7.1 2025-12-04T08:27:07.4234123Z * [new tag] v2.7.1-rc1 -> v2.7.1-rc1 2025-12-04T08:27:07.4234376Z * [new tag] v2.7.1-rc2 -> v2.7.1-rc2 2025-12-04T08:27:07.4234498Z * [new tag] v2.7.1-rc3 -> v2.7.1-rc3 2025-12-04T08:27:07.4234595Z * [new tag] v2.7.1-rc4 -> v2.7.1-rc4 2025-12-04T08:27:07.4234905Z * [new tag] v2.7.1-rc5 -> v2.7.1-rc5 2025-12-04T08:27:07.4235035Z * [new tag] v2.8.0 -> v2.8.0 2025-12-04T08:27:07.4236366Z * [new tag] v2.8.0-rc1 -> v2.8.0-rc1 2025-12-04T08:27:07.4237172Z * [new tag] v2.8.0-rc2 -> v2.8.0-rc2 2025-12-04T08:27:07.4237415Z * [new tag] v2.8.0-rc3 -> v2.8.0-rc3 2025-12-04T08:27:07.4238954Z * [new tag] v2.8.0-rc4 -> v2.8.0-rc4 2025-12-04T08:27:07.4239600Z * [new tag] v2.8.0-rc5 -> v2.8.0-rc5 2025-12-04T08:27:07.4240761Z * [new tag] v2.8.0-rc6 -> v2.8.0-rc6 2025-12-04T08:27:07.4241390Z * [new tag] v2.8.0-rc7 -> v2.8.0-rc7 2025-12-04T08:27:07.4242341Z * [new tag] v2.8.0-rc8 -> v2.8.0-rc8 2025-12-04T08:27:07.4243330Z * [new tag] v2.9.0 -> v2.9.0 2025-12-04T08:27:07.4244780Z * [new tag] v2.9.0-rc1 -> v2.9.0-rc1 2025-12-04T08:27:07.4245080Z * [new tag] v2.9.0-rc10 -> v2.9.0-rc10 2025-12-04T08:27:07.4245588Z * [new tag] v2.9.0-rc11 -> v2.9.0-rc11 2025-12-04T08:27:07.4250442Z * [new tag] v2.9.0-rc2 -> v2.9.0-rc2 2025-12-04T08:27:07.4250572Z * [new tag] v2.9.0-rc3 -> v2.9.0-rc3 2025-12-04T08:27:07.4251006Z * [new tag] v2.9.0-rc4 -> v2.9.0-rc4 2025-12-04T08:27:07.4251193Z * [new tag] v2.9.0-rc5 -> v2.9.0-rc5 2025-12-04T08:27:07.4251519Z * [new tag] v2.9.0-rc6 -> v2.9.0-rc6 2025-12-04T08:27:07.4251693Z * [new tag] v2.9.0-rc7 -> v2.9.0-rc7 2025-12-04T08:27:07.4251870Z * [new tag] v2.9.0-rc8 -> v2.9.0-rc8 2025-12-04T08:27:07.4252307Z * [new tag] v2.9.0-rc9 -> v2.9.0-rc9 2025-12-04T08:27:07.4252446Z * [new tag] v2.9.1 -> v2.9.1 2025-12-04T08:27:07.4255100Z * [new tag] v2.9.1-rc1 -> v2.9.1-rc1 2025-12-04T08:27:07.4255223Z * [new tag] v2.9.1-rc2 -> v2.9.1-rc2 2025-12-04T08:27:07.4255358Z * [new tag] viable/strict/1759343184 -> viable/strict/1759343184 2025-12-04T08:27:07.4255484Z * [new tag] viable/strict/1759346540 -> viable/strict/1759346540 2025-12-04T08:27:07.4255897Z * [new tag] viable/strict/1759348181 -> viable/strict/1759348181 2025-12-04T08:27:07.4256723Z * [new tag] viable/strict/1759350324 -> viable/strict/1759350324 2025-12-04T08:27:07.4257657Z * [new tag] viable/strict/1759351793 -> viable/strict/1759351793 2025-12-04T08:27:07.4257949Z * [new tag] viable/strict/1759353844 -> viable/strict/1759353844 2025-12-04T08:27:07.4260310Z * [new tag] viable/strict/1759355374 -> viable/strict/1759355374 2025-12-04T08:27:07.4260460Z * [new tag] viable/strict/1759357472 -> viable/strict/1759357472 2025-12-04T08:27:07.4260577Z * [new tag] viable/strict/1759361002 -> viable/strict/1759361002 2025-12-04T08:27:07.4260699Z * [new tag] viable/strict/1759362585 -> viable/strict/1759362585 2025-12-04T08:27:07.4261164Z * [new tag] viable/strict/1759365359 -> viable/strict/1759365359 2025-12-04T08:27:07.4261863Z * [new tag] viable/strict/1759370089 -> viable/strict/1759370089 2025-12-04T08:27:07.4262692Z * [new tag] viable/strict/1759377554 -> viable/strict/1759377554 2025-12-04T08:27:07.4262901Z * [new tag] viable/strict/1759379133 -> viable/strict/1759379133 2025-12-04T08:27:07.4263518Z * [new tag] viable/strict/1759389871 -> viable/strict/1759389871 2025-12-04T08:27:07.4263997Z * [new tag] viable/strict/1759393562 -> viable/strict/1759393562 2025-12-04T08:27:07.4264824Z * [new tag] viable/strict/1759395076 -> viable/strict/1759395076 2025-12-04T08:27:07.4265106Z * [new tag] viable/strict/1759398579 -> viable/strict/1759398579 2025-12-04T08:27:07.4266267Z * [new tag] viable/strict/1759404142 -> viable/strict/1759404142 2025-12-04T08:27:07.4266398Z * [new tag] viable/strict/1759405773 -> viable/strict/1759405773 2025-12-04T08:27:07.4266953Z * [new tag] viable/strict/1759408041 -> viable/strict/1759408041 2025-12-04T08:27:07.4267487Z * [new tag] viable/strict/1759411593 -> viable/strict/1759411593 2025-12-04T08:27:07.4268334Z * [new tag] viable/strict/1759427395 -> viable/strict/1759427395 2025-12-04T08:27:07.4268536Z * [new tag] viable/strict/1759434582 -> viable/strict/1759434582 2025-12-04T08:27:07.4269368Z * [new tag] viable/strict/1759436720 -> viable/strict/1759436720 2025-12-04T08:27:07.4269637Z * [new tag] viable/strict/1759440219 -> viable/strict/1759440219 2025-12-04T08:27:07.4272418Z * [new tag] viable/strict/1759441948 -> viable/strict/1759441948 2025-12-04T08:27:07.4272569Z * [new tag] viable/strict/1759443860 -> viable/strict/1759443860 2025-12-04T08:27:07.4272695Z * [new tag] viable/strict/1759445377 -> viable/strict/1759445377 2025-12-04T08:27:07.4272817Z * [new tag] viable/strict/1759447415 -> viable/strict/1759447415 2025-12-04T08:27:07.4273092Z * [new tag] viable/strict/1759451750 -> viable/strict/1759451750 2025-12-04T08:27:07.4273262Z * [new tag] viable/strict/1759453910 -> viable/strict/1759453910 2025-12-04T08:27:07.4274053Z * [new tag] viable/strict/1759456483 -> viable/strict/1759456483 2025-12-04T08:27:07.4274306Z * [new tag] viable/strict/1759459279 -> viable/strict/1759459279 2025-12-04T08:27:07.4275222Z * [new tag] viable/strict/1759460742 -> viable/strict/1759460742 2025-12-04T08:27:07.4275527Z * [new tag] viable/strict/1759462025 -> viable/strict/1759462025 2025-12-04T08:27:07.4276392Z * [new tag] viable/strict/1759469086 -> viable/strict/1759469086 2025-12-04T08:27:07.4276589Z * [new tag] viable/strict/1759470581 -> viable/strict/1759470581 2025-12-04T08:27:07.4277481Z * [new tag] viable/strict/1759472786 -> viable/strict/1759472786 2025-12-04T08:27:07.4277717Z * [new tag] viable/strict/1759476294 -> viable/strict/1759476294 2025-12-04T08:27:07.4278642Z * [new tag] viable/strict/1759479963 -> viable/strict/1759479963 2025-12-04T08:27:07.4278989Z * [new tag] viable/strict/1759492177 -> viable/strict/1759492177 2025-12-04T08:27:07.4282224Z * [new tag] viable/strict/1759519278 -> viable/strict/1759519278 2025-12-04T08:27:07.4282391Z * [new tag] viable/strict/1759524580 -> viable/strict/1759524580 2025-12-04T08:27:07.4282516Z * [new tag] viable/strict/1759528193 -> viable/strict/1759528193 2025-12-04T08:27:07.4282647Z * [new tag] viable/strict/1759533797 -> viable/strict/1759533797 2025-12-04T08:27:07.4282768Z * [new tag] viable/strict/1759542780 -> viable/strict/1759542780 2025-12-04T08:27:07.4282893Z * [new tag] viable/strict/1759549779 -> viable/strict/1759549779 2025-12-04T08:27:07.4283334Z * [new tag] viable/strict/1759555455 -> viable/strict/1759555455 2025-12-04T08:27:07.4284221Z * [new tag] viable/strict/1759559176 -> viable/strict/1759559176 2025-12-04T08:27:07.4284409Z * [new tag] viable/strict/1759560629 -> viable/strict/1759560629 2025-12-04T08:27:07.4287042Z * [new tag] viable/strict/1759569848 -> viable/strict/1759569848 2025-12-04T08:27:07.4287194Z * [new tag] viable/strict/1759571382 -> viable/strict/1759571382 2025-12-04T08:27:07.4287323Z * [new tag] viable/strict/1759573474 -> viable/strict/1759573474 2025-12-04T08:27:07.4287443Z * [new tag] viable/strict/1759618187 -> viable/strict/1759618187 2025-12-04T08:27:07.4287566Z * [new tag] viable/strict/1759626742 -> viable/strict/1759626742 2025-12-04T08:27:07.4288144Z * [new tag] viable/strict/1759632427 -> viable/strict/1759632427 2025-12-04T08:27:07.4288691Z * [new tag] viable/strict/1759634971 -> viable/strict/1759634971 2025-12-04T08:27:07.4289409Z * [new tag] viable/strict/1759661382 -> viable/strict/1759661382 2025-12-04T08:27:07.4289801Z * [new tag] viable/strict/1759663294 -> viable/strict/1759663294 2025-12-04T08:27:07.4290297Z * [new tag] viable/strict/1759708178 -> viable/strict/1759708178 2025-12-04T08:27:07.4291014Z * [new tag] viable/strict/1759715695 -> viable/strict/1759715695 2025-12-04T08:27:07.4291645Z * [new tag] viable/strict/1759728293 -> viable/strict/1759728293 2025-12-04T08:27:07.4291877Z * [new tag] viable/strict/1759735513 -> viable/strict/1759735513 2025-12-04T08:27:07.4292744Z * [new tag] viable/strict/1759739177 -> viable/strict/1759739177 2025-12-04T08:27:07.4292954Z * [new tag] viable/strict/1759758635 -> viable/strict/1759758635 2025-12-04T08:27:07.4296513Z * [new tag] viable/strict/1759765784 -> viable/strict/1759765784 2025-12-04T08:27:07.4296680Z * [new tag] viable/strict/1759767948 -> viable/strict/1759767948 2025-12-04T08:27:07.4296826Z * [new tag] viable/strict/1759771461 -> viable/strict/1759771461 2025-12-04T08:27:07.4296946Z * [new tag] viable/strict/1759776706 -> viable/strict/1759776706 2025-12-04T08:27:07.4297062Z * [new tag] viable/strict/1759782317 -> viable/strict/1759782317 2025-12-04T08:27:07.4297188Z * [new tag] viable/strict/1759783777 -> viable/strict/1759783777 2025-12-04T08:27:07.4297563Z * [new tag] viable/strict/1759785815 -> viable/strict/1759785815 2025-12-04T08:27:07.4297932Z * [new tag] viable/strict/1759789459 -> viable/strict/1759789459 2025-12-04T08:27:07.4298673Z * [new tag] viable/strict/1759790974 -> viable/strict/1759790974 2025-12-04T08:27:07.4298969Z * [new tag] viable/strict/1759794583 -> viable/strict/1759794583 2025-12-04T08:27:07.4299770Z * [new tag] viable/strict/1759797408 -> viable/strict/1759797408 2025-12-04T08:27:07.4300020Z * [new tag] viable/strict/1759799518 -> viable/strict/1759799518 2025-12-04T08:27:07.4303403Z * [new tag] viable/strict/1759804909 -> viable/strict/1759804909 2025-12-04T08:27:07.4303571Z * [new tag] viable/strict/1759807643 -> viable/strict/1759807643 2025-12-04T08:27:07.4303688Z * [new tag] viable/strict/1759809089 -> viable/strict/1759809089 2025-12-04T08:27:07.4303812Z * [new tag] viable/strict/1759811145 -> viable/strict/1759811145 2025-12-04T08:27:07.4303929Z * [new tag] viable/strict/1759812581 -> viable/strict/1759812581 2025-12-04T08:27:07.4304045Z * [new tag] viable/strict/1759814683 -> viable/strict/1759814683 2025-12-04T08:27:07.4304190Z * [new tag] viable/strict/1759821889 -> viable/strict/1759821889 2025-12-04T08:27:07.4304432Z * [new tag] viable/strict/1759823376 -> viable/strict/1759823376 2025-12-04T08:27:07.4305069Z * [new tag] viable/strict/1759827107 -> viable/strict/1759827107 2025-12-04T08:27:07.4305509Z * [new tag] viable/strict/1759830577 -> viable/strict/1759830577 2025-12-04T08:27:07.4306178Z * [new tag] viable/strict/1759832720 -> viable/strict/1759832720 2025-12-04T08:27:07.4306519Z * [new tag] viable/strict/1759842063 -> viable/strict/1759842063 2025-12-04T08:27:07.4307782Z * [new tag] viable/strict/1759847121 -> viable/strict/1759847121 2025-12-04T08:27:07.4308004Z * [new tag] viable/strict/1759850721 -> viable/strict/1759850721 2025-12-04T08:27:07.4308640Z * [new tag] viable/strict/1759857870 -> viable/strict/1759857870 2025-12-04T08:27:07.4309025Z * [new tag] viable/strict/1759863143 -> viable/strict/1759863143 2025-12-04T08:27:07.4312235Z * [new tag] viable/strict/1759875874 -> viable/strict/1759875874 2025-12-04T08:27:07.4312401Z * [new tag] viable/strict/1759877385 -> viable/strict/1759877385 2025-12-04T08:27:07.4312535Z * [new tag] viable/strict/1759883801 -> viable/strict/1759883801 2025-12-04T08:27:07.4312653Z * [new tag] viable/strict/1759885922 -> viable/strict/1759885922 2025-12-04T08:27:07.4312770Z * [new tag] viable/strict/1759888488 -> viable/strict/1759888488 2025-12-04T08:27:07.4312896Z * [new tag] viable/strict/1759895471 -> viable/strict/1759895471 2025-12-04T08:27:07.4313023Z * [new tag] viable/strict/1759904803 -> viable/strict/1759904803 2025-12-04T08:27:07.4313519Z * [new tag] viable/strict/1759908300 -> viable/strict/1759908300 2025-12-04T08:27:07.4314485Z * [new tag] viable/strict/1759915520 -> viable/strict/1759915520 2025-12-04T08:27:07.4314620Z * [new tag] viable/strict/1759916978 -> viable/strict/1759916978 2025-12-04T08:27:07.4315120Z * [new tag] viable/strict/1759930024 -> viable/strict/1759930024 2025-12-04T08:27:07.4315917Z * [new tag] viable/strict/1759948122 -> viable/strict/1759948122 2025-12-04T08:27:07.4316269Z * [new tag] viable/strict/1759952983 -> viable/strict/1759952983 2025-12-04T08:27:07.4317212Z * [new tag] viable/strict/1759955121 -> viable/strict/1759955121 2025-12-04T08:27:07.4317426Z * [new tag] viable/strict/1759962298 -> viable/strict/1759962298 2025-12-04T08:27:07.4318339Z * [new tag] viable/strict/1759965837 -> viable/strict/1759965837 2025-12-04T08:27:07.4319079Z * [new tag] viable/strict/1759970213 -> viable/strict/1759970213 2025-12-04T08:27:07.4319501Z * [new tag] viable/strict/1759974894 -> viable/strict/1759974894 2025-12-04T08:27:07.4324040Z * [new tag] viable/strict/1759977763 -> viable/strict/1759977763 2025-12-04T08:27:07.4324216Z * [new tag] viable/strict/1759979241 -> viable/strict/1759979241 2025-12-04T08:27:07.4324468Z * [new tag] viable/strict/1759985417 -> viable/strict/1759985417 2025-12-04T08:27:07.4324610Z * [new tag] viable/strict/1759987490 -> viable/strict/1759987490 2025-12-04T08:27:07.4324752Z * [new tag] viable/strict/1759996180 -> viable/strict/1759996180 2025-12-04T08:27:07.4324883Z * [new tag] viable/strict/1760065682 -> viable/strict/1760065682 2025-12-04T08:27:07.4325025Z * [new tag] viable/strict/1760066894 -> viable/strict/1760066894 2025-12-04T08:27:07.4325156Z * [new tag] viable/strict/1760070345 -> viable/strict/1760070345 2025-12-04T08:27:07.4325377Z * [new tag] viable/strict/1760089782 -> viable/strict/1760089782 2025-12-04T08:27:07.4325611Z * [new tag] viable/strict/1760091921 -> viable/strict/1760091921 2025-12-04T08:27:07.4328181Z * [new tag] viable/strict/1760127924 -> viable/strict/1760127924 2025-12-04T08:27:07.4328330Z * [new tag] viable/strict/1760129489 -> viable/strict/1760129489 2025-12-04T08:27:07.4328451Z * [new tag] viable/strict/1760132980 -> viable/strict/1760132980 2025-12-04T08:27:07.4328575Z * [new tag] viable/strict/1760135060 -> viable/strict/1760135060 2025-12-04T08:27:07.4328726Z * [new tag] viable/strict/1760215782 -> viable/strict/1760215782 2025-12-04T08:27:07.4329620Z * [new tag] viable/strict/1760273849 -> viable/strict/1760273849 2025-12-04T08:27:07.4329931Z * [new tag] viable/strict/1760275517 -> viable/strict/1760275517 2025-12-04T08:27:07.4330320Z * [new tag] viable/strict/1760276979 -> viable/strict/1760276979 2025-12-04T08:27:07.4333921Z * [new tag] viable/strict/1760279007 -> viable/strict/1760279007 2025-12-04T08:27:07.4334117Z * [new tag] viable/strict/1760286328 -> viable/strict/1760286328 2025-12-04T08:27:07.4334239Z * [new tag] viable/strict/1760493304 -> viable/strict/1760493304 2025-12-04T08:27:07.4334360Z * [new tag] viable/strict/1760496298 -> viable/strict/1760496298 2025-12-04T08:27:07.4334488Z * [new tag] viable/strict/1760518396 -> viable/strict/1760518396 2025-12-04T08:27:07.4334605Z * [new tag] viable/strict/1760534864 -> viable/strict/1760534864 2025-12-04T08:27:07.4334726Z * [new tag] viable/strict/1760549062 -> viable/strict/1760549062 2025-12-04T08:27:07.4335505Z * [new tag] viable/strict/1760552799 -> viable/strict/1760552799 2025-12-04T08:27:07.4336253Z * [new tag] viable/strict/1760554355 -> viable/strict/1760554355 2025-12-04T08:27:07.4337105Z * [new tag] viable/strict/1760556275 -> viable/strict/1760556275 2025-12-04T08:27:07.4337803Z * [new tag] viable/strict/1760564979 -> viable/strict/1760564979 2025-12-04T08:27:07.4338426Z * [new tag] viable/strict/1760567049 -> viable/strict/1760567049 2025-12-04T08:27:07.4341064Z * [new tag] viable/strict/1760568585 -> viable/strict/1760568585 2025-12-04T08:27:07.4341219Z * [new tag] viable/strict/1760570630 -> viable/strict/1760570630 2025-12-04T08:27:07.4341337Z * [new tag] viable/strict/1760572180 -> viable/strict/1760572180 2025-12-04T08:27:07.4341450Z * [new tag] viable/strict/1760575094 -> viable/strict/1760575094 2025-12-04T08:27:07.4342407Z * [new tag] viable/strict/1760579709 -> viable/strict/1760579709 2025-12-04T08:27:07.4346536Z * [new tag] viable/strict/1760582614 -> viable/strict/1760582614 2025-12-04T08:27:07.4350324Z * [new tag] viable/strict/1760586815 -> viable/strict/1760586815 2025-12-04T08:27:07.4352612Z * [new tag] viable/strict/1760588829 -> viable/strict/1760588829 2025-12-04T08:27:07.4352979Z * [new tag] viable/strict/1760590200 -> viable/strict/1760590200 2025-12-04T08:27:07.4353290Z * [new tag] viable/strict/1760592311 -> viable/strict/1760592311 2025-12-04T08:27:07.4353606Z * [new tag] viable/strict/1760619733 -> viable/strict/1760619733 2025-12-04T08:27:07.4353921Z * [new tag] viable/strict/1760628335 -> viable/strict/1760628335 2025-12-04T08:27:07.4354167Z * [new tag] viable/strict/1760635490 -> viable/strict/1760635490 2025-12-04T08:27:07.4354440Z * [new tag] viable/strict/1760640743 -> viable/strict/1760640743 2025-12-04T08:27:07.4354803Z * [new tag] viable/strict/1760642528 -> viable/strict/1760642528 2025-12-04T08:27:07.4355034Z * [new tag] viable/strict/1760646330 -> viable/strict/1760646330 2025-12-04T08:27:07.4355387Z * [new tag] viable/strict/1760666101 -> viable/strict/1760666101 2025-12-04T08:27:07.4355584Z * [new tag] viable/strict/1760668990 -> viable/strict/1760668990 2025-12-04T08:27:07.4355814Z * [new tag] viable/strict/1760670600 -> viable/strict/1760670600 2025-12-04T08:27:07.4356046Z * [new tag] viable/strict/1760671704 -> viable/strict/1760671704 2025-12-04T08:27:07.4356256Z * [new tag] viable/strict/1760673121 -> viable/strict/1760673121 2025-12-04T08:27:07.4356667Z * [new tag] viable/strict/1760675352 -> viable/strict/1760675352 2025-12-04T08:27:07.4356904Z * [new tag] viable/strict/1760696731 -> viable/strict/1760696731 2025-12-04T08:27:07.4357094Z * [new tag] viable/strict/1760723515 -> viable/strict/1760723515 2025-12-04T08:27:07.4357315Z * [new tag] viable/strict/1760727234 -> viable/strict/1760727234 2025-12-04T08:27:07.4357765Z * [new tag] viable/strict/1760730578 -> viable/strict/1760730578 2025-12-04T08:27:07.4359126Z * [new tag] viable/strict/1760732726 -> viable/strict/1760732726 2025-12-04T08:27:07.4363007Z * [new tag] viable/strict/1760734180 -> viable/strict/1760734180 2025-12-04T08:27:07.4363322Z * [new tag] viable/strict/1760736251 -> viable/strict/1760736251 2025-12-04T08:27:07.4369219Z * [new tag] viable/strict/1760737772 -> viable/strict/1760737772 2025-12-04T08:27:07.4374283Z * [new tag] viable/strict/1760758005 -> viable/strict/1760758005 2025-12-04T08:27:07.4378778Z * [new tag] viable/strict/1760761532 -> viable/strict/1760761532 2025-12-04T08:27:07.4379024Z * [new tag] viable/strict/1760802581 -> viable/strict/1760802581 2025-12-04T08:27:07.4379206Z * [new tag] viable/strict/1760827772 -> viable/strict/1760827772 2025-12-04T08:27:07.4379379Z * [new tag] viable/strict/1760834524 -> viable/strict/1760834524 2025-12-04T08:27:07.4379499Z * [new tag] viable/strict/1760845009 -> viable/strict/1760845009 2025-12-04T08:27:07.4379617Z * [new tag] viable/strict/1760876836 -> viable/strict/1760876836 2025-12-04T08:27:07.4379745Z * [new tag] viable/strict/1760880329 -> viable/strict/1760880329 2025-12-04T08:27:07.4379865Z * [new tag] viable/strict/1760888987 -> viable/strict/1760888987 2025-12-04T08:27:07.4379993Z * [new tag] viable/strict/1760912664 -> viable/strict/1760912664 2025-12-04T08:27:07.4380115Z * [new tag] viable/strict/1760925321 -> viable/strict/1760925321 2025-12-04T08:27:07.4380234Z * [new tag] viable/strict/1760931488 -> viable/strict/1760931488 2025-12-04T08:27:07.4380359Z * [new tag] viable/strict/1760932693 -> viable/strict/1760932693 2025-12-04T08:27:07.4380480Z * [new tag] viable/strict/1761004184 -> viable/strict/1761004184 2025-12-04T08:27:07.4380606Z * [new tag] viable/strict/1761014748 -> viable/strict/1761014748 2025-12-04T08:27:07.4380747Z * [new tag] viable/strict/1761017491 -> viable/strict/1761017491 2025-12-04T08:27:07.4380915Z * [new tag] viable/strict/1761018806 -> viable/strict/1761018806 2025-12-04T08:27:07.4381101Z * [new tag] viable/strict/1761020754 -> viable/strict/1761020754 2025-12-04T08:27:07.4381250Z * [new tag] viable/strict/1761024303 -> viable/strict/1761024303 2025-12-04T08:27:07.4381373Z * [new tag] viable/strict/1761029582 -> viable/strict/1761029582 2025-12-04T08:27:07.4381499Z * [new tag] viable/strict/1761031535 -> viable/strict/1761031535 2025-12-04T08:27:07.4381622Z * [new tag] viable/strict/1761035196 -> viable/strict/1761035196 2025-12-04T08:27:07.4381870Z * [new tag] viable/strict/1761045825 -> viable/strict/1761045825 2025-12-04T08:27:07.4382010Z * [new tag] viable/strict/1761054796 -> viable/strict/1761054796 2025-12-04T08:27:07.4382129Z * [new tag] viable/strict/1761060314 -> viable/strict/1761060314 2025-12-04T08:27:07.4382253Z * [new tag] viable/strict/1761071198 -> viable/strict/1761071198 2025-12-04T08:27:07.4382389Z * [new tag] viable/strict/1761074628 -> viable/strict/1761074628 2025-12-04T08:27:07.4382570Z * [new tag] viable/strict/1761078351 -> viable/strict/1761078351 2025-12-04T08:27:07.4382752Z * [new tag] viable/strict/1761079822 -> viable/strict/1761079822 2025-12-04T08:27:07.4382913Z * [new tag] viable/strict/1761081873 -> viable/strict/1761081873 2025-12-04T08:27:07.4383042Z * [new tag] viable/strict/1761083392 -> viable/strict/1761083392 2025-12-04T08:27:07.4383162Z * [new tag] viable/strict/1761085465 -> viable/strict/1761085465 2025-12-04T08:27:07.4383278Z * [new tag] viable/strict/1761089099 -> viable/strict/1761089099 2025-12-04T08:27:07.4383403Z * [new tag] viable/strict/1761095535 -> viable/strict/1761095535 2025-12-04T08:27:07.4383521Z * [new tag] viable/strict/1761098119 -> viable/strict/1761098119 2025-12-04T08:27:07.4384005Z * [new tag] viable/strict/1761101330 -> viable/strict/1761101330 2025-12-04T08:27:07.4385084Z * [new tag] viable/strict/1761114425 -> viable/strict/1761114425 2025-12-04T08:27:07.4385664Z * [new tag] viable/strict/1761116036 -> viable/strict/1761116036 2025-12-04T08:27:07.4386342Z * [new tag] viable/strict/1761119379 -> viable/strict/1761119379 2025-12-04T08:27:07.4387374Z * [new tag] viable/strict/1761121601 -> viable/strict/1761121601 2025-12-04T08:27:07.4387778Z * [new tag] viable/strict/1761123234 -> viable/strict/1761123234 2025-12-04T08:27:07.4391178Z * [new tag] viable/strict/1761126621 -> viable/strict/1761126621 2025-12-04T08:27:07.4391508Z * [new tag] viable/strict/1761132259 -> viable/strict/1761132259 2025-12-04T08:27:07.4395118Z * [new tag] viable/strict/1761146746 -> viable/strict/1761146746 2025-12-04T08:27:07.4395341Z * [new tag] viable/strict/1761164752 -> viable/strict/1761164752 2025-12-04T08:27:07.4395770Z * [new tag] viable/strict/1761166198 -> viable/strict/1761166198 2025-12-04T08:27:07.4395939Z * [new tag] viable/strict/1761175424 -> viable/strict/1761175424 2025-12-04T08:27:07.4396076Z * [new tag] viable/strict/1761176983 -> viable/strict/1761176983 2025-12-04T08:27:07.4396197Z * [new tag] viable/strict/1761179891 -> viable/strict/1761179891 2025-12-04T08:27:07.4396330Z * [new tag] viable/strict/1761181930 -> viable/strict/1761181930 2025-12-04T08:27:07.4396448Z * [new tag] viable/strict/1761184516 -> viable/strict/1761184516 2025-12-04T08:27:07.4396566Z * [new tag] viable/strict/1761190179 -> viable/strict/1761190179 2025-12-04T08:27:07.4396691Z * [new tag] viable/strict/1761193558 -> viable/strict/1761193558 2025-12-04T08:27:07.4397145Z * [new tag] viable/strict/1761207990 -> viable/strict/1761207990 2025-12-04T08:27:07.4397377Z * [new tag] viable/strict/1761229539 -> viable/strict/1761229539 2025-12-04T08:27:07.4399246Z * [new tag] viable/strict/1761244031 -> viable/strict/1761244031 2025-12-04T08:27:07.4399485Z * [new tag] viable/strict/1761248986 -> viable/strict/1761248986 2025-12-04T08:27:07.4400292Z * [new tag] viable/strict/1761259791 -> viable/strict/1761259791 2025-12-04T08:27:07.4401046Z * [new tag] viable/strict/1761266139 -> viable/strict/1761266139 2025-12-04T08:27:07.4403930Z * [new tag] viable/strict/1761268316 -> viable/strict/1761268316 2025-12-04T08:27:07.4404255Z * [new tag] viable/strict/1761273805 -> viable/strict/1761273805 2025-12-04T08:27:07.4404517Z * [new tag] viable/strict/1761275261 -> viable/strict/1761275261 2025-12-04T08:27:07.4404836Z * [new tag] viable/strict/1761277913 -> viable/strict/1761277913 2025-12-04T08:27:07.4412156Z * [new tag] viable/strict/1761290701 -> viable/strict/1761290701 2025-12-04T08:27:07.4412335Z * [new tag] viable/strict/1761294396 -> viable/strict/1761294396 2025-12-04T08:27:07.4412469Z * [new tag] viable/strict/1761303047 -> viable/strict/1761303047 2025-12-04T08:27:07.4412606Z * [new tag] viable/strict/1761335388 -> viable/strict/1761335388 2025-12-04T08:27:07.4412748Z * [new tag] viable/strict/1761337551 -> viable/strict/1761337551 2025-12-04T08:27:07.4412866Z * [new tag] viable/strict/1761339007 -> viable/strict/1761339007 2025-12-04T08:27:07.4412989Z * [new tag] viable/strict/1761341050 -> viable/strict/1761341050 2025-12-04T08:27:07.4413103Z * [new tag] viable/strict/1761346188 -> viable/strict/1761346188 2025-12-04T08:27:07.4413225Z * [new tag] viable/strict/1761349792 -> viable/strict/1761349792 2025-12-04T08:27:07.4413341Z * [new tag] viable/strict/1761352620 -> viable/strict/1761352620 2025-12-04T08:27:07.4413613Z * [new tag] viable/strict/1761354730 -> viable/strict/1761354730 2025-12-04T08:27:07.4413742Z * [new tag] viable/strict/1761357298 -> viable/strict/1761357298 2025-12-04T08:27:07.4413864Z * [new tag] viable/strict/1761360201 -> viable/strict/1761360201 2025-12-04T08:27:07.4413987Z * [new tag] viable/strict/1761361753 -> viable/strict/1761361753 2025-12-04T08:27:07.4414110Z * [new tag] viable/strict/1761364351 -> viable/strict/1761364351 2025-12-04T08:27:07.4418168Z * [new tag] viable/strict/1761366338 -> viable/strict/1761366338 2025-12-04T08:27:07.4418331Z * [new tag] viable/strict/1761367802 -> viable/strict/1761367802 2025-12-04T08:27:07.4418465Z * [new tag] viable/strict/1761369889 -> viable/strict/1761369889 2025-12-04T08:27:07.4418599Z * [new tag] viable/strict/1761371385 -> viable/strict/1761371385 2025-12-04T08:27:07.4418737Z * [new tag] viable/strict/1761373581 -> viable/strict/1761373581 2025-12-04T08:27:07.4418863Z * [new tag] viable/strict/1761375054 -> viable/strict/1761375054 2025-12-04T08:27:07.4418999Z * [new tag] viable/strict/1761421785 -> viable/strict/1761421785 2025-12-04T08:27:07.4419128Z * [new tag] viable/strict/1761434614 -> viable/strict/1761434614 2025-12-04T08:27:07.4419258Z * [new tag] viable/strict/1761439254 -> viable/strict/1761439254 2025-12-04T08:27:07.4419380Z * [new tag] viable/strict/1761454187 -> viable/strict/1761454187 2025-12-04T08:27:07.4419749Z * [new tag] viable/strict/1761459991 -> viable/strict/1761459991 2025-12-04T08:27:07.4421233Z * [new tag] viable/strict/1761470668 -> viable/strict/1761470668 2025-12-04T08:27:07.4427404Z * [new tag] viable/strict/1761472188 -> viable/strict/1761472188 2025-12-04T08:27:07.4427574Z * [new tag] viable/strict/1761503178 -> viable/strict/1761503178 2025-12-04T08:27:07.4427713Z * [new tag] viable/strict/1761517492 -> viable/strict/1761517492 2025-12-04T08:27:07.4427840Z * [new tag] viable/strict/1761518981 -> viable/strict/1761518981 2025-12-04T08:27:07.4428164Z * [new tag] viable/strict/1761533609 -> viable/strict/1761533609 2025-12-04T08:27:07.4428302Z * [new tag] viable/strict/1761546438 -> viable/strict/1761546438 2025-12-04T08:27:07.4428755Z * [new tag] viable/strict/1761548133 -> viable/strict/1761548133 2025-12-04T08:27:07.4429257Z * [new tag] viable/strict/1761555186 -> viable/strict/1761555186 2025-12-04T08:27:07.4429989Z * [new tag] viable/strict/1761557178 -> viable/strict/1761557178 2025-12-04T08:27:07.4433541Z * [new tag] viable/strict/1761560772 -> viable/strict/1761560772 2025-12-04T08:27:07.4433710Z * [new tag] viable/strict/1761562266 -> viable/strict/1761562266 2025-12-04T08:27:07.4433840Z * [new tag] viable/strict/1761564260 -> viable/strict/1761564260 2025-12-04T08:27:07.4433958Z * [new tag] viable/strict/1761568072 -> viable/strict/1761568072 2025-12-04T08:27:07.4434079Z * [new tag] viable/strict/1761571683 -> viable/strict/1761571683 2025-12-04T08:27:07.4434206Z * [new tag] viable/strict/1761580199 -> viable/strict/1761580199 2025-12-04T08:27:07.4435111Z * [new tag] viable/strict/1761587383 -> viable/strict/1761587383 2025-12-04T08:27:07.4435257Z * [new tag] viable/strict/1761591165 -> viable/strict/1761591165 2025-12-04T08:27:07.4440630Z * [new tag] viable/strict/1761594575 -> viable/strict/1761594575 2025-12-04T08:27:07.4440996Z * [new tag] viable/strict/1761596710 -> viable/strict/1761596710 2025-12-04T08:27:07.4446776Z * [new tag] viable/strict/1761598189 -> viable/strict/1761598189 2025-12-04T08:27:07.4451780Z * [new tag] viable/strict/1761600254 -> viable/strict/1761600254 2025-12-04T08:27:07.4454380Z * [new tag] viable/strict/1761603879 -> viable/strict/1761603879 2025-12-04T08:27:07.4454817Z * [new tag] viable/strict/1761605429 -> viable/strict/1761605429 2025-12-04T08:27:07.4455092Z * [new tag] viable/strict/1761607468 -> viable/strict/1761607468 2025-12-04T08:27:07.4455311Z * [new tag] viable/strict/1761608983 -> viable/strict/1761608983 2025-12-04T08:27:07.4455520Z * [new tag] viable/strict/1761611846 -> viable/strict/1761611846 2025-12-04T08:27:07.4455750Z * [new tag] viable/strict/1761613922 -> viable/strict/1761613922 2025-12-04T08:27:07.4455969Z * [new tag] viable/strict/1761616504 -> viable/strict/1761616504 2025-12-04T08:27:07.4456205Z * [new tag] viable/strict/1761619599 -> viable/strict/1761619599 2025-12-04T08:27:07.4456407Z * [new tag] viable/strict/1761686693 -> viable/strict/1761686693 2025-12-04T08:27:07.4456632Z * [new tag] viable/strict/1761688179 -> viable/strict/1761688179 2025-12-04T08:27:07.4456894Z * [new tag] viable/strict/1761691973 -> viable/strict/1761691973 2025-12-04T08:27:07.4457110Z * [new tag] viable/strict/1761693884 -> viable/strict/1761693884 2025-12-04T08:27:07.4457305Z * [new tag] viable/strict/1761695389 -> viable/strict/1761695389 2025-12-04T08:27:07.4457527Z * [new tag] viable/strict/1761698408 -> viable/strict/1761698408 2025-12-04T08:27:07.4457751Z * [new tag] viable/strict/1761702931 -> viable/strict/1761702931 2025-12-04T08:27:07.4457975Z * [new tag] viable/strict/1761706307 -> viable/strict/1761706307 2025-12-04T08:27:07.4458170Z * [new tag] viable/strict/1761709065 -> viable/strict/1761709065 2025-12-04T08:27:07.4458371Z * [new tag] viable/strict/1761710285 -> viable/strict/1761710285 2025-12-04T08:27:07.4458582Z * [new tag] viable/strict/1761711983 -> viable/strict/1761711983 2025-12-04T08:27:07.4458878Z * [new tag] viable/strict/1761713514 -> viable/strict/1761713514 2025-12-04T08:27:07.4459102Z * [new tag] viable/strict/1761715523 -> viable/strict/1761715523 2025-12-04T08:27:07.4459331Z * [new tag] viable/strict/1761727973 -> viable/strict/1761727973 2025-12-04T08:27:07.4459543Z * [new tag] viable/strict/1761751558 -> viable/strict/1761751558 2025-12-04T08:27:07.4459887Z * [new tag] viable/strict/1761755187 -> viable/strict/1761755187 2025-12-04T08:27:07.4465670Z * [new tag] viable/strict/1761756826 -> viable/strict/1761756826 2025-12-04T08:27:07.4470468Z * [new tag] viable/strict/1761769551 -> viable/strict/1761769551 2025-12-04T08:27:07.4474737Z * [new tag] viable/strict/1761771032 -> viable/strict/1761771032 2025-12-04T08:27:07.4476680Z * [new tag] viable/strict/1761773101 -> viable/strict/1761773101 2025-12-04T08:27:07.4476839Z * [new tag] viable/strict/1761781792 -> viable/strict/1761781792 2025-12-04T08:27:07.4477048Z * [new tag] viable/strict/1761784788 -> viable/strict/1761784788 2025-12-04T08:27:07.4477219Z * [new tag] viable/strict/1761786740 -> viable/strict/1761786740 2025-12-04T08:27:07.4477340Z * [new tag] viable/strict/1761789332 -> viable/strict/1761789332 2025-12-04T08:27:07.4477463Z * [new tag] viable/strict/1761792569 -> viable/strict/1761792569 2025-12-04T08:27:07.4477589Z * [new tag] viable/strict/1761795289 -> viable/strict/1761795289 2025-12-04T08:27:07.4477870Z * [new tag] viable/strict/1761798345 -> viable/strict/1761798345 2025-12-04T08:27:07.4478002Z * [new tag] viable/strict/1761799827 -> viable/strict/1761799827 2025-12-04T08:27:07.4478282Z * [new tag] viable/strict/1761805604 -> viable/strict/1761805604 2025-12-04T08:27:07.4478430Z * [new tag] viable/strict/1761807202 -> viable/strict/1761807202 2025-12-04T08:27:07.4478572Z * [new tag] viable/strict/1761809094 -> viable/strict/1761809094 2025-12-04T08:27:07.4478707Z * [new tag] viable/strict/1761810576 -> viable/strict/1761810576 2025-12-04T08:27:07.4478847Z * [new tag] viable/strict/1761812771 -> viable/strict/1761812771 2025-12-04T08:27:07.4478982Z * [new tag] viable/strict/1761814363 -> viable/strict/1761814363 2025-12-04T08:27:07.4479117Z * [new tag] viable/strict/1761857410 -> viable/strict/1761857410 2025-12-04T08:27:07.4479261Z * [new tag] viable/strict/1761860985 -> viable/strict/1761860985 2025-12-04T08:27:07.4479397Z * [new tag] viable/strict/1761863094 -> viable/strict/1761863094 2025-12-04T08:27:07.4479561Z * [new tag] viable/strict/1761864590 -> viable/strict/1761864590 2025-12-04T08:27:07.4479719Z * [new tag] viable/strict/1761866675 -> viable/strict/1761866675 2025-12-04T08:27:07.4479851Z * [new tag] viable/strict/1761868178 -> viable/strict/1761868178 2025-12-04T08:27:07.4479986Z * [new tag] viable/strict/1761871111 -> viable/strict/1761871111 2025-12-04T08:27:07.4480148Z * [new tag] viable/strict/1761873126 -> viable/strict/1761873126 2025-12-04T08:27:07.4480280Z * [new tag] viable/strict/1761875714 -> viable/strict/1761875714 2025-12-04T08:27:07.4480414Z * [new tag] viable/strict/1761878924 -> viable/strict/1761878924 2025-12-04T08:27:07.4480533Z * [new tag] viable/strict/1761881727 -> viable/strict/1761881727 2025-12-04T08:27:07.4480657Z * [new tag] viable/strict/1761882959 -> viable/strict/1761882959 2025-12-04T08:27:07.4480775Z * [new tag] viable/strict/1761886268 -> viable/strict/1761886268 2025-12-04T08:27:07.4480944Z * [new tag] viable/strict/1761893641 -> viable/strict/1761893641 2025-12-04T08:27:07.4481070Z * [new tag] viable/strict/1761931517 -> viable/strict/1761931517 2025-12-04T08:27:07.4481188Z * [new tag] viable/strict/1761933080 -> viable/strict/1761933080 2025-12-04T08:27:07.4481313Z * [new tag] viable/strict/1761935217 -> viable/strict/1761935217 2025-12-04T08:27:07.4481432Z * [new tag] viable/strict/1761938533 -> viable/strict/1761938533 2025-12-04T08:27:07.4481549Z * [new tag] viable/strict/1761940184 -> viable/strict/1761940184 2025-12-04T08:27:07.4483520Z * [new tag] viable/strict/1761942338 -> viable/strict/1761942338 2025-12-04T08:27:07.4483737Z * [new tag] viable/strict/1761946100 -> viable/strict/1761946100 2025-12-04T08:27:07.4483936Z * [new tag] viable/strict/1761947374 -> viable/strict/1761947374 2025-12-04T08:27:07.4484162Z * [new tag] viable/strict/1761950978 -> viable/strict/1761950978 2025-12-04T08:27:07.4484303Z * [new tag] viable/strict/1761957727 -> viable/strict/1761957727 2025-12-04T08:27:07.4484430Z * [new tag] viable/strict/1761959532 -> viable/strict/1761959532 2025-12-04T08:27:07.4484679Z * [new tag] viable/strict/1761965366 -> viable/strict/1761965366 2025-12-04T08:27:07.4485322Z * [new tag] viable/strict/1761968066 -> viable/strict/1761968066 2025-12-04T08:27:07.4490809Z * [new tag] viable/strict/1761969322 -> viable/strict/1761969322 2025-12-04T08:27:07.4491259Z * [new tag] viable/strict/1761974723 -> viable/strict/1761974723 2025-12-04T08:27:07.4491509Z * [new tag] viable/strict/1761981837 -> viable/strict/1761981837 2025-12-04T08:27:07.4491667Z * [new tag] viable/strict/1761985546 -> viable/strict/1761985546 2025-12-04T08:27:07.4491931Z * [new tag] viable/strict/1761987030 -> viable/strict/1761987030 2025-12-04T08:27:07.4492169Z * [new tag] viable/strict/1762003554 -> viable/strict/1762003554 2025-12-04T08:27:07.4492323Z * [new tag] viable/strict/1762021560 -> viable/strict/1762021560 2025-12-04T08:27:07.4492525Z * [new tag] viable/strict/1762032190 -> viable/strict/1762032190 2025-12-04T08:27:07.4492678Z * [new tag] viable/strict/1762040981 -> viable/strict/1762040981 2025-12-04T08:27:07.4492881Z * [new tag] viable/strict/1762048525 -> viable/strict/1762048525 2025-12-04T08:27:07.4493038Z * [new tag] viable/strict/1762104223 -> viable/strict/1762104223 2025-12-04T08:27:07.4493165Z * [new tag] viable/strict/1762105778 -> viable/strict/1762105778 2025-12-04T08:27:07.4493284Z * [new tag] viable/strict/1762115109 -> viable/strict/1762115109 2025-12-04T08:27:07.4493419Z * [new tag] viable/strict/1762125840 -> viable/strict/1762125840 2025-12-04T08:27:07.4493540Z * [new tag] viable/strict/1762127377 -> viable/strict/1762127377 2025-12-04T08:27:07.4493661Z * [new tag] viable/strict/1762134925 -> viable/strict/1762134925 2025-12-04T08:27:07.4499796Z * [new tag] viable/strict/1762138338 -> viable/strict/1762138338 2025-12-04T08:27:07.4499961Z * [new tag] viable/strict/1762148993 -> viable/strict/1762148993 2025-12-04T08:27:07.4500113Z * [new tag] viable/strict/1762152871 -> viable/strict/1762152871 2025-12-04T08:27:07.4500272Z * [new tag] viable/strict/1762156183 -> viable/strict/1762156183 2025-12-04T08:27:07.4500418Z * [new tag] viable/strict/1762163457 -> viable/strict/1762163457 2025-12-04T08:27:07.4500550Z * [new tag] viable/strict/1762165569 -> viable/strict/1762165569 2025-12-04T08:27:07.4500834Z * [new tag] viable/strict/1762169035 -> viable/strict/1762169035 2025-12-04T08:27:07.4500969Z * [new tag] viable/strict/1762174936 -> viable/strict/1762174936 2025-12-04T08:27:07.4501113Z * [new tag] viable/strict/1762194412 -> viable/strict/1762194412 2025-12-04T08:27:07.4501231Z * [new tag] viable/strict/1762195876 -> viable/strict/1762195876 2025-12-04T08:27:07.4501853Z * [new tag] viable/strict/1762197788 -> viable/strict/1762197788 2025-12-04T08:27:07.4502370Z * [new tag] viable/strict/1762199389 -> viable/strict/1762199389 2025-12-04T08:27:07.4502557Z * [new tag] viable/strict/1762206585 -> viable/strict/1762206585 2025-12-04T08:27:07.4502679Z * [new tag] viable/strict/1762210184 -> viable/strict/1762210184 2025-12-04T08:27:07.4502800Z * [new tag] viable/strict/1762218736 -> viable/strict/1762218736 2025-12-04T08:27:07.4502935Z * [new tag] viable/strict/1762224529 -> viable/strict/1762224529 2025-12-04T08:27:07.4503059Z * [new tag] viable/strict/1762227253 -> viable/strict/1762227253 2025-12-04T08:27:07.4503181Z * [new tag] viable/strict/1762228515 -> viable/strict/1762228515 2025-12-04T08:27:07.4506241Z * [new tag] viable/strict/1762230349 -> viable/strict/1762230349 2025-12-04T08:27:07.4506476Z * [new tag] viable/strict/1762231859 -> viable/strict/1762231859 2025-12-04T08:27:07.4506630Z * [new tag] viable/strict/1762233925 -> viable/strict/1762233925 2025-12-04T08:27:07.4506869Z * [new tag] viable/strict/1762237630 -> viable/strict/1762237630 2025-12-04T08:27:07.4507114Z * [new tag] viable/strict/1762253522 -> viable/strict/1762253522 2025-12-04T08:27:07.4507248Z * [new tag] viable/strict/1762278588 -> viable/strict/1762278588 2025-12-04T08:27:07.4512924Z * [new tag] viable/strict/1762284203 -> viable/strict/1762284203 2025-12-04T08:27:07.4513090Z * [new tag] viable/strict/1762289446 -> viable/strict/1762289446 2025-12-04T08:27:07.4513214Z * [new tag] viable/strict/1762291515 -> viable/strict/1762291515 2025-12-04T08:27:07.4513335Z * [new tag] viable/strict/1762295100 -> viable/strict/1762295100 2025-12-04T08:27:07.4513457Z * [new tag] viable/strict/1762296590 -> viable/strict/1762296590 2025-12-04T08:27:07.4513575Z * [new tag] viable/strict/1762300179 -> viable/strict/1762300179 2025-12-04T08:27:07.4513737Z * [new tag] viable/strict/1762303207 -> viable/strict/1762303207 2025-12-04T08:27:07.4513855Z * [new tag] viable/strict/1762386584 -> viable/strict/1762386584 2025-12-04T08:27:07.4513970Z * [new tag] viable/strict/1762391537 -> viable/strict/1762391537 2025-12-04T08:27:07.4514149Z * [new tag] viable/strict/1762394119 -> viable/strict/1762394119 2025-12-04T08:27:07.4514475Z * [new tag] viable/strict/1762397437 -> viable/strict/1762397437 2025-12-04T08:27:07.4514605Z * [new tag] viable/strict/1762400256 -> viable/strict/1762400256 2025-12-04T08:27:07.4514850Z * [new tag] viable/strict/1762401469 -> viable/strict/1762401469 2025-12-04T08:27:07.4514978Z * [new tag] viable/strict/1762408195 -> viable/strict/1762408195 2025-12-04T08:27:07.4515106Z * [new tag] viable/strict/1762410411 -> viable/strict/1762410411 2025-12-04T08:27:07.4515239Z * [new tag] viable/strict/1762417613 -> viable/strict/1762417613 2025-12-04T08:27:07.4515494Z * [new tag] viable/strict/1762419198 -> viable/strict/1762419198 2025-12-04T08:27:07.4516394Z * [new tag] viable/strict/1762422656 -> viable/strict/1762422656 2025-12-04T08:27:07.4517339Z * [new tag] viable/strict/1762424746 -> viable/strict/1762424746 2025-12-04T08:27:07.4517829Z * [new tag] viable/strict/1762446386 -> viable/strict/1762446386 2025-12-04T08:27:07.4518608Z * [new tag] viable/strict/1762449912 -> viable/strict/1762449912 2025-12-04T08:27:07.4519447Z * [new tag] viable/strict/1762457031 -> viable/strict/1762457031 2025-12-04T08:27:07.4519832Z * [new tag] viable/strict/1762462441 -> viable/strict/1762462441 2025-12-04T08:27:07.4521283Z * [new tag] viable/strict/1762467909 -> viable/strict/1762467909 2025-12-04T08:27:07.4521578Z * [new tag] viable/strict/1762471493 -> viable/strict/1762471493 2025-12-04T08:27:07.4522448Z * [new tag] viable/strict/1762475990 -> viable/strict/1762475990 2025-12-04T08:27:07.4522984Z * [new tag] viable/strict/1762477933 -> viable/strict/1762477933 2025-12-04T08:27:07.4523902Z * [new tag] viable/strict/1762491053 -> viable/strict/1762491053 2025-12-04T08:27:07.4524298Z * [new tag] viable/strict/1762493118 -> viable/strict/1762493118 2025-12-04T08:27:07.4525237Z * [new tag] viable/strict/1762498442 -> viable/strict/1762498442 2025-12-04T08:27:07.4525558Z * [new tag] viable/strict/1762501778 -> viable/strict/1762501778 2025-12-04T08:27:07.4526508Z * [new tag] viable/strict/1762504001 -> viable/strict/1762504001 2025-12-04T08:27:07.4527189Z * [new tag] viable/strict/1762505583 -> viable/strict/1762505583 2025-12-04T08:27:07.4528277Z * [new tag] viable/strict/1762507523 -> viable/strict/1762507523 2025-12-04T08:27:07.4528506Z * [new tag] viable/strict/1762511140 -> viable/strict/1762511140 2025-12-04T08:27:07.4529579Z * [new tag] viable/strict/1762512632 -> viable/strict/1762512632 2025-12-04T08:27:07.4529978Z * [new tag] viable/strict/1762520467 -> viable/strict/1762520467 2025-12-04T08:27:07.4530865Z * [new tag] viable/strict/1762522016 -> viable/strict/1762522016 2025-12-04T08:27:07.4531227Z * [new tag] viable/strict/1762530591 -> viable/strict/1762530591 2025-12-04T08:27:07.4532389Z * [new tag] viable/strict/1762543405 -> viable/strict/1762543405 2025-12-04T08:27:07.4532799Z * [new tag] viable/strict/1762544998 -> viable/strict/1762544998 2025-12-04T08:27:07.4533720Z * [new tag] viable/strict/1762552182 -> viable/strict/1762552182 2025-12-04T08:27:07.4534172Z * [new tag] viable/strict/1762554297 -> viable/strict/1762554297 2025-12-04T08:27:07.4534959Z * [new tag] viable/strict/1762559381 -> viable/strict/1762559381 2025-12-04T08:27:07.4535329Z * [new tag] viable/strict/1762562222 -> viable/strict/1762562222 2025-12-04T08:27:07.4536264Z * [new tag] viable/strict/1762564319 -> viable/strict/1762564319 2025-12-04T08:27:07.4536571Z * [new tag] viable/strict/1762566904 -> viable/strict/1762566904 2025-12-04T08:27:07.4537382Z * [new tag] viable/strict/1762569781 -> viable/strict/1762569781 2025-12-04T08:27:07.4537711Z * [new tag] viable/strict/1762575940 -> viable/strict/1762575940 2025-12-04T08:27:07.4538627Z * [new tag] viable/strict/1762580974 -> viable/strict/1762580974 2025-12-04T08:27:07.4539068Z * [new tag] viable/strict/1762583185 -> viable/strict/1762583185 2025-12-04T08:27:07.4539934Z * [new tag] viable/strict/1762586647 -> viable/strict/1762586647 2025-12-04T08:27:07.4540333Z * [new tag] viable/strict/1762588183 -> viable/strict/1762588183 2025-12-04T08:27:07.4543734Z * [new tag] viable/strict/1762593886 -> viable/strict/1762593886 2025-12-04T08:27:07.4544044Z * [new tag] viable/strict/1762650743 -> viable/strict/1762650743 2025-12-04T08:27:07.4544436Z * [new tag] viable/strict/1762653328 -> viable/strict/1762653328 2025-12-04T08:27:07.4544577Z * [new tag] viable/strict/1762659342 -> viable/strict/1762659342 2025-12-04T08:27:07.4544806Z * [new tag] viable/strict/1762662360 -> viable/strict/1762662360 2025-12-04T08:27:07.4544958Z * [new tag] viable/strict/1762667377 -> viable/strict/1762667377 2025-12-04T08:27:07.4545327Z * [new tag] viable/strict/1762671090 -> viable/strict/1762671090 2025-12-04T08:27:07.4545791Z * [new tag] viable/strict/1762680284 -> viable/strict/1762680284 2025-12-04T08:27:07.4546721Z * [new tag] viable/strict/1762683900 -> viable/strict/1762683900 2025-12-04T08:27:07.4546887Z * [new tag] viable/strict/1762705541 -> viable/strict/1762705541 2025-12-04T08:27:07.4547850Z * [new tag] viable/strict/1762709004 -> viable/strict/1762709004 2025-12-04T08:27:07.4548261Z * [new tag] viable/strict/1762746004 -> viable/strict/1762746004 2025-12-04T08:27:07.4549313Z * [new tag] viable/strict/1762748799 -> viable/strict/1762748799 2025-12-04T08:27:07.4550277Z * [new tag] viable/strict/1762759504 -> viable/strict/1762759504 2025-12-04T08:27:07.4551059Z * [new tag] viable/strict/1762760973 -> viable/strict/1762760973 2025-12-04T08:27:07.4552082Z * [new tag] viable/strict/1762775374 -> viable/strict/1762775374 2025-12-04T08:27:07.4552858Z * [new tag] viable/strict/1762777661 -> viable/strict/1762777661 2025-12-04T08:27:07.4553613Z * [new tag] viable/strict/1762779774 -> viable/strict/1762779774 2025-12-04T08:27:07.4554885Z * [new tag] viable/strict/1762781259 -> viable/strict/1762781259 2025-12-04T08:27:07.4555642Z * [new tag] viable/strict/1762793628 -> viable/strict/1762793628 2025-12-04T08:27:07.4556767Z * [new tag] viable/strict/1762800711 -> viable/strict/1762800711 2025-12-04T08:27:07.4557427Z * [new tag] viable/strict/1762809894 -> viable/strict/1762809894 2025-12-04T08:27:07.4558630Z * [new tag] viable/strict/1762811384 -> viable/strict/1762811384 2025-12-04T08:27:07.4559391Z * [new tag] viable/strict/1762813841 -> viable/strict/1762813841 2025-12-04T08:27:07.4560719Z * [new tag] viable/strict/1762815047 -> viable/strict/1762815047 2025-12-04T08:27:07.4561417Z * [new tag] viable/strict/1762817094 -> viable/strict/1762817094 2025-12-04T08:27:07.4562390Z * [new tag] viable/strict/1762818582 -> viable/strict/1762818582 2025-12-04T08:27:07.4563085Z * [new tag] viable/strict/1762821623 -> viable/strict/1762821623 2025-12-04T08:27:07.4566115Z * [new tag] viable/strict/1762823531 -> viable/strict/1762823531 2025-12-04T08:27:07.4566376Z * [new tag] viable/strict/1762849583 -> viable/strict/1762849583 2025-12-04T08:27:07.4566675Z * [new tag] viable/strict/1762851200 -> viable/strict/1762851200 2025-12-04T08:27:07.4566879Z * [new tag] viable/strict/1762854603 -> viable/strict/1762854603 2025-12-04T08:27:07.4567073Z * [new tag] viable/strict/1762858276 -> viable/strict/1762858276 2025-12-04T08:27:07.4570258Z * [new tag] viable/strict/1762860891 -> viable/strict/1762860891 2025-12-04T08:27:07.4570575Z * [new tag] viable/strict/1762866174 -> viable/strict/1762866174 2025-12-04T08:27:07.4570938Z * [new tag] viable/strict/1762867653 -> viable/strict/1762867653 2025-12-04T08:27:07.4571429Z * [new tag] viable/strict/1762872669 -> viable/strict/1762872669 2025-12-04T08:27:07.4572070Z * [new tag] viable/strict/1762878380 -> viable/strict/1762878380 2025-12-04T08:27:07.4577299Z * [new tag] viable/strict/1762889003 -> viable/strict/1762889003 2025-12-04T08:27:07.4577461Z * [new tag] viable/strict/1762890589 -> viable/strict/1762890589 2025-12-04T08:27:07.4577965Z * [new tag] viable/strict/1762892743 -> viable/strict/1762892743 2025-12-04T08:27:07.4578207Z * [new tag] viable/strict/1762894271 -> viable/strict/1762894271 2025-12-04T08:27:07.4578407Z * [new tag] viable/strict/1762896287 -> viable/strict/1762896287 2025-12-04T08:27:07.4578641Z * [new tag] viable/strict/1762915871 -> viable/strict/1762915871 2025-12-04T08:27:07.4578865Z * [new tag] viable/strict/1762918569 -> viable/strict/1762918569 2025-12-04T08:27:07.4579238Z * [new tag] viable/strict/1762919776 -> viable/strict/1762919776 2025-12-04T08:27:07.4579494Z * [new tag] viable/strict/1762923072 -> viable/strict/1762923072 2025-12-04T08:27:07.4579720Z * [new tag] viable/strict/1762928826 -> viable/strict/1762928826 2025-12-04T08:27:07.4579956Z * [new tag] viable/strict/1762930451 -> viable/strict/1762930451 2025-12-04T08:27:07.4580168Z * [new tag] viable/strict/1762933780 -> viable/strict/1762933780 2025-12-04T08:27:07.4580392Z * [new tag] viable/strict/1762937638 -> viable/strict/1762937638 2025-12-04T08:27:07.4583833Z * [new tag] viable/strict/1762939545 -> viable/strict/1762939545 2025-12-04T08:27:07.4584113Z * [new tag] viable/strict/1762962692 -> viable/strict/1762962692 2025-12-04T08:27:07.4584371Z * [new tag] viable/strict/1762979143 -> viable/strict/1762979143 2025-12-04T08:27:07.4584613Z * [new tag] viable/strict/1762984188 -> viable/strict/1762984188 2025-12-04T08:27:07.4584842Z * [new tag] viable/strict/1762986306 -> viable/strict/1762986306 2025-12-04T08:27:07.4585058Z * [new tag] viable/strict/1762989903 -> viable/strict/1762989903 2025-12-04T08:27:07.4585270Z * [new tag] viable/strict/1762991377 -> viable/strict/1762991377 2025-12-04T08:27:07.4588470Z * [new tag] viable/strict/1762998921 -> viable/strict/1762998921 2025-12-04T08:27:07.4588713Z * [new tag] viable/strict/1763002287 -> viable/strict/1763002287 2025-12-04T08:27:07.4588964Z * [new tag] viable/strict/1763016840 -> viable/strict/1763016840 2025-12-04T08:27:07.4589194Z * [new tag] viable/strict/1763020180 -> viable/strict/1763020180 2025-12-04T08:27:07.4589399Z * [new tag] viable/strict/1763027421 -> viable/strict/1763027421 2025-12-04T08:27:07.4589645Z * [new tag] viable/strict/1763031120 -> viable/strict/1763031120 2025-12-04T08:27:07.4589884Z * [new tag] viable/strict/1763036861 -> viable/strict/1763036861 2025-12-04T08:27:07.4594813Z * [new tag] viable/strict/1763038993 -> viable/strict/1763038993 2025-12-04T08:27:07.4594974Z * [new tag] viable/strict/1763054703 -> viable/strict/1763054703 2025-12-04T08:27:07.4595164Z * [new tag] viable/strict/1763067061 -> viable/strict/1763067061 2025-12-04T08:27:07.4595299Z * [new tag] viable/strict/1763070847 -> viable/strict/1763070847 2025-12-04T08:27:07.4595431Z * [new tag] viable/strict/1763072706 -> viable/strict/1763072706 2025-12-04T08:27:07.4595571Z * [new tag] viable/strict/1763076302 -> viable/strict/1763076302 2025-12-04T08:27:07.4595699Z * [new tag] viable/strict/1763080816 -> viable/strict/1763080816 2025-12-04T08:27:07.4595996Z * [new tag] viable/strict/1763082732 -> viable/strict/1763082732 2025-12-04T08:27:07.4596139Z * [new tag] viable/strict/1763085329 -> viable/strict/1763085329 2025-12-04T08:27:07.4596420Z * [new tag] viable/strict/1763088623 -> viable/strict/1763088623 2025-12-04T08:27:07.4596617Z * [new tag] viable/strict/1763091402 -> viable/strict/1763091402 2025-12-04T08:27:07.4597793Z * [new tag] viable/strict/1763092602 -> viable/strict/1763092602 2025-12-04T08:27:07.4598390Z * [new tag] viable/strict/1763094355 -> viable/strict/1763094355 2025-12-04T08:27:07.4603649Z * [new tag] viable/strict/1763099390 -> viable/strict/1763099390 2025-12-04T08:27:07.4606080Z * [new tag] viable/strict/1763101608 -> viable/strict/1763101608 2025-12-04T08:27:07.4610709Z * [new tag] viable/strict/1763105102 -> viable/strict/1763105102 2025-12-04T08:27:07.4610857Z * [new tag] viable/strict/1763112347 -> viable/strict/1763112347 2025-12-04T08:27:07.4610982Z * [new tag] viable/strict/1763119471 -> viable/strict/1763119471 2025-12-04T08:27:07.4611114Z * [new tag] viable/strict/1763126835 -> viable/strict/1763126835 2025-12-04T08:27:07.4611248Z * [new tag] viable/strict/1763149779 -> viable/strict/1763149779 2025-12-04T08:27:07.4611374Z * [new tag] viable/strict/1763164178 -> viable/strict/1763164178 2025-12-04T08:27:07.4611495Z * [new tag] viable/strict/1763167104 -> viable/strict/1763167104 2025-12-04T08:27:07.4611612Z * [new tag] viable/strict/1763169132 -> viable/strict/1763169132 2025-12-04T08:27:07.4611883Z * [new tag] viable/strict/1763171708 -> viable/strict/1763171708 2025-12-04T08:27:07.4612014Z * [new tag] viable/strict/1763174759 -> viable/strict/1763174759 2025-12-04T08:27:07.4612148Z * [new tag] viable/strict/1763180744 -> viable/strict/1763180744 2025-12-04T08:27:07.4612318Z * [new tag] viable/strict/1763182227 -> viable/strict/1763182227 2025-12-04T08:27:07.4612447Z * [new tag] viable/strict/1763184309 -> viable/strict/1763184309 2025-12-04T08:27:07.4612576Z * [new tag] viable/strict/1763187991 -> viable/strict/1763187991 2025-12-04T08:27:07.4612699Z * [new tag] viable/strict/1763191445 -> viable/strict/1763191445 2025-12-04T08:27:07.4612836Z * [new tag] viable/strict/1763195152 -> viable/strict/1763195152 2025-12-04T08:27:07.4615569Z * [new tag] viable/strict/1763205769 -> viable/strict/1763205769 2025-12-04T08:27:07.4615743Z * [new tag] viable/strict/1763246990 -> viable/strict/1763246990 2025-12-04T08:27:07.4615887Z * [new tag] viable/strict/1763261578 -> viable/strict/1763261578 2025-12-04T08:27:07.4616012Z * [new tag] viable/strict/1763286573 -> viable/strict/1763286573 2025-12-04T08:27:07.4616143Z * [new tag] viable/strict/1763292167 -> viable/strict/1763292167 2025-12-04T08:27:07.4616266Z * [new tag] viable/strict/1763333386 -> viable/strict/1763333386 2025-12-04T08:27:07.4616451Z * [new tag] viable/strict/1763340082 -> viable/strict/1763340082 2025-12-04T08:27:07.4621847Z * [new tag] viable/strict/1763364324 -> viable/strict/1763364324 2025-12-04T08:27:07.4624677Z * [new tag] viable/strict/1763371569 -> viable/strict/1763371569 2025-12-04T08:27:07.4624896Z * [new tag] viable/strict/1763373067 -> viable/strict/1763373067 2025-12-04T08:27:07.4625027Z * [new tag] viable/strict/1763375157 -> viable/strict/1763375157 2025-12-04T08:27:07.4625157Z * [new tag] viable/strict/1763382462 -> viable/strict/1763382462 2025-12-04T08:27:07.4625384Z * [new tag] viable/strict/1763394661 -> viable/strict/1763394661 2025-12-04T08:27:07.4625508Z * [new tag] viable/strict/1763396797 -> viable/strict/1763396797 2025-12-04T08:27:07.4625642Z * [new tag] viable/strict/1763398542 -> viable/strict/1763398542 2025-12-04T08:27:07.4625761Z * [new tag] viable/strict/1763401807 -> viable/strict/1763401807 2025-12-04T08:27:07.4625888Z * [new tag] viable/strict/1763414698 -> viable/strict/1763414698 2025-12-04T08:27:07.4626008Z * [new tag] viable/strict/1763419807 -> viable/strict/1763419807 2025-12-04T08:27:07.4626129Z * [new tag] viable/strict/1763426369 -> viable/strict/1763426369 2025-12-04T08:27:07.4626261Z * [new tag] viable/strict/1763428331 -> viable/strict/1763428331 2025-12-04T08:27:07.4628805Z * [new tag] viable/strict/1763430922 -> viable/strict/1763430922 2025-12-04T08:27:07.4629540Z * [new tag] viable/strict/1763434184 -> viable/strict/1763434184 2025-12-04T08:27:07.4629797Z * [new tag] viable/strict/1763439973 -> viable/strict/1763439973 2025-12-04T08:27:07.4630047Z * [new tag] viable/strict/1763444995 -> viable/strict/1763444995 2025-12-04T08:27:07.4630245Z * [new tag] viable/strict/1763447206 -> viable/strict/1763447206 2025-12-04T08:27:07.4630454Z * [new tag] viable/strict/1763448826 -> viable/strict/1763448826 2025-12-04T08:27:07.4630668Z * [new tag] viable/strict/1763450717 -> viable/strict/1763450717 2025-12-04T08:27:07.4630890Z * [new tag] viable/strict/1763452183 -> viable/strict/1763452183 2025-12-04T08:27:07.4634442Z * [new tag] viable/strict/1763457945 -> viable/strict/1763457945 2025-12-04T08:27:07.4634631Z * [new tag] viable/strict/1763459439 -> viable/strict/1763459439 2025-12-04T08:27:07.4634776Z * [new tag] viable/strict/1763461556 -> viable/strict/1763461556 2025-12-04T08:27:07.4634897Z * [new tag] viable/strict/1763463103 -> viable/strict/1763463103 2025-12-04T08:27:07.4635023Z * [new tag] viable/strict/1763465100 -> viable/strict/1763465100 2025-12-04T08:27:07.4635142Z * [new tag] viable/strict/1763468866 -> viable/strict/1763468866 2025-12-04T08:27:07.4635266Z * [new tag] viable/strict/1763493823 -> viable/strict/1763493823 2025-12-04T08:27:07.4635436Z * [new tag] viable/strict/1763496249 -> viable/strict/1763496249 2025-12-04T08:27:07.4636670Z * [new tag] viable/strict/1763502620 -> viable/strict/1763502620 2025-12-04T08:27:07.4637321Z * [new tag] viable/strict/1763504715 -> viable/strict/1763504715 2025-12-04T08:27:07.4638492Z * [new tag] viable/strict/1763506208 -> viable/strict/1763506208 2025-12-04T08:27:07.4642107Z * [new tag] viable/strict/1763520590 -> viable/strict/1763520590 2025-12-04T08:27:07.4642337Z * [new tag] viable/strict/1763523357 -> viable/strict/1763523357 2025-12-04T08:27:07.4642542Z * [new tag] viable/strict/1763529922 -> viable/strict/1763529922 2025-12-04T08:27:07.4642769Z * [new tag] viable/strict/1763531408 -> viable/strict/1763531408 2025-12-04T08:27:07.4642992Z * [new tag] viable/strict/1763533622 -> viable/strict/1763533622 2025-12-04T08:27:07.4649945Z * [new tag] viable/strict/1763538576 -> viable/strict/1763538576 2025-12-04T08:27:07.4650331Z * [new tag] viable/strict/1763545823 -> viable/strict/1763545823 2025-12-04T08:27:07.4650658Z * [new tag] viable/strict/1763547951 -> viable/strict/1763547951 2025-12-04T08:27:07.4650931Z * [new tag] viable/strict/1763551477 -> viable/strict/1763551477 2025-12-04T08:27:07.4651677Z * [new tag] viable/strict/1763552982 -> viable/strict/1763552982 2025-12-04T08:27:07.4651831Z * [new tag] viable/strict/1763594698 -> viable/strict/1763594698 2025-12-04T08:27:07.4651983Z * [new tag] viable/strict/1763596178 -> viable/strict/1763596178 2025-12-04T08:27:07.4652105Z * [new tag] viable/strict/1763599155 -> viable/strict/1763599155 2025-12-04T08:27:07.4652224Z * [new tag] viable/strict/1763603717 -> viable/strict/1763603717 2025-12-04T08:27:07.4652352Z * [new tag] viable/strict/1763606923 -> viable/strict/1763606923 2025-12-04T08:27:07.4652471Z * [new tag] viable/strict/1763609715 -> viable/strict/1763609715 2025-12-04T08:27:07.4652607Z * [new tag] viable/strict/1763612757 -> viable/strict/1763612757 2025-12-04T08:27:07.4652735Z * [new tag] viable/strict/1763616325 -> viable/strict/1763616325 2025-12-04T08:27:07.4653960Z * [new tag] viable/strict/1763623509 -> viable/strict/1763623509 2025-12-04T08:27:07.4654409Z * [new tag] viable/strict/1763624984 -> viable/strict/1763624984 2025-12-04T08:27:07.4654673Z * [new tag] viable/strict/1763628796 -> viable/strict/1763628796 2025-12-04T08:27:07.4654900Z * [new tag] viable/strict/1763634343 -> viable/strict/1763634343 2025-12-04T08:27:07.4655111Z * [new tag] viable/strict/1763635867 -> viable/strict/1763635867 2025-12-04T08:27:07.4657632Z * [new tag] viable/strict/1763639382 -> viable/strict/1763639382 2025-12-04T08:27:07.4658138Z * [new tag] viable/strict/1763646626 -> viable/strict/1763646626 2025-12-04T08:27:07.4658374Z * [new tag] viable/strict/1763655997 -> viable/strict/1763655997 2025-12-04T08:27:07.4658507Z * [new tag] viable/strict/1763659444 -> viable/strict/1763659444 2025-12-04T08:27:07.4658654Z * [new tag] viable/strict/1763660992 -> viable/strict/1763660992 2025-12-04T08:27:07.4658772Z * [new tag] viable/strict/1763663201 -> viable/strict/1763663201 2025-12-04T08:27:07.4665191Z * [new tag] viable/strict/1763670362 -> viable/strict/1763670362 2025-12-04T08:27:07.4665347Z * [new tag] viable/strict/1763675378 -> viable/strict/1763675378 2025-12-04T08:27:07.4665817Z * [new tag] viable/strict/1763693343 -> viable/strict/1763693343 2025-12-04T08:27:07.4666052Z * [new tag] viable/strict/1763696088 -> viable/strict/1763696088 2025-12-04T08:27:07.4666266Z * [new tag] viable/strict/1763697343 -> viable/strict/1763697343 2025-12-04T08:27:07.4666499Z * [new tag] viable/strict/1763699165 -> viable/strict/1763699165 2025-12-04T08:27:07.4666721Z * [new tag] viable/strict/1763700660 -> viable/strict/1763700660 2025-12-04T08:27:07.4666946Z * [new tag] viable/strict/1763704209 -> viable/strict/1763704209 2025-12-04T08:27:07.4667160Z * [new tag] viable/strict/1763706411 -> viable/strict/1763706411 2025-12-04T08:27:07.4667381Z * [new tag] viable/strict/1763708082 -> viable/strict/1763708082 2025-12-04T08:27:07.4667615Z * [new tag] viable/strict/1763711381 -> viable/strict/1763711381 2025-12-04T08:27:07.4667810Z * [new tag] viable/strict/1763713593 -> viable/strict/1763713593 2025-12-04T08:27:07.4668014Z * [new tag] viable/strict/1763715201 -> viable/strict/1763715201 2025-12-04T08:27:07.4668241Z * [new tag] viable/strict/1763733017 -> viable/strict/1763733017 2025-12-04T08:27:07.4668478Z * [new tag] viable/strict/1763735108 -> viable/strict/1763735108 2025-12-04T08:27:07.4668692Z * [new tag] viable/strict/1763749579 -> viable/strict/1763749579 2025-12-04T08:27:07.4668979Z * [new tag] viable/strict/1763751113 -> viable/strict/1763751113 2025-12-04T08:27:07.4669243Z * [new tag] viable/strict/1763753035 -> viable/strict/1763753035 2025-12-04T08:27:07.4670524Z * [new tag] viable/strict/1763754578 -> viable/strict/1763754578 2025-12-04T08:27:07.4670954Z * [new tag] viable/strict/1763756748 -> viable/strict/1763756748 2025-12-04T08:27:07.4673539Z * [new tag] viable/strict/1763758205 -> viable/strict/1763758205 2025-12-04T08:27:07.4673925Z * [new tag] viable/strict/1763764050 -> viable/strict/1763764050 2025-12-04T08:27:07.4674068Z * [new tag] viable/strict/1763771887 -> viable/strict/1763771887 2025-12-04T08:27:07.4674224Z * [new tag] viable/strict/1763773920 -> viable/strict/1763773920 2025-12-04T08:27:07.4674927Z * [new tag] viable/strict/1763776501 -> viable/strict/1763776501 2025-12-04T08:27:07.4676915Z * [new tag] viable/strict/1763779437 -> viable/strict/1763779437 2025-12-04T08:27:07.4677187Z * [new tag] viable/strict/1763781038 -> viable/strict/1763781038 2025-12-04T08:27:07.4677448Z * [new tag] viable/strict/1763782245 -> viable/strict/1763782245 2025-12-04T08:27:07.4678416Z * [new tag] viable/strict/1763785568 -> viable/strict/1763785568 2025-12-04T08:27:07.4682554Z * [new tag] viable/strict/1763787006 -> viable/strict/1763787006 2025-12-04T08:27:07.4682921Z * [new tag] viable/strict/1763789103 -> viable/strict/1763789103 2025-12-04T08:27:07.4683260Z * [new tag] viable/strict/1763790578 -> viable/strict/1763790578 2025-12-04T08:27:07.4683630Z * [new tag] viable/strict/1763796275 -> viable/strict/1763796275 2025-12-04T08:27:07.4687548Z * [new tag] viable/strict/1763801465 -> viable/strict/1763801465 2025-12-04T08:27:07.4687829Z * [new tag] viable/strict/1763803522 -> viable/strict/1763803522 2025-12-04T08:27:07.4688197Z * [new tag] viable/strict/1763808581 -> viable/strict/1763808581 2025-12-04T08:27:07.4688485Z * [new tag] viable/strict/1763840977 -> viable/strict/1763840977 2025-12-04T08:27:07.4688748Z * [new tag] viable/strict/1763846659 -> viable/strict/1763846659 2025-12-04T08:27:07.4688979Z * [new tag] viable/strict/1763872065 -> viable/strict/1763872065 2025-12-04T08:27:07.4689222Z * [new tag] viable/strict/1763873648 -> viable/strict/1763873648 2025-12-04T08:27:07.4692586Z * [new tag] viable/strict/1763875506 -> viable/strict/1763875506 2025-12-04T08:27:07.4692989Z * [new tag] viable/strict/1763889904 -> viable/strict/1763889904 2025-12-04T08:27:07.4693232Z * [new tag] viable/strict/1763930999 -> viable/strict/1763930999 2025-12-04T08:27:07.4693445Z * [new tag] viable/strict/1763944964 -> viable/strict/1763944964 2025-12-04T08:27:07.4693678Z * [new tag] viable/strict/1763958474 -> viable/strict/1763958474 2025-12-04T08:27:07.4693910Z * [new tag] viable/strict/1763967263 -> viable/strict/1763967263 2025-12-04T08:27:07.4694147Z * [new tag] viable/strict/1763972803 -> viable/strict/1763972803 2025-12-04T08:27:07.4694336Z * [new tag] viable/strict/1763976376 -> viable/strict/1763976376 2025-12-04T08:27:07.4694553Z * [new tag] viable/strict/1763989404 -> viable/strict/1763989404 2025-12-04T08:27:07.4697132Z * [new tag] viable/strict/1763990887 -> viable/strict/1763990887 2025-12-04T08:27:07.4697351Z * [new tag] viable/strict/1764019919 -> viable/strict/1764019919 2025-12-04T08:27:07.4697664Z * [new tag] viable/strict/1764023134 -> viable/strict/1764023134 2025-12-04T08:27:07.4701264Z * [new tag] viable/strict/1764024593 -> viable/strict/1764024593 2025-12-04T08:27:07.4701421Z * [new tag] viable/strict/1764026706 -> viable/strict/1764026706 2025-12-04T08:27:07.4701593Z * [new tag] viable/strict/1764031139 -> viable/strict/1764031139 2025-12-04T08:27:07.4701730Z * [new tag] viable/strict/1764033131 -> viable/strict/1764033131 2025-12-04T08:27:07.4701877Z * [new tag] viable/strict/1764035725 -> viable/strict/1764035725 2025-12-04T08:27:07.4702014Z * [new tag] viable/strict/1764624265 -> viable/strict/1764624265 2025-12-04T08:27:07.4702157Z * [new tag] viable/strict/1764631514 -> viable/strict/1764631514 2025-12-04T08:27:07.4702318Z * [new tag] viable/strict/1764632987 -> viable/strict/1764632987 2025-12-04T08:27:07.4702458Z * [new tag] viable/strict/1764636063 -> viable/strict/1764636063 2025-12-04T08:27:07.4702615Z * [new tag] viable/strict/1764643975 -> viable/strict/1764643975 2025-12-04T08:27:07.4702757Z * [new tag] viable/strict/1764646859 -> viable/strict/1764646859 2025-12-04T08:27:07.4702904Z * [new tag] viable/strict/1764653120 -> viable/strict/1764653120 2025-12-04T08:27:07.4703089Z * [new tag] viable/strict/1764654632 -> viable/strict/1764654632 2025-12-04T08:27:07.4704192Z * [new tag] viable/strict/1764656821 -> viable/strict/1764656821 2025-12-04T08:27:07.4704524Z * [new tag] viable/strict/1764658557 -> viable/strict/1764658557 2025-12-04T08:27:07.4705197Z * [new tag] viable/strict/1764660333 -> viable/strict/1764660333 2025-12-04T08:27:07.4705723Z * [new tag] viable/strict/1764661812 -> viable/strict/1764661812 2025-12-04T08:27:07.4706558Z * [new tag] viable/strict/1764664023 -> viable/strict/1764664023 2025-12-04T08:27:07.4706791Z * [new tag] viable/strict/1764669150 -> viable/strict/1764669150 2025-12-04T08:27:07.4707954Z * [new tag] viable/strict/1764680709 -> viable/strict/1764680709 2025-12-04T08:27:07.4708196Z * [new tag] viable/strict/1764687619 -> viable/strict/1764687619 2025-12-04T08:27:07.4709136Z * [new tag] viable/strict/1764696355 -> viable/strict/1764696355 2025-12-04T08:27:07.4709730Z * [new tag] viable/strict/1764701767 -> viable/strict/1764701767 2025-12-04T08:27:07.4710391Z * [new tag] viable/strict/1764710768 -> viable/strict/1764710768 2025-12-04T08:27:07.4711052Z * [new tag] viable/strict/1764716202 -> viable/strict/1764716202 2025-12-04T08:27:07.4711774Z * [new tag] viable/strict/1764793566 -> viable/strict/1764793566 2025-12-04T08:27:07.4712449Z * [new tag] viable/strict/1764797093 -> viable/strict/1764797093 2025-12-04T08:27:07.4713124Z * [new tag] viable/strict/1764800729 -> viable/strict/1764800729 2025-12-04T08:27:07.4714296Z * [new tag] whc_flight_1 -> whc_flight_1 2025-12-04T08:27:07.4714952Z * [new tag] whc_flight_2 -> whc_flight_2 2025-12-04T08:27:07.4716140Z * [new tag] whc_flight_4 -> whc_flight_4 2025-12-04T08:27:07.5219869Z [command]/usr/bin/git rev-parse --verify --quiet ffd9b0fb4355e97af82fc42cf185c3ffa0fc0a32^{object} 2025-12-04T08:27:07.5246886Z ffd9b0fb4355e97af82fc42cf185c3ffa0fc0a32 2025-12-04T08:27:07.5261561Z ##[endgroup] 2025-12-04T08:27:07.5266071Z ##[group]Determining the checkout info 2025-12-04T08:27:07.5270643Z ##[endgroup] 2025-12-04T08:27:07.5272408Z [command]/usr/bin/git sparse-checkout disable 2025-12-04T08:27:07.5285581Z [command]/usr/bin/git config --local --unset-all extensions.worktreeConfig 2025-12-04T08:27:07.5310532Z ##[group]Checking out the ref 2025-12-04T08:27:07.5312691Z [command]/usr/bin/git checkout --progress --force ffd9b0fb4355e97af82fc42cf185c3ffa0fc0a32 2025-12-04T08:27:08.5582254Z Updating files: 94% (18996/20121) 2025-12-04T08:27:08.5718944Z Updating files: 95% (19115/20121) 2025-12-04T08:27:08.5844255Z Updating files: 96% (19317/20121) 2025-12-04T08:27:08.5979880Z Updating files: 97% (19518/20121) 2025-12-04T08:27:08.6213295Z Updating files: 98% (19719/20121) 2025-12-04T08:27:08.6351644Z Updating files: 99% (19920/20121) 2025-12-04T08:27:08.6351973Z Updating files: 100% (20121/20121) 2025-12-04T08:27:08.6352193Z Updating files: 100% (20121/20121), done. 2025-12-04T08:27:08.6578066Z Note: switching to 'ffd9b0fb4355e97af82fc42cf185c3ffa0fc0a32'. 2025-12-04T08:27:08.6580342Z 2025-12-04T08:27:08.6580647Z You are in 'detached HEAD' state. You can look around, make experimental 2025-12-04T08:27:08.6581005Z changes and commit them, and you can discard any commits you make in this 2025-12-04T08:27:08.6581331Z state without impacting any branches by switching back to a branch. 2025-12-04T08:27:08.6581537Z 2025-12-04T08:27:08.6581669Z If you want to create a new branch to retain commits you create, you may 2025-12-04T08:27:08.6581978Z do so (now or later) by using -c with the switch command. Example: 2025-12-04T08:27:08.6582147Z 2025-12-04T08:27:08.6582239Z git switch -c 2025-12-04T08:27:08.6582368Z 2025-12-04T08:27:08.6582447Z Or undo this operation with: 2025-12-04T08:27:08.6582572Z 2025-12-04T08:27:08.6582638Z git switch - 2025-12-04T08:27:08.6582743Z 2025-12-04T08:27:08.6582897Z Turn off this advice by setting config variable advice.detachedHead to false 2025-12-04T08:27:08.6583102Z 2025-12-04T08:27:08.6583280Z HEAD is now at ffd9b0fb435 Resolve collective autotuning test failure on arm (#168919) 2025-12-04T08:27:08.6645611Z ##[endgroup] 2025-12-04T08:27:08.6645962Z ##[group]Setting up auth for fetching submodules 2025-12-04T08:27:08.6650979Z [command]/usr/bin/git config --global http.https://github.com/.extraheader AUTHORIZATION: basic *** 2025-12-04T08:27:08.6716443Z [command]/usr/bin/git config --global --unset-all url.https://github.com/.insteadOf 2025-12-04T08:27:08.6742535Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf git@github.com: 2025-12-04T08:27:08.6776468Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf org-21003710@github.com: 2025-12-04T08:27:08.6815810Z ##[endgroup] 2025-12-04T08:27:08.6816419Z ##[group]Fetching submodules 2025-12-04T08:27:08.6816879Z [command]/usr/bin/git submodule sync --recursive 2025-12-04T08:27:08.7111135Z [command]/usr/bin/git -c protocol.version=2 submodule update --init --force --recursive 2025-12-04T08:27:08.7413181Z Submodule 'android/libs/fbjni' (https://github.com/facebookincubator/fbjni.git) registered for path 'android/libs/fbjni' 2025-12-04T08:27:08.7416066Z Submodule 'third_party/NNPACK_deps/FP16' (https://github.com/Maratyszcza/FP16.git) registered for path 'third_party/FP16' 2025-12-04T08:27:08.7416693Z Submodule 'third_party/NNPACK_deps/FXdiv' (https://github.com/Maratyszcza/FXdiv.git) registered for path 'third_party/FXdiv' 2025-12-04T08:27:08.7417273Z Submodule 'third_party/NNPACK' (https://github.com/Maratyszcza/NNPACK.git) registered for path 'third_party/NNPACK' 2025-12-04T08:27:08.7417818Z Submodule 'third_party/NVTX' (https://github.com/NVIDIA/NVTX.git) registered for path 'third_party/NVTX' 2025-12-04T08:27:08.7419276Z Submodule 'third_party/VulkanMemoryAllocator' (https://github.com/GPUOpen-LibrariesAndSDKs/VulkanMemoryAllocator.git) registered for path 'third_party/VulkanMemoryAllocator' 2025-12-04T08:27:08.7420806Z Submodule 'third_party/XNNPACK' (https://github.com/google/XNNPACK.git) registered for path 'third_party/XNNPACK' 2025-12-04T08:27:08.7431042Z Submodule 'third_party/aiter' (https://github.com/ROCm/aiter.git) registered for path 'third_party/aiter' 2025-12-04T08:27:08.7431695Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/benchmark' 2025-12-04T08:27:08.7432663Z Submodule 'third_party/composable_kernel' (https://github.com/ROCm/composable_kernel.git) registered for path 'third_party/composable_kernel' 2025-12-04T08:27:08.7433400Z Submodule 'third_party/cpp-httplib' (https://github.com/yhirose/cpp-httplib.git) registered for path 'third_party/cpp-httplib' 2025-12-04T08:27:08.7434058Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo.git) registered for path 'third_party/cpuinfo' 2025-12-04T08:27:08.7434734Z Submodule 'third_party/cudnn_frontend' (https://github.com/NVIDIA/cudnn-frontend.git) registered for path 'third_party/cudnn_frontend' 2025-12-04T08:27:08.7435403Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/cutlass' 2025-12-04T08:27:08.7437309Z Submodule 'third_party/fbgemm' (https://github.com/pytorch/fbgemm) registered for path 'third_party/fbgemm' 2025-12-04T08:27:08.7444427Z Submodule 'third_party/flash-attention' (https://github.com/Dao-AILab/flash-attention.git) registered for path 'third_party/flash-attention' 2025-12-04T08:27:08.7445356Z Submodule 'third_party/flatbuffers' (https://github.com/google/flatbuffers.git) registered for path 'third_party/flatbuffers' 2025-12-04T08:27:08.7449381Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/fmt' 2025-12-04T08:27:08.7450171Z Submodule 'third_party/gemmlowp/gemmlowp' (https://github.com/google/gemmlowp.git) registered for path 'third_party/gemmlowp/gemmlowp' 2025-12-04T08:27:08.7450926Z Submodule 'third_party/gloo' (https://github.com/pytorch/gloo) registered for path 'third_party/gloo' 2025-12-04T08:27:08.7457593Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/googletest' 2025-12-04T08:27:08.7459917Z Submodule 'third_party/ideep' (https://github.com/intel/ideep) registered for path 'third_party/ideep' 2025-12-04T08:27:08.7460675Z Submodule 'third_party/ittapi' (https://github.com/intel/ittapi.git) registered for path 'third_party/ittapi' 2025-12-04T08:27:08.7465140Z Submodule 'third_party/kineto' (https://github.com/pytorch/kineto) registered for path 'third_party/kineto' 2025-12-04T08:27:08.7467343Z Submodule 'third_party/kleidiai' (https://github.com/ARM-software/kleidiai.git) registered for path 'third_party/kleidiai' 2025-12-04T08:27:08.7468171Z Submodule 'third_party/mimalloc' (https://github.com/microsoft/mimalloc.git) registered for path 'third_party/mimalloc' 2025-12-04T08:27:08.7468919Z Submodule 'third_party/nlohmann' (https://github.com/nlohmann/json.git) registered for path 'third_party/nlohmann' 2025-12-04T08:27:08.7469596Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx' 2025-12-04T08:27:08.7472497Z Submodule 'third_party/opentelemetry-cpp' (https://github.com/open-telemetry/opentelemetry-cpp.git) registered for path 'third_party/opentelemetry-cpp' 2025-12-04T08:27:08.7473354Z Submodule 'third_party/pocketfft' (https://github.com/mreineck/pocketfft) registered for path 'third_party/pocketfft' 2025-12-04T08:27:08.7475073Z Submodule 'third_party/protobuf' (https://github.com/protocolbuffers/protobuf.git) registered for path 'third_party/protobuf' 2025-12-04T08:27:08.7477817Z Submodule 'third_party/NNPACK_deps/psimd' (https://github.com/Maratyszcza/psimd.git) registered for path 'third_party/psimd' 2025-12-04T08:27:08.7482811Z Submodule 'third_party/NNPACK_deps/pthreadpool' (https://github.com/Maratyszcza/pthreadpool.git) registered for path 'third_party/pthreadpool' 2025-12-04T08:27:08.7489487Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/pybind11' 2025-12-04T08:27:08.7490246Z Submodule 'third_party/python-peachpy' (https://github.com/malfet/PeachPy.git) registered for path 'third_party/python-peachpy' 2025-12-04T08:27:08.7490857Z Submodule 'third_party/sleef' (https://github.com/shibatch/sleef) registered for path 'third_party/sleef' 2025-12-04T08:27:08.7493312Z Submodule 'third_party/tensorpipe' (https://github.com/pytorch/tensorpipe.git) registered for path 'third_party/tensorpipe' 2025-12-04T08:27:08.7520030Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/android/libs/fbjni'... 2025-12-04T08:27:08.9614898Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FXdiv'... 2025-12-04T08:27:08.9615410Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FP16'... 2025-12-04T08:27:08.9615871Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/NNPACK'... 2025-12-04T08:27:08.9636708Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/composable_kernel'... 2025-12-04T08:27:16.7781468Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/NVTX'... 2025-12-04T08:27:16.7782329Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/benchmark'... 2025-12-04T08:27:16.7782976Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpuinfo'... 2025-12-04T08:27:16.7783625Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpp-httplib'... 2025-12-04T08:27:16.7784283Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flash-attention'... 2025-12-04T08:27:16.7784924Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cudnn_frontend'... 2025-12-04T08:27:16.7785586Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/VulkanMemoryAllocator'... 2025-12-04T08:27:16.7786263Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gemmlowp/gemmlowp'... 2025-12-04T08:27:16.7786887Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gloo'... 2025-12-04T08:27:16.7787614Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cutlass'... 2025-12-04T08:27:16.7788220Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flatbuffers'... 2025-12-04T08:27:16.7788828Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep'... 2025-12-04T08:27:16.7789415Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fmt'... 2025-12-04T08:27:16.7790000Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm'... 2025-12-04T08:27:16.7790591Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ittapi'... 2025-12-04T08:27:16.7791196Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/googletest'... 2025-12-04T08:27:16.7791814Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kleidiai'... 2025-12-04T08:27:16.7792411Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto'... 2025-12-04T08:27:16.7793004Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pocketfft'... 2025-12-04T08:27:16.7793620Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/mimalloc'... 2025-12-04T08:27:16.7794222Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/psimd'... 2025-12-04T08:27:16.7794831Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pthreadpool'... 2025-12-04T08:27:16.8059835Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf'... 2025-12-04T08:27:40.3576956Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-peachpy'... 2025-12-04T08:27:40.3577592Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pybind11'... 2025-12-04T08:27:40.3578095Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe'... 2025-12-04T08:27:40.3578612Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/sleef'... 2025-12-04T08:27:40.3579191Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx'... 2025-12-04T08:27:40.3579887Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/aiter'... 2025-12-04T08:27:40.3580338Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nlohmann'... 2025-12-04T08:27:40.4580820Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp'... 2025-12-04T08:27:42.6780840Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/XNNPACK'... 2025-12-04T08:27:42.6915691Z Submodule path 'android/libs/fbjni': checked out '7e1e1fe3858c63c251c637ae41a20de425dde96f' 2025-12-04T08:27:42.7035654Z Submodule path 'third_party/FP16': checked out '4dfe081cf6bcd15db339cf2680b9281b8451eeb3' 2025-12-04T08:27:42.7125981Z Submodule path 'third_party/FXdiv': checked out 'b408327ac2a15ec3e43352421954f5b1967701d1' 2025-12-04T08:27:42.7336015Z Submodule path 'third_party/NNPACK': checked out 'c07e3a0400713d546e0dea2d5466dd22ea389c73' 2025-12-04T08:27:42.8059078Z Submodule path 'third_party/NVTX': checked out '3ebbc93ded7285963bff932c678fa367eb393ba6' 2025-12-04T08:27:42.8521194Z Submodule path 'third_party/VulkanMemoryAllocator': checked out '1d8f600fd424278486eade7ed3e877c99f0846b1' 2025-12-04T08:27:43.4216905Z Submodule path 'third_party/XNNPACK': checked out '51a0103656eff6fc9bfd39a4597923c4b542c883' 2025-12-04T08:27:43.5828053Z Submodule path 'third_party/aiter': checked out '01aae101b9e5e94d6c16a9514c9fb8df99c93150' 2025-12-04T08:27:43.5850606Z Submodule '3rdparty/composable_kernel' (https://github.com/ROCm/composable_kernel.git) registered for path 'third_party/aiter/3rdparty/composable_kernel' 2025-12-04T08:27:43.5875823Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/aiter/3rdparty/composable_kernel'... 2025-12-04T08:27:48.5711236Z Submodule path 'third_party/aiter/3rdparty/composable_kernel': checked out 'cffe8fa2a442ac8e80dd236a1a5d24fe3d7e0cbf' 2025-12-04T08:27:48.5913288Z Submodule path 'third_party/benchmark': checked out '299e5928955cc62af9968370293b916f5130916f' 2025-12-04T08:27:48.8679643Z Submodule path 'third_party/composable_kernel': checked out '7fe50dc3da2069d6645d9deb8c017a876472a977' 2025-12-04T08:27:48.9095818Z Submodule path 'third_party/cpp-httplib': checked out '89c932f313c6437c38f2982869beacc89c2f2246' 2025-12-04T08:27:48.9943592Z Submodule path 'third_party/cpuinfo': checked out 'f858c30bcb16f8effd5ff46996f0514539e17abc' 2025-12-04T08:27:49.0369944Z Submodule path 'third_party/cudnn_frontend': checked out '0b1577c8c83401237d601d0d0db5210506705396' 2025-12-04T08:27:49.5653974Z Submodule path 'third_party/cutlass': checked out 'f88806b1e31dfa579842638740216dd41fc6c588' 2025-12-04T08:27:49.6846742Z Submodule path 'third_party/fbgemm': checked out 'c0b988d39a9e47c794d699f29930ed4d7c7e13a4' 2025-12-04T08:27:49.6866284Z Submodule 'external/asmjit' (https://github.com/asmjit/asmjit.git) registered for path 'third_party/fbgemm/external/asmjit' 2025-12-04T08:27:49.6869135Z Submodule 'external/composable_kernel' (https://github.com/ROCm/composable_kernel.git) registered for path 'third_party/fbgemm/external/composable_kernel' 2025-12-04T08:27:49.6874322Z Submodule 'external/cpuinfo' (https://github.com/pytorch/cpuinfo) registered for path 'third_party/fbgemm/external/cpuinfo' 2025-12-04T08:27:49.6876389Z Submodule 'external/cutlass' (https://github.com/jwfromm/cutlass) registered for path 'third_party/fbgemm/external/cutlass' 2025-12-04T08:27:49.6877053Z Submodule 'external/googletest' (https://github.com/google/googletest) registered for path 'third_party/fbgemm/external/googletest' 2025-12-04T08:27:49.6877820Z Submodule 'external/hipify_torch' (https://github.com/ROCmSoftwarePlatform/hipify_torch.git) registered for path 'third_party/fbgemm/external/hipify_torch' 2025-12-04T08:27:49.6878643Z Submodule 'external/json' (https://github.com/nlohmann/json.git) registered for path 'third_party/fbgemm/external/json' 2025-12-04T08:27:49.6900752Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/external/asmjit'... 2025-12-04T08:27:50.8845629Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/external/hipify_torch'... 2025-12-04T08:27:50.8846227Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/external/cpuinfo'... 2025-12-04T08:27:50.8846733Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/external/googletest'... 2025-12-04T08:27:50.9844641Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/external/composable_kernel'... 2025-12-04T08:27:54.5554105Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/external/cutlass'... 2025-12-04T08:27:54.6555225Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/external/json'... 2025-12-04T08:27:57.7814078Z Submodule path 'third_party/fbgemm/external/asmjit': checked out 'a3199e8857792cd10b7589ff5d58343d2c9008ea' 2025-12-04T08:27:58.0645670Z Submodule path 'third_party/fbgemm/external/composable_kernel': checked out '7fe50dc3da2069d6645d9deb8c017a876472a977' 2025-12-04T08:27:58.1532251Z Submodule path 'third_party/fbgemm/external/cpuinfo': checked out '6543fec09b2f04ac4a666882998b534afc9c1349' 2025-12-04T08:27:58.6707574Z Submodule path 'third_party/fbgemm/external/cutlass': checked out '98125ce499b0fdf7ffbe0e3052f5b8709f4840f8' 2025-12-04T08:27:58.7104224Z Submodule path 'third_party/fbgemm/external/googletest': checked out '52eb8108c5bdec04579160ae17225d66034bd723' 2025-12-04T08:27:58.7206488Z Submodule path 'third_party/fbgemm/external/hipify_torch': checked out '63b6a7b541fa7f08f8475ca7d74054db36ff2691' 2025-12-04T08:27:58.8079535Z Submodule path 'third_party/fbgemm/external/json': checked out '9cca280a4d0ccf0c08f47a99aa71d1b0e52f8d03' 2025-12-04T08:27:58.8662700Z Submodule path 'third_party/flash-attention': checked out '979702c87a8713a8e0a5e9fee122b90d2ef13be5' 2025-12-04T08:27:58.8677853Z Submodule 'csrc/composable_kernel' (https://github.com/ROCm/composable_kernel.git) registered for path 'third_party/flash-attention/csrc/composable_kernel' 2025-12-04T08:27:58.8678918Z Submodule 'csrc/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/flash-attention/csrc/cutlass' 2025-12-04T08:27:58.8708871Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flash-attention/csrc/composable_kernel'... 2025-12-04T08:28:03.5239146Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flash-attention/csrc/cutlass'... 2025-12-04T08:28:03.7047887Z Submodule path 'third_party/flash-attention/csrc/composable_kernel': checked out '888317e698e9803c62bd38568abc9e05d7709f33' 2025-12-04T08:28:04.1663690Z Submodule path 'third_party/flash-attention/csrc/cutlass': checked out 'c506e16788cb08416a4a57e11a9067beeee29420' 2025-12-04T08:28:04.2734757Z Submodule path 'third_party/flatbuffers': checked out 'a2cd1ea3b6d3fee220106b5fed3f7ce8da9eb757' 2025-12-04T08:28:04.3008731Z Submodule path 'third_party/fmt': checked out '407c905e45ad75fc29bf0f9bb7c5c2fd3475976f' 2025-12-04T08:28:04.3359554Z Submodule path 'third_party/gemmlowp/gemmlowp': checked out '3fb5c176c17c765a3492cd2f0321b0dab712f350' 2025-12-04T08:28:04.3569123Z Submodule path 'third_party/gloo': checked out '54cbae0d3a67fa890b4c3d9ee162b7860315e341' 2025-12-04T08:28:04.3987315Z Submodule path 'third_party/googletest': checked out '52eb8108c5bdec04579160ae17225d66034bd723' 2025-12-04T08:28:04.4115625Z Submodule path 'third_party/ideep': checked out '719d8e6cd7f7a0e01b155657526d693acf97c2b3' 2025-12-04T08:28:04.4133883Z Submodule 'mkl-dnn' (https://github.com/intel/mkl-dnn.git) registered for path 'third_party/ideep/mkl-dnn' 2025-12-04T08:28:04.4157176Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn'... 2025-12-04T08:28:20.4500705Z Submodule path 'third_party/ideep/mkl-dnn': checked out '8d263e693366ef8db40acc569cc7d8edf644556d' 2025-12-04T08:28:20.4673043Z Submodule path 'third_party/ittapi': checked out 'dec1d23ca65ab069d225dfe40dea14f455170959' 2025-12-04T08:28:20.5503158Z Submodule path 'third_party/kineto': checked out '31f85df8fbd89c188f14ef10f1ec65379786b943' 2025-12-04T08:28:20.5516766Z Submodule 'libkineto/third_party/dynolog' (https://github.com/facebookincubator/dynolog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog' 2025-12-04T08:28:20.5518925Z Submodule 'libkineto/third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/fmt' 2025-12-04T08:28:20.5520833Z Submodule 'libkineto/third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/googletest' 2025-12-04T08:28:20.5548923Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog'... 2025-12-04T08:28:21.9468907Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/fmt'... 2025-12-04T08:28:22.2130079Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/googletest'... 2025-12-04T08:28:22.2887042Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog': checked out 'd2ffe0a4e3acace628db49974246b66fc3e85fb1' 2025-12-04T08:28:22.2896584Z Submodule 'third_party/DCGM' (https://github.com/NVIDIA/DCGM.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2025-12-04T08:28:22.2897557Z Submodule 'third_party/cpr' (https://github.com/libcpr/cpr.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2025-12-04T08:28:22.2898376Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2025-12-04T08:28:22.2899433Z Submodule 'third_party/gflags' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2025-12-04T08:28:22.2900346Z Submodule 'third_party/glog' (https://github.com/google/glog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2025-12-04T08:28:22.2901947Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2025-12-04T08:28:22.2908092Z Submodule 'third_party/json' (https://github.com/nlohmann/json.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2025-12-04T08:28:22.2909052Z Submodule 'third_party/pfs' (https://github.com/dtrugman/pfs.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2025-12-04T08:28:22.2910054Z Submodule 'third_party/prometheus-cpp' (https://github.com/jupp0r/prometheus-cpp.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp' 2025-12-04T08:28:22.2935564Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'... 2025-12-04T08:28:24.1890178Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'... 2025-12-04T08:28:24.1895519Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'... 2025-12-04T08:28:24.1900138Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'... 2025-12-04T08:28:24.1904399Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/glog'... 2025-12-04T08:28:24.1906262Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp'... 2025-12-04T08:28:24.1907001Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'... 2025-12-04T08:28:24.1907933Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'... 2025-12-04T08:28:24.2891183Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/json'... 2025-12-04T08:28:30.2836512Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM': checked out 'ffde4e54bc7249a6039a5e6b45b395141e1217f9' 2025-12-04T08:28:30.2992659Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr': checked out '871ed52d350214a034f6ef8a3b8f51c5ce1bd400' 2025-12-04T08:28:30.3307405Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt': checked out 'cd4af11efc9c622896a3e4cb599fa28668ca3d05' 2025-12-04T08:28:30.3434621Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags': checked out 'e171aa2d15ed9eb17054558e0b3a6a413bb01067' 2025-12-04T08:28:30.3447685Z Submodule 'doc' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2025-12-04T08:28:30.3474096Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'... 2025-12-04T08:28:30.6182052Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc': checked out '8411df715cf522606e3b1aca386ddfc0b63d34b4' 2025-12-04T08:28:30.6346048Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog': checked out 'b33e3bad4c46c8a6345525fd822af355e5ef9446' 2025-12-04T08:28:30.6729716Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest': checked out '52eb8108c5bdec04579160ae17225d66034bd723' 2025-12-04T08:28:30.7593497Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json': checked out '4f8fba14066156b73f1189a2b8bd568bde5284c5' 2025-12-04T08:28:30.7741485Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs': checked out 'f68a2fa8ea36c783bdd760371411fcb495aa3150' 2025-12-04T08:28:30.7920001Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp': checked out 'b1234816facfdda29845c46696a02998a4af115a' 2025-12-04T08:28:30.7937155Z Submodule 'civetweb' (https://github.com/civetweb/civetweb.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/civetweb' 2025-12-04T08:28:30.7938344Z Submodule 'googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/googletest' 2025-12-04T08:28:30.7961169Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/civetweb'... 2025-12-04T08:28:33.0394008Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/googletest'... 2025-12-04T08:28:33.2557321Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/civetweb': checked out 'd7ba35bbb649209c66e582d5a0244ba988a15159' 2025-12-04T08:28:33.2992193Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2025-12-04T08:28:33.3288592Z Submodule path 'third_party/kineto/libkineto/third_party/fmt': checked out '40626af88bd7df9a5fb80be7b25ac85b122d6c21' 2025-12-04T08:28:33.3679722Z Submodule path 'third_party/kineto/libkineto/third_party/googletest': checked out '52eb8108c5bdec04579160ae17225d66034bd723' 2025-12-04T08:28:33.4141715Z Submodule path 'third_party/kleidiai': checked out 'd7770c89632329a9914ef1a90289917597639cbe' 2025-12-04T08:28:33.4489920Z Submodule path 'third_party/mimalloc': checked out 'fbd8b99c2b828428947d70fdc046bb55609be93e' 2025-12-04T08:28:33.5366052Z Submodule path 'third_party/nlohmann': checked out '55f93686c01528224f448c19128836e7df245f72' 2025-12-04T08:28:33.8306611Z Submodule path 'third_party/onnx': checked out 'e709452ef2bbc1d113faf678c24e6d3467696e83' 2025-12-04T08:28:33.8334944Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx/third_party/pybind11' 2025-12-04T08:28:33.8363500Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/pybind11'... 2025-12-04T08:28:35.3503411Z Submodule path 'third_party/onnx/third_party/pybind11': checked out 'a2e59f0e7065404b44dfe92a28aca47ba1378dc4' 2025-12-04T08:28:35.4048600Z Submodule path 'third_party/opentelemetry-cpp': checked out 'a799f4aed9c94b765dcdaabaeab7d5e7e2310878' 2025-12-04T08:28:35.4069221Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark) registered for path 'third_party/opentelemetry-cpp/third_party/benchmark' 2025-12-04T08:28:35.4070145Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/opentelemetry-cpp/third_party/googletest' 2025-12-04T08:28:35.4073456Z Submodule 'third_party/ms-gsl' (https://github.com/microsoft/GSL) registered for path 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2025-12-04T08:28:35.4074181Z Submodule 'third_party/nlohmann-json' (https://github.com/nlohmann/json) registered for path 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2025-12-04T08:28:35.4074974Z Submodule 'third_party/opentelemetry-proto' (https://github.com/open-telemetry/opentelemetry-proto) registered for path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2025-12-04T08:28:35.4076079Z Submodule 'third_party/opentracing-cpp' (https://github.com/opentracing/opentracing-cpp.git) registered for path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2025-12-04T08:28:35.4076873Z Submodule 'third_party/prometheus-cpp' (https://github.com/jupp0r/prometheus-cpp) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2025-12-04T08:28:35.4077545Z Submodule 'tools/vcpkg' (https://github.com/Microsoft/vcpkg) registered for path 'third_party/opentelemetry-cpp/tools/vcpkg' 2025-12-04T08:28:35.4101409Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/benchmark'... 2025-12-04T08:28:35.8214906Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentracing-cpp'... 2025-12-04T08:28:35.8216207Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentelemetry-proto'... 2025-12-04T08:28:35.8217403Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp'... 2025-12-04T08:28:35.8218460Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/ms-gsl'... 2025-12-04T08:28:35.9216019Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/googletest'... 2025-12-04T08:28:36.6846274Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/nlohmann-json'... 2025-12-04T08:28:43.6036478Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/tools/vcpkg'... 2025-12-04T08:28:44.2760514Z Submodule path 'third_party/opentelemetry-cpp/third_party/benchmark': checked out 'd572f4777349d43653b21d6c2fc63020ab326db2' 2025-12-04T08:28:44.3112176Z Submodule path 'third_party/opentelemetry-cpp/third_party/googletest': checked out 'b796f7d44681514f58a683a3a71ff17c94edb0c1' 2025-12-04T08:28:44.3269189Z Submodule path 'third_party/opentelemetry-cpp/third_party/ms-gsl': checked out '6f4529395c5b7c2d661812257cd6780c67e54afa' 2025-12-04T08:28:44.4188449Z Submodule path 'third_party/opentelemetry-cpp/third_party/nlohmann-json': checked out 'bc889afb4c5bf1c0d8ee29ef35eaaf4c8bef8a5d' 2025-12-04T08:28:44.4315844Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto': checked out '4ca4f0335c63cda7ab31ea7ed70d6553aee14dce' 2025-12-04T08:28:44.4445085Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp': checked out '06b57f48ded1fa3bdd3d4346f6ef29e40e08eaf5' 2025-12-04T08:28:44.4584850Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp': checked out 'c9ffcdda9086ffd9e1283ea7a0276d831f3c8a8d' 2025-12-04T08:28:44.4597951Z Submodule 'civetweb' (https://github.com/civetweb/civetweb.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2025-12-04T08:28:44.4599043Z Submodule 'googletest' (https://github.com/google/googletest.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2025-12-04T08:28:44.4630051Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'... 2025-12-04T08:28:46.8186149Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'... 2025-12-04T08:28:47.0316782Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb': checked out 'eefb26f82b233268fc98577d265352720d477ba4' 2025-12-04T08:28:47.0705150Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2025-12-04T08:28:47.4036766Z Submodule path 'third_party/opentelemetry-cpp/tools/vcpkg': checked out '8eb57355a4ffb410a2e94c07b4dca2dffbee8e50' 2025-12-04T08:28:47.4139619Z Submodule path 'third_party/pocketfft': checked out '0fa0ef591e38c2758e3184c6c23e497b9f732ffa' 2025-12-04T08:28:47.6349492Z Submodule path 'third_party/protobuf': checked out 'd1eca4e4b421cd2997495c4b4e65cea6be4e9b8a' 2025-12-04T08:28:47.6366701Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/protobuf/third_party/benchmark' 2025-12-04T08:28:47.6372675Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/protobuf/third_party/googletest' 2025-12-04T08:28:47.6394433Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/benchmark'... 2025-12-04T08:28:48.1569461Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/googletest'... 2025-12-04T08:28:48.6920372Z Submodule path 'third_party/protobuf/third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8' 2025-12-04T08:28:48.7542535Z Submodule path 'third_party/protobuf/third_party/googletest': checked out '5ec7f0c4a113e2f18ac2c6cc7df51ad6afc24081' 2025-12-04T08:28:48.7637539Z Submodule path 'third_party/psimd': checked out '072586a71b55b7f8c584153d223e95687148a900' 2025-12-04T08:28:48.7755844Z Submodule path 'third_party/pthreadpool': checked out '4fe0e1e183925bf8cfa6aae24237e724a96479b8' 2025-12-04T08:28:48.8145055Z Submodule path 'third_party/pybind11': checked out 'f5fbe867d2d26e4a0a9177a51f6e568868ad3dc8' 2025-12-04T08:28:48.8407881Z Submodule path 'third_party/python-peachpy': checked out 'f45429b087dd7d5bc78bb40dc7cf06425c252d67' 2025-12-04T08:28:48.8790234Z Submodule path 'third_party/sleef': checked out '5a1d179df9cf652951b59010a2d2075372d67f68' 2025-12-04T08:28:48.9034306Z Submodule path 'third_party/tensorpipe': checked out '2b4cd91092d335a697416b2a3cb398283246849d' 2025-12-04T08:28:48.9053113Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/tensorpipe/third_party/googletest' 2025-12-04T08:28:48.9055384Z Submodule 'third_party/libnop' (https://github.com/google/libnop.git) registered for path 'third_party/tensorpipe/third_party/libnop' 2025-12-04T08:28:48.9056602Z Submodule 'third_party/libuv' (https://github.com/libuv/libuv.git) registered for path 'third_party/tensorpipe/third_party/libuv' 2025-12-04T08:28:48.9057462Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/tensorpipe/third_party/pybind11' 2025-12-04T08:28:48.9079020Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/googletest'... 2025-12-04T08:28:50.0174546Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libnop'... 2025-12-04T08:28:50.0175231Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11'... 2025-12-04T08:28:50.0175899Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libuv'... 2025-12-04T08:28:50.0673762Z Submodule path 'third_party/tensorpipe/third_party/googletest': checked out 'aee0f9d9b5b87796ee8a0ab26b7587ec30e8858e' 2025-12-04T08:28:50.0818789Z Submodule path 'third_party/tensorpipe/third_party/libnop': checked out '910b55815be16109f04f4180e9adee14fb4ce281' 2025-12-04T08:28:50.1474240Z Submodule path 'third_party/tensorpipe/third_party/libuv': checked out '5152db2cbfeb5582e9c27c5ea1dba2cd9e10759b' 2025-12-04T08:28:50.1733475Z Submodule path 'third_party/tensorpipe/third_party/pybind11': checked out 'a23996fce38ff6ccfbcdc09f1e63f2c4be5ea2ef' 2025-12-04T08:28:50.1750159Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2025-12-04T08:28:50.1780540Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11/tools/clang'... 2025-12-04T08:28:50.3636348Z Submodule path 'third_party/tensorpipe/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5' 2025-12-04T08:28:50.3674770Z [command]/usr/bin/git submodule foreach --recursive git config --local gc.auto 0 2025-12-04T08:28:50.3975325Z Entering 'android/libs/fbjni' 2025-12-04T08:28:50.4025604Z Entering 'third_party/FP16' 2025-12-04T08:28:50.4067647Z Entering 'third_party/FXdiv' 2025-12-04T08:28:50.4107593Z Entering 'third_party/NNPACK' 2025-12-04T08:28:50.4151998Z Entering 'third_party/NVTX' 2025-12-04T08:28:50.4186420Z Entering 'third_party/VulkanMemoryAllocator' 2025-12-04T08:28:50.4222023Z Entering 'third_party/XNNPACK' 2025-12-04T08:28:50.4276069Z Entering 'third_party/aiter' 2025-12-04T08:28:50.4315367Z Entering 'third_party/aiter/3rdparty/composable_kernel' 2025-12-04T08:28:50.4364770Z Entering 'third_party/benchmark' 2025-12-04T08:28:50.4404702Z Entering 'third_party/composable_kernel' 2025-12-04T08:28:50.4452366Z Entering 'third_party/cpp-httplib' 2025-12-04T08:28:50.4487937Z Entering 'third_party/cpuinfo' 2025-12-04T08:28:50.4525050Z Entering 'third_party/cudnn_frontend' 2025-12-04T08:28:50.4568617Z Entering 'third_party/cutlass' 2025-12-04T08:28:50.4617735Z Entering 'third_party/fbgemm' 2025-12-04T08:28:50.4655336Z Entering 'third_party/fbgemm/external/asmjit' 2025-12-04T08:28:50.4697447Z Entering 'third_party/fbgemm/external/composable_kernel' 2025-12-04T08:28:50.4745402Z Entering 'third_party/fbgemm/external/cpuinfo' 2025-12-04T08:28:50.4786206Z Entering 'third_party/fbgemm/external/cutlass' 2025-12-04T08:28:50.4836082Z Entering 'third_party/fbgemm/external/googletest' 2025-12-04T08:28:50.4877164Z Entering 'third_party/fbgemm/external/hipify_torch' 2025-12-04T08:28:50.4917781Z Entering 'third_party/fbgemm/external/json' 2025-12-04T08:28:50.4963039Z Entering 'third_party/flash-attention' 2025-12-04T08:28:50.4998629Z Entering 'third_party/flash-attention/csrc/composable_kernel' 2025-12-04T08:28:50.5043773Z Entering 'third_party/flash-attention/csrc/cutlass' 2025-12-04T08:28:50.5089630Z Entering 'third_party/flatbuffers' 2025-12-04T08:28:50.5130654Z Entering 'third_party/fmt' 2025-12-04T08:28:50.5175362Z Entering 'third_party/gemmlowp/gemmlowp' 2025-12-04T08:28:50.5216908Z Entering 'third_party/gloo' 2025-12-04T08:28:50.5265312Z Entering 'third_party/googletest' 2025-12-04T08:28:50.5300525Z Entering 'third_party/ideep' 2025-12-04T08:28:50.5336481Z Entering 'third_party/ideep/mkl-dnn' 2025-12-04T08:28:50.5389248Z Entering 'third_party/ittapi' 2025-12-04T08:28:50.5430333Z Entering 'third_party/kineto' 2025-12-04T08:28:50.5467053Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2025-12-04T08:28:50.5503474Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2025-12-04T08:28:50.5548421Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2025-12-04T08:28:50.5596714Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2025-12-04T08:28:50.5632994Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2025-12-04T08:28:50.5674255Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2025-12-04T08:28:50.5710210Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2025-12-04T08:28:50.5760378Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2025-12-04T08:28:50.5798587Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2025-12-04T08:28:50.5847040Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2025-12-04T08:28:50.5885790Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp' 2025-12-04T08:28:50.5920986Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/civetweb' 2025-12-04T08:28:50.5965573Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/googletest' 2025-12-04T08:28:50.6008526Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2025-12-04T08:28:50.6048422Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2025-12-04T08:28:50.6096615Z Entering 'third_party/kleidiai' 2025-12-04T08:28:50.6138521Z Entering 'third_party/mimalloc' 2025-12-04T08:28:50.6184761Z Entering 'third_party/nlohmann' 2025-12-04T08:28:50.6235282Z Entering 'third_party/onnx' 2025-12-04T08:28:50.6279416Z Entering 'third_party/onnx/third_party/pybind11' 2025-12-04T08:28:50.6326808Z Entering 'third_party/opentelemetry-cpp' 2025-12-04T08:28:50.6366220Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2025-12-04T08:28:50.6405562Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2025-12-04T08:28:50.6443682Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2025-12-04T08:28:50.6483323Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2025-12-04T08:28:50.6520473Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2025-12-04T08:28:50.6562595Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2025-12-04T08:28:50.6595298Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2025-12-04T08:28:50.6633554Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2025-12-04T08:28:50.6687303Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2025-12-04T08:28:50.6734823Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2025-12-04T08:28:50.6790200Z Entering 'third_party/pocketfft' 2025-12-04T08:28:50.6833514Z Entering 'third_party/protobuf' 2025-12-04T08:28:50.6875933Z Entering 'third_party/protobuf/third_party/benchmark' 2025-12-04T08:28:50.6906880Z Entering 'third_party/protobuf/third_party/googletest' 2025-12-04T08:28:50.6951486Z Entering 'third_party/psimd' 2025-12-04T08:28:50.6991363Z Entering 'third_party/pthreadpool' 2025-12-04T08:28:50.7034429Z Entering 'third_party/pybind11' 2025-12-04T08:28:50.7074406Z Entering 'third_party/python-peachpy' 2025-12-04T08:28:50.7116071Z Entering 'third_party/sleef' 2025-12-04T08:28:50.7156131Z Entering 'third_party/tensorpipe' 2025-12-04T08:28:50.7195363Z Entering 'third_party/tensorpipe/third_party/googletest' 2025-12-04T08:28:50.7231612Z Entering 'third_party/tensorpipe/third_party/libnop' 2025-12-04T08:28:50.7270335Z Entering 'third_party/tensorpipe/third_party/libuv' 2025-12-04T08:28:50.7308801Z Entering 'third_party/tensorpipe/third_party/pybind11' 2025-12-04T08:28:50.7351910Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2025-12-04T08:28:50.7403904Z ##[endgroup] 2025-12-04T08:28:50.7404300Z ##[group]Persisting credentials for submodules 2025-12-04T08:28:50.7409609Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'url\.https\:\/\/github\.com\/\.insteadOf' && git config --local --unset-all 'url.https://github.com/.insteadOf' || :" 2025-12-04T08:28:50.7708298Z Entering 'android/libs/fbjni' 2025-12-04T08:28:50.7758913Z Entering 'third_party/FP16' 2025-12-04T08:28:50.7814015Z Entering 'third_party/FXdiv' 2025-12-04T08:28:50.7868651Z Entering 'third_party/NNPACK' 2025-12-04T08:28:50.7916295Z Entering 'third_party/NVTX' 2025-12-04T08:28:50.7971853Z Entering 'third_party/VulkanMemoryAllocator' 2025-12-04T08:28:50.8023404Z Entering 'third_party/XNNPACK' 2025-12-04T08:28:50.8086541Z Entering 'third_party/aiter' 2025-12-04T08:28:50.8142594Z Entering 'third_party/aiter/3rdparty/composable_kernel' 2025-12-04T08:28:50.8205441Z Entering 'third_party/benchmark' 2025-12-04T08:28:50.8261488Z Entering 'third_party/composable_kernel' 2025-12-04T08:28:50.8321879Z Entering 'third_party/cpp-httplib' 2025-12-04T08:28:50.8380937Z Entering 'third_party/cpuinfo' 2025-12-04T08:28:50.8436509Z Entering 'third_party/cudnn_frontend' 2025-12-04T08:28:50.8491839Z Entering 'third_party/cutlass' 2025-12-04T08:28:50.8549540Z Entering 'third_party/fbgemm' 2025-12-04T08:28:50.8606550Z Entering 'third_party/fbgemm/external/asmjit' 2025-12-04T08:28:50.8661016Z Entering 'third_party/fbgemm/external/composable_kernel' 2025-12-04T08:28:50.8720321Z Entering 'third_party/fbgemm/external/cpuinfo' 2025-12-04T08:28:50.8774772Z Entering 'third_party/fbgemm/external/cutlass' 2025-12-04T08:28:50.8838684Z Entering 'third_party/fbgemm/external/googletest' 2025-12-04T08:28:50.8888651Z Entering 'third_party/fbgemm/external/hipify_torch' 2025-12-04T08:28:50.8941346Z Entering 'third_party/fbgemm/external/json' 2025-12-04T08:28:50.8993700Z Entering 'third_party/flash-attention' 2025-12-04T08:28:50.9050849Z Entering 'third_party/flash-attention/csrc/composable_kernel' 2025-12-04T08:28:50.9106657Z Entering 'third_party/flash-attention/csrc/cutlass' 2025-12-04T08:28:50.9170521Z Entering 'third_party/flatbuffers' 2025-12-04T08:28:50.9220342Z Entering 'third_party/fmt' 2025-12-04T08:28:50.9276485Z Entering 'third_party/gemmlowp/gemmlowp' 2025-12-04T08:28:50.9334507Z Entering 'third_party/gloo' 2025-12-04T08:28:50.9395186Z Entering 'third_party/googletest' 2025-12-04T08:28:50.9447686Z Entering 'third_party/ideep' 2025-12-04T08:28:50.9498222Z Entering 'third_party/ideep/mkl-dnn' 2025-12-04T08:28:50.9560182Z Entering 'third_party/ittapi' 2025-12-04T08:28:50.9617853Z Entering 'third_party/kineto' 2025-12-04T08:28:50.9672726Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2025-12-04T08:28:50.9731525Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2025-12-04T08:28:50.9776990Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2025-12-04T08:28:50.9830893Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2025-12-04T08:28:50.9887749Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2025-12-04T08:28:50.9939575Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2025-12-04T08:28:50.9995026Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2025-12-04T08:28:51.0049644Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2025-12-04T08:28:51.0103012Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2025-12-04T08:28:51.0154166Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2025-12-04T08:28:51.0209315Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp' 2025-12-04T08:28:51.0257733Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/civetweb' 2025-12-04T08:28:51.0312907Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/googletest' 2025-12-04T08:28:51.0377404Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2025-12-04T08:28:51.0436950Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2025-12-04T08:28:51.0495042Z Entering 'third_party/kleidiai' 2025-12-04T08:28:51.0550363Z Entering 'third_party/mimalloc' 2025-12-04T08:28:51.0605384Z Entering 'third_party/nlohmann' 2025-12-04T08:28:51.0662599Z Entering 'third_party/onnx' 2025-12-04T08:28:51.0728920Z Entering 'third_party/onnx/third_party/pybind11' 2025-12-04T08:28:51.0789854Z Entering 'third_party/opentelemetry-cpp' 2025-12-04T08:28:51.0846904Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2025-12-04T08:28:51.0908002Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2025-12-04T08:28:51.0964023Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2025-12-04T08:28:51.1017584Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2025-12-04T08:28:51.1070921Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2025-12-04T08:28:51.1121084Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2025-12-04T08:28:51.1176042Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2025-12-04T08:28:51.1229238Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2025-12-04T08:28:51.1278591Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2025-12-04T08:28:51.1334131Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2025-12-04T08:28:51.1402272Z Entering 'third_party/pocketfft' 2025-12-04T08:28:51.1457064Z Entering 'third_party/protobuf' 2025-12-04T08:28:51.1506071Z Entering 'third_party/protobuf/third_party/benchmark' 2025-12-04T08:28:51.1562574Z Entering 'third_party/protobuf/third_party/googletest' 2025-12-04T08:28:51.1609076Z Entering 'third_party/psimd' 2025-12-04T08:28:51.1665023Z Entering 'third_party/pthreadpool' 2025-12-04T08:28:51.1716678Z Entering 'third_party/pybind11' 2025-12-04T08:28:51.1767463Z Entering 'third_party/python-peachpy' 2025-12-04T08:28:51.1821489Z Entering 'third_party/sleef' 2025-12-04T08:28:51.1871301Z Entering 'third_party/tensorpipe' 2025-12-04T08:28:51.1920002Z Entering 'third_party/tensorpipe/third_party/googletest' 2025-12-04T08:28:51.1976528Z Entering 'third_party/tensorpipe/third_party/libnop' 2025-12-04T08:28:51.2025026Z Entering 'third_party/tensorpipe/third_party/libuv' 2025-12-04T08:28:51.2074641Z Entering 'third_party/tensorpipe/third_party/pybind11' 2025-12-04T08:28:51.2123460Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2025-12-04T08:28:51.2199036Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local 'http.https://github.com/.extraheader' 'AUTHORIZATION: basic ***' && git config --local --show-origin --name-only --get-regexp remote.origin.url" 2025-12-04T08:28:51.2509419Z Entering 'android/libs/fbjni' 2025-12-04T08:28:51.2557648Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config remote.origin.url 2025-12-04T08:28:51.2577930Z Entering 'third_party/FP16' 2025-12-04T08:28:51.2628344Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config remote.origin.url 2025-12-04T08:28:51.2643781Z Entering 'third_party/FXdiv' 2025-12-04T08:28:51.2693102Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config remote.origin.url 2025-12-04T08:28:51.2709045Z Entering 'third_party/NNPACK' 2025-12-04T08:28:51.2754214Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config remote.origin.url 2025-12-04T08:28:51.2774274Z Entering 'third_party/NVTX' 2025-12-04T08:28:51.2825284Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NVTX/config remote.origin.url 2025-12-04T08:28:51.2842225Z Entering 'third_party/VulkanMemoryAllocator' 2025-12-04T08:28:51.2891747Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config remote.origin.url 2025-12-04T08:28:51.2911472Z Entering 'third_party/XNNPACK' 2025-12-04T08:28:51.2956494Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config remote.origin.url 2025-12-04T08:28:51.2986166Z Entering 'third_party/aiter' 2025-12-04T08:28:51.3033321Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/aiter/config remote.origin.url 2025-12-04T08:28:51.3054362Z Entering 'third_party/aiter/3rdparty/composable_kernel' 2025-12-04T08:28:51.3098067Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/aiter/modules/3rdparty/composable_kernel/config remote.origin.url 2025-12-04T08:28:51.3119317Z Entering 'third_party/benchmark' 2025-12-04T08:28:51.3171060Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config remote.origin.url 2025-12-04T08:28:51.3189474Z Entering 'third_party/composable_kernel' 2025-12-04T08:28:51.3234579Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/composable_kernel/config remote.origin.url 2025-12-04T08:28:51.3261550Z Entering 'third_party/cpp-httplib' 2025-12-04T08:28:51.3306475Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpp-httplib/config remote.origin.url 2025-12-04T08:28:51.3318492Z Entering 'third_party/cpuinfo' 2025-12-04T08:28:51.3371506Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config remote.origin.url 2025-12-04T08:28:51.3391180Z Entering 'third_party/cudnn_frontend' 2025-12-04T08:28:51.3445258Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config remote.origin.url 2025-12-04T08:28:51.3464874Z Entering 'third_party/cutlass' 2025-12-04T08:28:51.3511184Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config remote.origin.url 2025-12-04T08:28:51.3539960Z Entering 'third_party/fbgemm' 2025-12-04T08:28:51.3587753Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config remote.origin.url 2025-12-04T08:28:51.3604571Z Entering 'third_party/fbgemm/external/asmjit' 2025-12-04T08:28:51.3648153Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/asmjit/config remote.origin.url 2025-12-04T08:28:51.3670803Z Entering 'third_party/fbgemm/external/composable_kernel' 2025-12-04T08:28:51.3713812Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/composable_kernel/config remote.origin.url 2025-12-04T08:28:51.3739737Z Entering 'third_party/fbgemm/external/cpuinfo' 2025-12-04T08:28:51.3792108Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/cpuinfo/config remote.origin.url 2025-12-04T08:28:51.3810319Z Entering 'third_party/fbgemm/external/cutlass' 2025-12-04T08:28:51.3860366Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/cutlass/config remote.origin.url 2025-12-04T08:28:51.3886070Z Entering 'third_party/fbgemm/external/googletest' 2025-12-04T08:28:51.3934230Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/googletest/config remote.origin.url 2025-12-04T08:28:51.3950401Z Entering 'third_party/fbgemm/external/hipify_torch' 2025-12-04T08:28:51.3995408Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/hipify_torch/config remote.origin.url 2025-12-04T08:28:51.4009942Z Entering 'third_party/fbgemm/external/json' 2025-12-04T08:28:51.4064281Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/json/config remote.origin.url 2025-12-04T08:28:51.4087221Z Entering 'third_party/flash-attention' 2025-12-04T08:28:51.4130382Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flash-attention/config remote.origin.url 2025-12-04T08:28:51.4150791Z Entering 'third_party/flash-attention/csrc/composable_kernel' 2025-12-04T08:28:51.4201583Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flash-attention/modules/csrc/composable_kernel/config remote.origin.url 2025-12-04T08:28:51.4218502Z Entering 'third_party/flash-attention/csrc/cutlass' 2025-12-04T08:28:51.4264713Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flash-attention/modules/csrc/cutlass/config remote.origin.url 2025-12-04T08:28:51.4295447Z Entering 'third_party/flatbuffers' 2025-12-04T08:28:51.4342130Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config remote.origin.url 2025-12-04T08:28:51.4358642Z Entering 'third_party/fmt' 2025-12-04T08:28:51.4406904Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config remote.origin.url 2025-12-04T08:28:51.4419498Z Entering 'third_party/gemmlowp/gemmlowp' 2025-12-04T08:28:51.4468894Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config remote.origin.url 2025-12-04T08:28:51.4494823Z Entering 'third_party/gloo' 2025-12-04T08:28:51.4533372Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config remote.origin.url 2025-12-04T08:28:51.4548993Z Entering 'third_party/googletest' 2025-12-04T08:28:51.4596197Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config remote.origin.url 2025-12-04T08:28:51.4612278Z Entering 'third_party/ideep' 2025-12-04T08:28:51.4662730Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config remote.origin.url 2025-12-04T08:28:51.4675328Z Entering 'third_party/ideep/mkl-dnn' 2025-12-04T08:28:51.4724230Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config remote.origin.url 2025-12-04T08:28:51.4751226Z Entering 'third_party/ittapi' 2025-12-04T08:28:51.4798465Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config remote.origin.url 2025-12-04T08:28:51.4816278Z Entering 'third_party/kineto' 2025-12-04T08:28:51.4864633Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config remote.origin.url 2025-12-04T08:28:51.4879315Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2025-12-04T08:28:51.4933460Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/config remote.origin.url 2025-12-04T08:28:51.4944369Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2025-12-04T08:28:51.4995462Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/DCGM/config remote.origin.url 2025-12-04T08:28:51.5009567Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2025-12-04T08:28:51.5057278Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/cpr/config remote.origin.url 2025-12-04T08:28:51.5071185Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2025-12-04T08:28:51.5114140Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/fmt/config remote.origin.url 2025-12-04T08:28:51.5134145Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2025-12-04T08:28:51.5183085Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/config remote.origin.url 2025-12-04T08:28:51.5195289Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2025-12-04T08:28:51.5247117Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/modules/doc/config remote.origin.url 2025-12-04T08:28:51.5266276Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2025-12-04T08:28:51.5310378Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/glog/config remote.origin.url 2025-12-04T08:28:51.5325155Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2025-12-04T08:28:51.5370019Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/googletest/config remote.origin.url 2025-12-04T08:28:51.5388293Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2025-12-04T08:28:51.5433381Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/json/config remote.origin.url 2025-12-04T08:28:51.5451992Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2025-12-04T08:28:51.5505139Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/pfs/config remote.origin.url 2025-12-04T08:28:51.5522659Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp' 2025-12-04T08:28:51.5571067Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/prometheus-cpp/config remote.origin.url 2025-12-04T08:28:51.5587650Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/civetweb' 2025-12-04T08:28:51.5632271Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/prometheus-cpp/modules/civetweb/config remote.origin.url 2025-12-04T08:28:51.5653938Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/googletest' 2025-12-04T08:28:51.5701470Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/prometheus-cpp/modules/googletest/config remote.origin.url 2025-12-04T08:28:51.5714801Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2025-12-04T08:28:51.5764106Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config remote.origin.url 2025-12-04T08:28:51.5783258Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2025-12-04T08:28:51.5832201Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config remote.origin.url 2025-12-04T08:28:51.5857902Z Entering 'third_party/kleidiai' 2025-12-04T08:28:51.5904541Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kleidiai/config remote.origin.url 2025-12-04T08:28:51.5919066Z Entering 'third_party/mimalloc' 2025-12-04T08:28:51.5968308Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/mimalloc/config remote.origin.url 2025-12-04T08:28:51.5987868Z Entering 'third_party/nlohmann' 2025-12-04T08:28:51.6033533Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config remote.origin.url 2025-12-04T08:28:51.6056745Z Entering 'third_party/onnx' 2025-12-04T08:28:51.6103346Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config remote.origin.url 2025-12-04T08:28:51.6139178Z Entering 'third_party/onnx/third_party/pybind11' 2025-12-04T08:28:51.6181888Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config remote.origin.url 2025-12-04T08:28:51.6195825Z Entering 'third_party/opentelemetry-cpp' 2025-12-04T08:28:51.6244943Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/config remote.origin.url 2025-12-04T08:28:51.6265309Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2025-12-04T08:28:51.6310512Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/benchmark/config remote.origin.url 2025-12-04T08:28:51.6329305Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2025-12-04T08:28:51.6378498Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/googletest/config remote.origin.url 2025-12-04T08:28:51.6397664Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2025-12-04T08:28:51.6445661Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/ms-gsl/config remote.origin.url 2025-12-04T08:28:51.6462063Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2025-12-04T08:28:51.6509694Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/nlohmann-json/config remote.origin.url 2025-12-04T08:28:51.6530856Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2025-12-04T08:28:51.6578322Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentelemetry-proto/config remote.origin.url 2025-12-04T08:28:51.6592428Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2025-12-04T08:28:51.6643562Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentracing-cpp/config remote.origin.url 2025-12-04T08:28:51.6659246Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2025-12-04T08:28:51.6706383Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/config remote.origin.url 2025-12-04T08:28:51.6720590Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2025-12-04T08:28:51.6766929Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/civetweb/config remote.origin.url 2025-12-04T08:28:51.6790375Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2025-12-04T08:28:51.6836946Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/googletest/config remote.origin.url 2025-12-04T08:28:51.6862571Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2025-12-04T08:28:51.6911927Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/tools/vcpkg/config remote.origin.url 2025-12-04T08:28:51.6950527Z Entering 'third_party/pocketfft' 2025-12-04T08:28:51.6994688Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config remote.origin.url 2025-12-04T08:28:51.7009797Z Entering 'third_party/protobuf' 2025-12-04T08:28:51.7060783Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config remote.origin.url 2025-12-04T08:28:51.7078923Z Entering 'third_party/protobuf/third_party/benchmark' 2025-12-04T08:28:51.7126926Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config remote.origin.url 2025-12-04T08:28:51.7141423Z Entering 'third_party/protobuf/third_party/googletest' 2025-12-04T08:28:51.7189570Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config remote.origin.url 2025-12-04T08:28:51.7204767Z Entering 'third_party/psimd' 2025-12-04T08:28:51.7253531Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config remote.origin.url 2025-12-04T08:28:51.7270759Z Entering 'third_party/pthreadpool' 2025-12-04T08:28:51.7314316Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config remote.origin.url 2025-12-04T08:28:51.7330124Z Entering 'third_party/pybind11' 2025-12-04T08:28:51.7374463Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config remote.origin.url 2025-12-04T08:28:51.7393411Z Entering 'third_party/python-peachpy' 2025-12-04T08:28:51.7438825Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config remote.origin.url 2025-12-04T08:28:51.7455256Z Entering 'third_party/sleef' 2025-12-04T08:28:51.7502458Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config remote.origin.url 2025-12-04T08:28:51.7518876Z Entering 'third_party/tensorpipe' 2025-12-04T08:28:51.7567863Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config remote.origin.url 2025-12-04T08:28:51.7586642Z Entering 'third_party/tensorpipe/third_party/googletest' 2025-12-04T08:28:51.7634782Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config remote.origin.url 2025-12-04T08:28:51.7653434Z Entering 'third_party/tensorpipe/third_party/libnop' 2025-12-04T08:28:51.7696182Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config remote.origin.url 2025-12-04T08:28:51.7711353Z Entering 'third_party/tensorpipe/third_party/libuv' 2025-12-04T08:28:51.7763094Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config remote.origin.url 2025-12-04T08:28:51.7775809Z Entering 'third_party/tensorpipe/third_party/pybind11' 2025-12-04T08:28:51.7823037Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config remote.origin.url 2025-12-04T08:28:51.7840364Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2025-12-04T08:28:51.7892841Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config remote.origin.url 2025-12-04T08:28:51.8970877Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'git@github.com:' 2025-12-04T08:28:51.9298114Z Entering 'android/libs/fbjni' 2025-12-04T08:28:51.9336669Z Entering 'third_party/FP16' 2025-12-04T08:28:51.9375549Z Entering 'third_party/FXdiv' 2025-12-04T08:28:51.9415848Z Entering 'third_party/NNPACK' 2025-12-04T08:28:51.9453069Z Entering 'third_party/NVTX' 2025-12-04T08:28:51.9495838Z Entering 'third_party/VulkanMemoryAllocator' 2025-12-04T08:28:51.9538992Z Entering 'third_party/XNNPACK' 2025-12-04T08:28:51.9590397Z Entering 'third_party/aiter' 2025-12-04T08:28:51.9629261Z Entering 'third_party/aiter/3rdparty/composable_kernel' 2025-12-04T08:28:51.9681909Z Entering 'third_party/benchmark' 2025-12-04T08:28:51.9719299Z Entering 'third_party/composable_kernel' 2025-12-04T08:28:51.9770380Z Entering 'third_party/cpp-httplib' 2025-12-04T08:28:51.9810134Z Entering 'third_party/cpuinfo' 2025-12-04T08:28:51.9859238Z Entering 'third_party/cudnn_frontend' 2025-12-04T08:28:51.9891908Z Entering 'third_party/cutlass' 2025-12-04T08:28:51.9944230Z Entering 'third_party/fbgemm' 2025-12-04T08:28:51.9983682Z Entering 'third_party/fbgemm/external/asmjit' 2025-12-04T08:28:52.0017764Z Entering 'third_party/fbgemm/external/composable_kernel' 2025-12-04T08:28:52.0067286Z Entering 'third_party/fbgemm/external/cpuinfo' 2025-12-04T08:28:52.0103363Z Entering 'third_party/fbgemm/external/cutlass' 2025-12-04T08:28:52.0152902Z Entering 'third_party/fbgemm/external/googletest' 2025-12-04T08:28:52.0200448Z Entering 'third_party/fbgemm/external/hipify_torch' 2025-12-04T08:28:52.0232383Z Entering 'third_party/fbgemm/external/json' 2025-12-04T08:28:52.0279583Z Entering 'third_party/flash-attention' 2025-12-04T08:28:52.0320357Z Entering 'third_party/flash-attention/csrc/composable_kernel' 2025-12-04T08:28:52.0370402Z Entering 'third_party/flash-attention/csrc/cutlass' 2025-12-04T08:28:52.0418225Z Entering 'third_party/flatbuffers' 2025-12-04T08:28:52.0459309Z Entering 'third_party/fmt' 2025-12-04T08:28:52.0497147Z Entering 'third_party/gemmlowp/gemmlowp' 2025-12-04T08:28:52.0536649Z Entering 'third_party/gloo' 2025-12-04T08:28:52.0582905Z Entering 'third_party/googletest' 2025-12-04T08:28:52.0621208Z Entering 'third_party/ideep' 2025-12-04T08:28:52.0667400Z Entering 'third_party/ideep/mkl-dnn' 2025-12-04T08:28:52.0712872Z Entering 'third_party/ittapi' 2025-12-04T08:28:52.0754763Z Entering 'third_party/kineto' 2025-12-04T08:28:52.0796942Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2025-12-04T08:28:52.0836420Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2025-12-04T08:28:52.0879054Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2025-12-04T08:28:52.0917706Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2025-12-04T08:28:52.0963005Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2025-12-04T08:28:52.0994738Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2025-12-04T08:28:52.1034777Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2025-12-04T08:28:52.1077418Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2025-12-04T08:28:52.1117244Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2025-12-04T08:28:52.1154772Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2025-12-04T08:28:52.1197055Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp' 2025-12-04T08:28:52.1233918Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/civetweb' 2025-12-04T08:28:52.1277005Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/googletest' 2025-12-04T08:28:52.1321305Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2025-12-04T08:28:52.1365139Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2025-12-04T08:28:52.1411165Z Entering 'third_party/kleidiai' 2025-12-04T08:28:52.1451922Z Entering 'third_party/mimalloc' 2025-12-04T08:28:52.1489372Z Entering 'third_party/nlohmann' 2025-12-04T08:28:52.1530071Z Entering 'third_party/onnx' 2025-12-04T08:28:52.1580991Z Entering 'third_party/onnx/third_party/pybind11' 2025-12-04T08:28:52.1619774Z Entering 'third_party/opentelemetry-cpp' 2025-12-04T08:28:52.1663703Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2025-12-04T08:28:52.1700336Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2025-12-04T08:28:52.1744026Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2025-12-04T08:28:52.1778957Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2025-12-04T08:28:52.1818091Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2025-12-04T08:28:52.1857930Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2025-12-04T08:28:52.1897355Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2025-12-04T08:28:52.1936463Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2025-12-04T08:28:52.1976399Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2025-12-04T08:28:52.2011962Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2025-12-04T08:28:52.2069632Z Entering 'third_party/pocketfft' 2025-12-04T08:28:52.2105215Z Entering 'third_party/protobuf' 2025-12-04T08:28:52.2146164Z Entering 'third_party/protobuf/third_party/benchmark' 2025-12-04T08:28:52.2195795Z Entering 'third_party/protobuf/third_party/googletest' 2025-12-04T08:28:52.2236864Z Entering 'third_party/psimd' 2025-12-04T08:28:52.2275567Z Entering 'third_party/pthreadpool' 2025-12-04T08:28:52.2319975Z Entering 'third_party/pybind11' 2025-12-04T08:28:52.2356348Z Entering 'third_party/python-peachpy' 2025-12-04T08:28:52.2397343Z Entering 'third_party/sleef' 2025-12-04T08:28:52.2437892Z Entering 'third_party/tensorpipe' 2025-12-04T08:28:52.2477065Z Entering 'third_party/tensorpipe/third_party/googletest' 2025-12-04T08:28:52.2512072Z Entering 'third_party/tensorpipe/third_party/libnop' 2025-12-04T08:28:52.2554038Z Entering 'third_party/tensorpipe/third_party/libuv' 2025-12-04T08:28:52.2593053Z Entering 'third_party/tensorpipe/third_party/pybind11' 2025-12-04T08:28:52.2631909Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2025-12-04T08:28:52.2691892Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'org-21003710@github.com:' 2025-12-04T08:28:52.2994329Z Entering 'android/libs/fbjni' 2025-12-04T08:28:52.3033911Z Entering 'third_party/FP16' 2025-12-04T08:28:52.3072357Z Entering 'third_party/FXdiv' 2025-12-04T08:28:52.3110486Z Entering 'third_party/NNPACK' 2025-12-04T08:28:52.3155765Z Entering 'third_party/NVTX' 2025-12-04T08:28:52.3195846Z Entering 'third_party/VulkanMemoryAllocator' 2025-12-04T08:28:52.3235121Z Entering 'third_party/XNNPACK' 2025-12-04T08:28:52.3285713Z Entering 'third_party/aiter' 2025-12-04T08:28:52.3326499Z Entering 'third_party/aiter/3rdparty/composable_kernel' 2025-12-04T08:28:52.3379389Z Entering 'third_party/benchmark' 2025-12-04T08:28:52.3420676Z Entering 'third_party/composable_kernel' 2025-12-04T08:28:52.3467322Z Entering 'third_party/cpp-httplib' 2025-12-04T08:28:52.3508567Z Entering 'third_party/cpuinfo' 2025-12-04T08:28:52.3552306Z Entering 'third_party/cudnn_frontend' 2025-12-04T08:28:52.3596483Z Entering 'third_party/cutlass' 2025-12-04T08:28:52.3643595Z Entering 'third_party/fbgemm' 2025-12-04T08:28:52.3691153Z Entering 'third_party/fbgemm/external/asmjit' 2025-12-04T08:28:52.3734116Z Entering 'third_party/fbgemm/external/composable_kernel' 2025-12-04T08:28:52.3780663Z Entering 'third_party/fbgemm/external/cpuinfo' 2025-12-04T08:28:52.3818485Z Entering 'third_party/fbgemm/external/cutlass' 2025-12-04T08:28:52.3863349Z Entering 'third_party/fbgemm/external/googletest' 2025-12-04T08:28:52.3902895Z Entering 'third_party/fbgemm/external/hipify_torch' 2025-12-04T08:28:52.3940167Z Entering 'third_party/fbgemm/external/json' 2025-12-04T08:28:52.3983619Z Entering 'third_party/flash-attention' 2025-12-04T08:28:52.4018951Z Entering 'third_party/flash-attention/csrc/composable_kernel' 2025-12-04T08:28:52.4064324Z Entering 'third_party/flash-attention/csrc/cutlass' 2025-12-04T08:28:52.4109410Z Entering 'third_party/flatbuffers' 2025-12-04T08:28:52.4155663Z Entering 'third_party/fmt' 2025-12-04T08:28:52.4197434Z Entering 'third_party/gemmlowp/gemmlowp' 2025-12-04T08:28:52.4236841Z Entering 'third_party/gloo' 2025-12-04T08:28:52.4273156Z Entering 'third_party/googletest' 2025-12-04T08:28:52.4313632Z Entering 'third_party/ideep' 2025-12-04T08:28:52.4346792Z Entering 'third_party/ideep/mkl-dnn' 2025-12-04T08:28:52.4398868Z Entering 'third_party/ittapi' 2025-12-04T08:28:52.4451019Z Entering 'third_party/kineto' 2025-12-04T08:28:52.4491347Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2025-12-04T08:28:52.4529629Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2025-12-04T08:28:52.4575258Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2025-12-04T08:28:52.4612869Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2025-12-04T08:28:52.4660383Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2025-12-04T08:28:52.4700904Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2025-12-04T08:28:52.4752562Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2025-12-04T08:28:52.4792401Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2025-12-04T08:28:52.4833502Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2025-12-04T08:28:52.4873931Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2025-12-04T08:28:52.4918218Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp' 2025-12-04T08:28:52.4954518Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/civetweb' 2025-12-04T08:28:52.4993710Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/googletest' 2025-12-04T08:28:52.5042850Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2025-12-04T08:28:52.5082525Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2025-12-04T08:28:52.5121092Z Entering 'third_party/kleidiai' 2025-12-04T08:28:52.5163492Z Entering 'third_party/mimalloc' 2025-12-04T08:28:52.5206700Z Entering 'third_party/nlohmann' 2025-12-04T08:28:52.5246695Z Entering 'third_party/onnx' 2025-12-04T08:28:52.5298112Z Entering 'third_party/onnx/third_party/pybind11' 2025-12-04T08:28:52.5340780Z Entering 'third_party/opentelemetry-cpp' 2025-12-04T08:28:52.5383073Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2025-12-04T08:28:52.5417655Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2025-12-04T08:28:52.5454536Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2025-12-04T08:28:52.5491454Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2025-12-04T08:28:52.5530149Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2025-12-04T08:28:52.5573128Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2025-12-04T08:28:52.5608482Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2025-12-04T08:28:52.5654438Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2025-12-04T08:28:52.5691689Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2025-12-04T08:28:52.5733110Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2025-12-04T08:28:52.5794017Z Entering 'third_party/pocketfft' 2025-12-04T08:28:52.5832989Z Entering 'third_party/protobuf' 2025-12-04T08:28:52.5874747Z Entering 'third_party/protobuf/third_party/benchmark' 2025-12-04T08:28:52.5913131Z Entering 'third_party/protobuf/third_party/googletest' 2025-12-04T08:28:52.5958795Z Entering 'third_party/psimd' 2025-12-04T08:28:52.5997786Z Entering 'third_party/pthreadpool' 2025-12-04T08:28:52.6044019Z Entering 'third_party/pybind11' 2025-12-04T08:28:52.6088277Z Entering 'third_party/python-peachpy' 2025-12-04T08:28:52.6124263Z Entering 'third_party/sleef' 2025-12-04T08:28:52.6167340Z Entering 'third_party/tensorpipe' 2025-12-04T08:28:52.6205333Z Entering 'third_party/tensorpipe/third_party/googletest' 2025-12-04T08:28:52.6250427Z Entering 'third_party/tensorpipe/third_party/libnop' 2025-12-04T08:28:52.6287012Z Entering 'third_party/tensorpipe/third_party/libuv' 2025-12-04T08:28:52.6322457Z Entering 'third_party/tensorpipe/third_party/pybind11' 2025-12-04T08:28:52.6372200Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2025-12-04T08:28:52.6420334Z ##[endgroup] 2025-12-04T08:28:52.6463749Z [command]/usr/bin/git log -1 --format=%H 2025-12-04T08:28:52.6489436Z ffd9b0fb4355e97af82fc42cf185c3ffa0fc0a32 2025-12-04T08:28:52.6600850Z ##[group]Run cd "${GITHUB_WORKSPACE}" 2025-12-04T08:28:52.6601138Z cd "${GITHUB_WORKSPACE}" 2025-12-04T08:28:52.6601471Z # Clean stale submodule dirs 2025-12-04T08:28:52.6601706Z if [ -z "${NO_SUDO}" ]; then 2025-12-04T08:28:52.6601984Z  sudo git submodule foreach --recursive git clean -ffdx 2025-12-04T08:28:52.6602247Z else 2025-12-04T08:28:52.6602480Z  git submodule foreach --recursive git clean -ffdx 2025-12-04T08:28:52.6602737Z fi 2025-12-04T08:28:52.6610017Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T08:28:52.6610289Z env: 2025-12-04T08:28:52.6610463Z GIT_DEFAULT_BRANCH: main 2025-12-04T08:28:52.6610649Z NO_SUDO: true 2025-12-04T08:28:52.6610828Z ##[endgroup] 2025-12-04T08:28:52.6935796Z Entering 'android/libs/fbjni' 2025-12-04T08:28:52.6965238Z Entering 'third_party/FP16' 2025-12-04T08:28:52.6997160Z Entering 'third_party/FXdiv' 2025-12-04T08:28:52.7028576Z Entering 'third_party/NNPACK' 2025-12-04T08:28:52.7066394Z Entering 'third_party/NVTX' 2025-12-04T08:28:52.7103149Z Entering 'third_party/VulkanMemoryAllocator' 2025-12-04T08:28:52.7137461Z Entering 'third_party/XNNPACK' 2025-12-04T08:28:52.7237274Z Entering 'third_party/aiter' 2025-12-04T08:28:52.7273335Z Entering 'third_party/aiter/3rdparty/composable_kernel' 2025-12-04T08:28:52.7373372Z Entering 'third_party/benchmark' 2025-12-04T08:28:52.7403274Z Entering 'third_party/composable_kernel' 2025-12-04T08:28:52.7496207Z Entering 'third_party/cpp-httplib' 2025-12-04T08:28:52.7534913Z Entering 'third_party/cpuinfo' 2025-12-04T08:28:52.7571945Z Entering 'third_party/cudnn_frontend' 2025-12-04T08:28:52.7610655Z Entering 'third_party/cutlass' 2025-12-04T08:28:52.7693933Z Entering 'third_party/fbgemm' 2025-12-04T08:28:52.7747234Z Entering 'third_party/fbgemm/external/asmjit' 2025-12-04T08:28:52.7780118Z Entering 'third_party/fbgemm/external/composable_kernel' 2025-12-04T08:28:52.7873884Z Entering 'third_party/fbgemm/external/cpuinfo' 2025-12-04T08:28:52.7908377Z Entering 'third_party/fbgemm/external/cutlass' 2025-12-04T08:28:52.7992267Z Entering 'third_party/fbgemm/external/googletest' 2025-12-04T08:28:52.8025466Z Entering 'third_party/fbgemm/external/hipify_torch' 2025-12-04T08:28:52.8058102Z Entering 'third_party/fbgemm/external/json' 2025-12-04T08:28:52.8102621Z Entering 'third_party/flash-attention' 2025-12-04T08:28:52.8139099Z Entering 'third_party/flash-attention/csrc/composable_kernel' 2025-12-04T08:28:52.8215229Z Entering 'third_party/flash-attention/csrc/cutlass' 2025-12-04T08:28:52.8287719Z Entering 'third_party/flatbuffers' 2025-12-04T08:28:52.8343375Z Entering 'third_party/fmt' 2025-12-04T08:28:52.8375301Z Entering 'third_party/gemmlowp/gemmlowp' 2025-12-04T08:28:52.8411406Z Entering 'third_party/gloo' 2025-12-04T08:28:52.8447185Z Entering 'third_party/googletest' 2025-12-04T08:28:52.8479154Z Entering 'third_party/ideep' 2025-12-04T08:28:52.8506850Z Entering 'third_party/ideep/mkl-dnn' 2025-12-04T08:28:52.8577499Z Entering 'third_party/ittapi' 2025-12-04T08:28:52.8608909Z Entering 'third_party/kineto' 2025-12-04T08:28:52.8643822Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2025-12-04T08:28:52.8677416Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2025-12-04T08:28:52.8718104Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2025-12-04T08:28:52.8753836Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2025-12-04T08:28:52.8782483Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2025-12-04T08:28:52.8810687Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2025-12-04T08:28:52.8848026Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2025-12-04T08:28:52.8882358Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2025-12-04T08:28:52.8912069Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2025-12-04T08:28:52.8956810Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2025-12-04T08:28:52.8981708Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp' 2025-12-04T08:28:52.9010678Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/civetweb' 2025-12-04T08:28:52.9052700Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/googletest' 2025-12-04T08:28:52.9092454Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2025-12-04T08:28:52.9121663Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2025-12-04T08:28:52.9157808Z Entering 'third_party/kleidiai' 2025-12-04T08:28:52.9194753Z Entering 'third_party/mimalloc' 2025-12-04T08:28:52.9227719Z Entering 'third_party/nlohmann' 2025-12-04T08:28:52.9272641Z Entering 'third_party/onnx' 2025-12-04T08:28:52.9519473Z Entering 'third_party/onnx/third_party/pybind11' 2025-12-04T08:28:52.9560185Z Entering 'third_party/opentelemetry-cpp' 2025-12-04T08:28:52.9608635Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2025-12-04T08:28:52.9637501Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2025-12-04T08:28:52.9670853Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2025-12-04T08:28:52.9700455Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2025-12-04T08:28:52.9737630Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2025-12-04T08:28:52.9766715Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2025-12-04T08:28:52.9794729Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2025-12-04T08:28:52.9821766Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2025-12-04T08:28:52.9863434Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2025-12-04T08:28:52.9900815Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2025-12-04T08:28:53.0109562Z Entering 'third_party/pocketfft' 2025-12-04T08:28:53.0136603Z Entering 'third_party/protobuf' 2025-12-04T08:28:53.0203131Z Entering 'third_party/protobuf/third_party/benchmark' 2025-12-04T08:28:53.0230724Z Entering 'third_party/protobuf/third_party/googletest' 2025-12-04T08:28:53.0266087Z Entering 'third_party/psimd' 2025-12-04T08:28:53.0298368Z Entering 'third_party/pthreadpool' 2025-12-04T08:28:53.0331050Z Entering 'third_party/pybind11' 2025-12-04T08:28:53.0366090Z Entering 'third_party/python-peachpy' 2025-12-04T08:28:53.0395054Z Entering 'third_party/sleef' 2025-12-04T08:28:53.0431648Z Entering 'third_party/tensorpipe' 2025-12-04T08:28:53.0465510Z Entering 'third_party/tensorpipe/third_party/googletest' 2025-12-04T08:28:53.0496039Z Entering 'third_party/tensorpipe/third_party/libnop' 2025-12-04T08:28:53.0527669Z Entering 'third_party/tensorpipe/third_party/libuv' 2025-12-04T08:28:53.0558678Z Entering 'third_party/tensorpipe/third_party/pybind11' 2025-12-04T08:28:53.0593150Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2025-12-04T08:28:53.0721953Z Prepare all required actions 2025-12-04T08:28:53.0722764Z Getting action download info 2025-12-04T08:28:53.2402339Z ##[group]Run ./.github/actions/setup-linux 2025-12-04T08:28:53.2402580Z env: 2025-12-04T08:28:53.2402789Z GIT_DEFAULT_BRANCH: main 2025-12-04T08:28:53.2402996Z ##[endgroup] 2025-12-04T08:28:53.2441919Z ##[group]Run set -euo pipefail 2025-12-04T08:28:53.2442208Z set -euo pipefail 2025-12-04T08:28:53.2442435Z function get_ec2_metadata() { 2025-12-04T08:28:53.2442704Z  # Pulled from instance metadata endpoint for EC2 2025-12-04T08:28:53.2443099Z  # see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html 2025-12-04T08:28:53.2443448Z  category=$1 2025-12-04T08:28:53.2443687Z  # If it is GCP runner (runner name contains gcp), do not run this 2025-12-04T08:28:53.2443959Z  runner_name_str=i-02dcdd5317aed8049 2025-12-04T08:28:53.2444240Z  if [[ -f /.inarc ]]; then 2025-12-04T08:28:53.2444611Z  echo "ARC Runner, no info on ec2 metadata" 2025-12-04T08:28:53.2444859Z  elif [[ $runner_name_str == *"gcp"* ]]; then 2025-12-04T08:28:53.2445148Z  echo "Runner is from Google Cloud Platform, No info on ec2 metadata" 2025-12-04T08:28:53.2445411Z  else 2025-12-04T08:28:53.2445923Z  curl -H "X-aws-ec2-metadata-token: $(curl -s -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 30")" -fsSL "http://169.254.169.254/latest/meta-data/${category}" 2025-12-04T08:28:53.2446447Z  fi 2025-12-04T08:28:53.2446602Z } 2025-12-04T08:28:53.2446789Z echo "ami-id: $(get_ec2_metadata ami-id)" 2025-12-04T08:28:53.2447051Z echo "instance-id: $(get_ec2_metadata instance-id)" 2025-12-04T08:28:53.2447353Z echo "instance-type: $(get_ec2_metadata instance-type)" 2025-12-04T08:28:53.2447621Z echo "system info $(uname -a)" 2025-12-04T08:28:53.2453243Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T08:28:53.2453494Z env: 2025-12-04T08:28:53.2453660Z GIT_DEFAULT_BRANCH: main 2025-12-04T08:28:53.2453835Z ##[endgroup] 2025-12-04T08:28:53.2591625Z ami-id: ami-08982f1c5bf93d976 2025-12-04T08:28:53.2686044Z instance-id: i-02dcdd5317aed8049 2025-12-04T08:28:53.2792624Z instance-type: m7i-flex.2xlarge 2025-12-04T08:28:53.2805047Z system info Linux ip-10-0-12-49.ec2.internal 6.1.150-174.273.amzn2023.x86_64 #1 SMP PREEMPT_DYNAMIC Tue Sep 9 12:21:26 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux 2025-12-04T08:28:53.2928852Z ##[group]Run if [ -f /usr/bin/nvidia-smi ]; then nvidia-smi; fi 2025-12-04T08:28:53.2929239Z if [ -f /usr/bin/nvidia-smi ]; then nvidia-smi; fi 2025-12-04T08:28:53.2933959Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T08:28:53.2934213Z env: 2025-12-04T08:28:53.2934366Z GIT_DEFAULT_BRANCH: main 2025-12-04T08:28:53.2934553Z ##[endgroup] 2025-12-04T08:28:53.2984284Z ##[group]Run echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2025-12-04T08:28:53.2984932Z echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2025-12-04T08:28:53.2989289Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T08:28:53.2989531Z env: 2025-12-04T08:28:53.2989695Z GIT_DEFAULT_BRANCH: main 2025-12-04T08:28:53.2989884Z ##[endgroup] 2025-12-04T08:28:53.3044971Z ##[group]Run if systemctl is-active --quiet docker; then 2025-12-04T08:28:53.3045291Z if systemctl is-active --quiet docker; then 2025-12-04T08:28:53.3045542Z  echo "Docker daemon is running..."; 2025-12-04T08:28:53.3045766Z else 2025-12-04T08:28:53.3046010Z  echo "Starting docker daemon..." && sudo systemctl start docker; 2025-12-04T08:28:53.3046279Z fi 2025-12-04T08:28:53.3050302Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T08:28:53.3050558Z env: 2025-12-04T08:28:53.3050719Z GIT_DEFAULT_BRANCH: main 2025-12-04T08:28:53.3050892Z ##[endgroup] 2025-12-04T08:28:53.3129119Z Docker daemon is running... 2025-12-04T08:28:53.3173748Z ##[group]Run nick-fields/retry@v3.0.0 2025-12-04T08:28:53.3173978Z with: 2025-12-04T08:28:53.3174145Z shell: bash 2025-12-04T08:28:53.3174322Z timeout_minutes: 5 2025-12-04T08:28:53.3174507Z max_attempts: 3 2025-12-04T08:28:53.3174686Z retry_wait_seconds: 30 2025-12-04T08:28:53.3176173Z command: AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\") aws ecr get-login-password --region "$AWS_DEFAULT_REGION" | docker login --username AWS \ --password-stdin "$AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com" # For LF Runners we need to make sure we also login to Meta's ECR docker registry too. META_AWS_ACCOUNT_ID=308535385114 if [ "$AWS_ACCOUNT_ID" != "$META_AWS_ACCOUNT_ID" ] ; then aws ecr get-login-password --region "$AWS_DEFAULT_REGION" | docker login --username AWS \ --password-stdin "$META_AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com" fi 2025-12-04T08:28:53.3177742Z polling_interval_seconds: 1 2025-12-04T08:28:53.3177957Z warning_on_retry: true 2025-12-04T08:28:53.3178155Z continue_on_error: false 2025-12-04T08:28:53.3178347Z env: 2025-12-04T08:28:53.3178520Z GIT_DEFAULT_BRANCH: main 2025-12-04T08:28:53.3178726Z AWS_RETRY_MODE: standard 2025-12-04T08:28:53.3178918Z AWS_MAX_ATTEMPTS: 5 2025-12-04T08:28:53.3179116Z AWS_DEFAULT_REGION: us-east-1 2025-12-04T08:28:53.3179336Z ##[endgroup] 2025-12-04T08:28:54.3085411Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2025-12-04T08:28:54.3085954Z Configure a credential helper to remove this warning. See 2025-12-04T08:28:54.3086424Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2025-12-04T08:28:54.3086757Z 2025-12-04T08:28:54.3086834Z Login Succeeded 2025-12-04T08:28:54.4477542Z Command completed after 1 attempt(s). 2025-12-04T08:28:54.4546834Z ##[group]Run env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2025-12-04T08:28:54.4547181Z env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2025-12-04T08:28:54.4547480Z env | grep '^CI' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2025-12-04T08:28:54.4555254Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T08:28:54.4555497Z env: 2025-12-04T08:28:54.4555662Z GIT_DEFAULT_BRANCH: main 2025-12-04T08:28:54.4555850Z ##[endgroup] 2025-12-04T08:28:54.4632970Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2025-12-04T08:28:54.4633340Z # ignore expansion of "docker ps -q" since it could be empty 2025-12-04T08:28:54.4633625Z # shellcheck disable=SC2046 2025-12-04T08:28:54.4633850Z docker stop $(docker ps -q) || true 2025-12-04T08:28:54.4634070Z # Prune all of the docker images 2025-12-04T08:28:54.4634285Z docker system prune -af 2025-12-04T08:28:54.4639590Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T08:28:54.4639859Z env: 2025-12-04T08:28:54.4640015Z GIT_DEFAULT_BRANCH: main 2025-12-04T08:28:54.4640202Z ##[endgroup] 2025-12-04T08:28:54.5116033Z "docker stop" requires at least 1 argument. 2025-12-04T08:28:54.5116365Z See 'docker stop --help'. 2025-12-04T08:28:54.5116505Z 2025-12-04T08:28:54.5116636Z Usage: docker stop [OPTIONS] CONTAINER [CONTAINER...] 2025-12-04T08:28:54.5116834Z 2025-12-04T08:28:54.5116926Z Stop one or more running containers 2025-12-04T08:28:54.5331324Z Total reclaimed space: 0B 2025-12-04T08:28:54.5465019Z ##[group]Run pytorch/test-infra/.github/actions/calculate-docker-image@main 2025-12-04T08:28:54.5465324Z with: 2025-12-04T08:28:54.5465836Z docker-image-name: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-f0cd68561080d537ef3d3d6f81b25a6416ad600a 2025-12-04T08:28:54.5466401Z use-custom-docker-registry: true 2025-12-04T08:28:54.5466632Z docker-build-dir: .ci/docker 2025-12-04T08:28:54.5466833Z docker-build-script: ./build.sh 2025-12-04T08:28:54.5467025Z working-directory: . 2025-12-04T08:28:54.5467258Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-12-04T08:28:54.5467512Z force-push: false 2025-12-04T08:28:54.5467664Z env: 2025-12-04T08:28:54.5467815Z GIT_DEFAULT_BRANCH: main 2025-12-04T08:28:54.5467992Z ##[endgroup] 2025-12-04T08:28:54.5491899Z ##[group]Run set -ex 2025-12-04T08:28:54.5492120Z set -ex 2025-12-04T08:28:54.5492279Z  2025-12-04T08:28:54.5492585Z # If the docker build directory or the build script doesn't exist, the action will 2025-12-04T08:28:54.5492997Z # gracefully return the docker image name as it is. Pulling docker image in Linux 2025-12-04T08:28:54.5493346Z # job could then download the pre-built image as usual 2025-12-04T08:28:54.5493766Z if [[ -d "${DOCKER_BUILD_DIR}" ]] && [[ -f "${DOCKER_BUILD_DIR}/${DOCKER_BUILD_SCRIPT}" ]] && [[ "${USE_CUSTOM_DOCKER_REGISTRY}" == "true" ]]; then 2025-12-04T08:28:54.5494245Z  echo "skip=false" >> "${GITHUB_OUTPUT}" 2025-12-04T08:28:54.5494460Z else 2025-12-04T08:28:54.5494641Z  echo "skip=true" >> "${GITHUB_OUTPUT}" 2025-12-04T08:28:54.5494922Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2025-12-04T08:28:54.5495168Z  2025-12-04T08:28:54.5495518Z  echo "Not using custom ECR registry. Either it was not requested or there is no Docker build script in the ${REPO_NAME} repo..." 2025-12-04T08:28:54.5495901Z  exit 0 2025-12-04T08:28:54.5496054Z fi 2025-12-04T08:28:54.5496207Z  2025-12-04T08:28:54.5496441Z if [[ "${DOCKER_IMAGE_NAME}" == *"${DOCKER_REGISTRY}/${REPO_NAME}"* ]]; then 2025-12-04T08:28:54.5496812Z  # The docker image name already includes the ECR prefix and tag, so we can just 2025-12-04T08:28:54.5497143Z  # use it as it is, but first let's extract the tag 2025-12-04T08:28:54.5497457Z  DOCKER_TAG=$(echo "${DOCKER_IMAGE_NAME}" | awk -F '[:,]' '{print $2}') 2025-12-04T08:28:54.5497790Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2025-12-04T08:28:54.5498110Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2025-12-04T08:28:54.5498378Z else 2025-12-04T08:28:54.5498578Z  if [[ "${DOCKER_IMAGE_NAME}" == *:* ]]; then 2025-12-04T08:28:54.5498847Z  CUSTOM_TAG_PREFIX=${DOCKER_IMAGE_NAME#*:} 2025-12-04T08:28:54.5499114Z  DOCKER_IMAGE_NAME=${DOCKER_IMAGE_NAME%%:*} 2025-12-04T08:28:54.5499334Z  fi 2025-12-04T08:28:54.5499634Z  DOCKER_TAG=${CUSTOM_TAG_PREFIX:+${CUSTOM_TAG_PREFIX}-}$(git rev-parse HEAD:"${DOCKER_BUILD_DIR}") 2025-12-04T08:28:54.5499999Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2025-12-04T08:28:54.5500377Z  echo "docker-image=${DOCKER_REGISTRY}/${REPO_NAME}/${DOCKER_IMAGE_NAME}:${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2025-12-04T08:28:54.5500795Z  echo "custom-tag-prefix=${CUSTOM_TAG_PREFIX}" >> "${GITHUB_OUTPUT}" 2025-12-04T08:28:54.5501063Z fi 2025-12-04T08:28:54.5505827Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T08:28:54.5506075Z env: 2025-12-04T08:28:54.5506239Z GIT_DEFAULT_BRANCH: main 2025-12-04T08:28:54.5506420Z REPO_NAME: pytorch 2025-12-04T08:28:54.5507056Z DOCKER_IMAGE_NAME: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-f0cd68561080d537ef3d3d6f81b25a6416ad600a 2025-12-04T08:28:54.5507627Z DOCKER_BUILD_DIR: .ci/docker 2025-12-04T08:28:54.5507834Z DOCKER_BUILD_SCRIPT: ./build.sh 2025-12-04T08:28:54.5508093Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-12-04T08:28:54.5508365Z USE_CUSTOM_DOCKER_REGISTRY: true 2025-12-04T08:28:54.5508575Z CUSTOM_TAG_PREFIX: 2025-12-04T08:28:54.5508751Z ##[endgroup] 2025-12-04T08:28:54.5533462Z + [[ -d .ci/docker ]] 2025-12-04T08:28:54.5535601Z + [[ -f .ci/docker/./build.sh ]] 2025-12-04T08:28:54.5535876Z + [[ true == \t\r\u\e ]] 2025-12-04T08:28:54.5536082Z + echo skip=false 2025-12-04T08:28:54.5537022Z + [[ 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-f0cd68561080d537ef3d3d6f81b25a6416ad600a == *\3\0\8\5\3\5\3\8\5\1\1\4\.\d\k\r\.\e\c\r\.\u\s\-\e\a\s\t\-\1\.\a\m\a\z\o\n\a\w\s\.\c\o\m\/\p\y\t\o\r\c\h* ]] 2025-12-04T08:28:54.5538874Z ++ echo 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-f0cd68561080d537ef3d3d6f81b25a6416ad600a 2025-12-04T08:28:54.5541866Z ++ awk -F '[:,]' '{print $2}' 2025-12-04T08:28:54.5639470Z + DOCKER_TAG=pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-f0cd68561080d537ef3d3d6f81b25a6416ad600a 2025-12-04T08:28:54.5640513Z + echo docker-tag=pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-f0cd68561080d537ef3d3d6f81b25a6416ad600a 2025-12-04T08:28:54.5641755Z + echo docker-image=308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-f0cd68561080d537ef3d3d6f81b25a6416ad600a 2025-12-04T08:28:54.5693139Z ##[group]Run set +e 2025-12-04T08:28:54.5693371Z set +e 2025-12-04T08:28:54.5693534Z set -x 2025-12-04T08:28:54.5693693Z  2025-12-04T08:28:54.5693847Z login() { 2025-12-04T08:28:54.5694162Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2025-12-04T08:28:54.5694504Z } 2025-12-04T08:28:54.5694656Z  2025-12-04T08:28:54.5694800Z retry () { 2025-12-04T08:28:54.5694989Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2025-12-04T08:28:54.5695201Z } 2025-12-04T08:28:54.5695340Z  2025-12-04T08:28:54.5695501Z retry login "${DOCKER_REGISTRY}" 2025-12-04T08:28:54.5695703Z  2025-12-04T08:28:54.5695860Z START_TIME=$(date +%s) 2025-12-04T08:28:54.5696066Z # Wait up to 120 minutes 2025-12-04T08:28:54.5696314Z while [[ $(( $(date +%s) - 7200 )) -lt $START_TIME ]]; do 2025-12-04T08:28:54.5696629Z  # Check if image already exists, if it does then skip building it 2025-12-04T08:28:54.5696937Z  if docker manifest inspect "${DOCKER_IMAGE}"; then 2025-12-04T08:28:54.5697179Z  exit 0 2025-12-04T08:28:54.5697342Z  fi 2025-12-04T08:28:54.5697489Z  2025-12-04T08:28:54.5697746Z  # NB: This flag is used by Docker build workflow to push the image to ECR, so we can 2025-12-04T08:28:54.5698161Z  # use this to differentiate between the Docker build and regular build jobs. For the 2025-12-04T08:28:54.5698598Z  # latter, it will wait for the Docker images to become available before continuing 2025-12-04T08:28:54.5698945Z  if [ "${DOCKER_PUSH:-false}" == "true" ]; then 2025-12-04T08:28:54.5699233Z  # It's a Docker build job, let's build the image 2025-12-04T08:28:54.5699480Z  break 2025-12-04T08:28:54.5699644Z  else 2025-12-04T08:28:54.5699869Z  # It's a regular build job, wait for the image to become available 2025-12-04T08:28:54.5700140Z  sleep 300 2025-12-04T08:28:54.5700323Z  fi 2025-12-04T08:28:54.5700478Z done 2025-12-04T08:28:54.5700668Z  2025-12-04T08:28:54.5700908Z # NB: This part requires a full checkout. Otherwise, the merge base will 2025-12-04T08:28:54.5701408Z # be empty. The default action would be to continue rebuild the image 2025-12-04T08:28:54.5701760Z if [[ "$BASE_REVISION" = "$(git rev-parse HEAD)" ]]; then 2025-12-04T08:28:54.5702077Z  # if we're on the base branch then use the parent commit 2025-12-04T08:28:54.5702359Z  MERGE_BASE=$(git rev-parse HEAD~) 2025-12-04T08:28:54.5702587Z else 2025-12-04T08:28:54.5702826Z  # otherwise we're on a PR, so use the most recent base commit 2025-12-04T08:28:54.5703141Z  MERGE_BASE=$(git merge-base HEAD "$BASE_REVISION") 2025-12-04T08:28:54.5703390Z fi 2025-12-04T08:28:54.5703554Z  2025-12-04T08:28:54.5703757Z if [[ -z "${MERGE_BASE}" ]]; then 2025-12-04T08:28:54.5704056Z  echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2025-12-04T08:28:54.5704273Z  2025-12-04T08:28:54.5704571Z  echo "Finding merge base only works with full checkout, please set fetch-depth to 0, continuing ..." 2025-12-04T08:28:54.5704901Z  exit 0 2025-12-04T08:28:54.5705058Z fi 2025-12-04T08:28:54.5705209Z  2025-12-04T08:28:54.5705410Z if ! git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}"; then 2025-12-04T08:28:54.5705832Z  echo "Directory '${DOCKER_BUILD_DIR}' not found in commit $MERGE_BASE, you should rebase onto a more recent commit" 2025-12-04T08:28:54.5706252Z  exit 1 2025-12-04T08:28:54.5706406Z fi 2025-12-04T08:28:54.5706544Z  2025-12-04T08:28:54.5706783Z PREVIOUS_DOCKER_TAG=$(git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}") 2025-12-04T08:28:54.5707182Z # If no image exists but the hash is the same as the previous hash then we should error out here 2025-12-04T08:28:54.5707543Z if [[ "${PREVIOUS_DOCKER_TAG}" == "${DOCKER_TAG}" ]]; then 2025-12-04T08:28:54.5707963Z  echo "WARNING: Something has gone wrong and the previous image isn't available for the merge-base of your branch" 2025-12-04T08:28:54.5708425Z  echo " Will re-build docker image to store in local cache, TTS may be longer" 2025-12-04T08:28:54.5708707Z fi 2025-12-04T08:28:54.5708846Z  2025-12-04T08:28:54.5709025Z echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2025-12-04T08:28:54.5713581Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T08:28:54.5713823Z env: 2025-12-04T08:28:54.5713993Z GIT_DEFAULT_BRANCH: main 2025-12-04T08:28:54.5714189Z DOCKER_BUILD_DIR: .ci/docker 2025-12-04T08:28:54.5714433Z BASE_REVISION: ffd9b0fb4355e97af82fc42cf185c3ffa0fc0a32 2025-12-04T08:28:54.5715041Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-f0cd68561080d537ef3d3d6f81b25a6416ad600a 2025-12-04T08:28:54.5715826Z DOCKER_TAG: pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-f0cd68561080d537ef3d3d6f81b25a6416ad600a 2025-12-04T08:28:54.5716312Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-12-04T08:28:54.5716581Z DOCKER_PUSH: 2025-12-04T08:28:54.5716751Z ##[endgroup] 2025-12-04T08:28:54.5738120Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-12-04T08:28:54.5738670Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-12-04T08:28:54.5744721Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-12-04T08:28:54.5745139Z + aws ecr get-login-password --region us-east-1 2025-12-04T08:28:55.0052171Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2025-12-04T08:28:55.0052618Z Configure a credential helper to remove this warning. See 2025-12-04T08:28:55.0053032Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2025-12-04T08:28:55.0053286Z 2025-12-04T08:28:55.0053484Z Login Succeeded 2025-12-04T08:28:55.0071152Z ++ date +%s 2025-12-04T08:28:55.0084086Z + START_TIME=1764836935 2025-12-04T08:28:55.0084358Z ++ date +%s 2025-12-04T08:28:55.0095868Z + [[ 1764829735 -lt 1764836935 ]] 2025-12-04T08:28:55.0096562Z + docker manifest inspect 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-f0cd68561080d537ef3d3d6f81b25a6416ad600a 2025-12-04T08:28:55.2304395Z { 2025-12-04T08:28:55.2309205Z "schemaVersion": 2, 2025-12-04T08:28:55.2309588Z "mediaType": "application/vnd.docker.distribution.manifest.v2+json", 2025-12-04T08:28:55.2309974Z "config": { 2025-12-04T08:28:55.2310260Z "mediaType": "application/vnd.docker.container.image.v1+json", 2025-12-04T08:28:55.2310552Z "size": 33581, 2025-12-04T08:28:55.2310853Z "digest": "sha256:40bd34edc3497e05e2150804a15313235b6426bcc4b8c19c03c3daf5524be913" 2025-12-04T08:28:55.2311170Z }, 2025-12-04T08:28:55.2311324Z "layers": [ 2025-12-04T08:28:55.2311475Z { 2025-12-04T08:28:55.2311834Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:28:55.2312122Z "size": 30447951, 2025-12-04T08:28:55.2312438Z "digest": "sha256:63e5bc7682b85ae57a1221210f64d62e7a90b0a30f19af4ca734b8242ae49d63" 2025-12-04T08:28:55.2312768Z }, 2025-12-04T08:28:55.2312917Z { 2025-12-04T08:28:55.2313150Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:28:55.2313433Z "size": 1554, 2025-12-04T08:28:55.2313732Z "digest": "sha256:6dc15eca51381c13be16385052dd2378ab1dce5fb77f7e7bceab34ed72e6e0e5" 2025-12-04T08:28:55.2314383Z }, 2025-12-04T08:28:55.2314522Z { 2025-12-04T08:28:55.2314757Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:28:55.2315051Z "size": 313274856, 2025-12-04T08:28:55.2315346Z "digest": "sha256:459f4df18f07c473cb662e8033f9dab9f68c9ac71608767ad055999fae6b31e2" 2025-12-04T08:28:55.2315668Z }, 2025-12-04T08:28:55.2315816Z { 2025-12-04T08:28:55.2316033Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:28:55.2316319Z "size": 787, 2025-12-04T08:28:55.2316604Z "digest": "sha256:82108541691974f1cc469d0ec89a048191b8de4dc36bf9a0184a4bd50120b69d" 2025-12-04T08:28:55.2316909Z }, 2025-12-04T08:28:55.2317051Z { 2025-12-04T08:28:55.2317278Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:28:55.2317552Z "size": 106, 2025-12-04T08:28:55.2317825Z "digest": "sha256:3738646b3d92a3f1b6529ba588c00216c306c997ad77730f207c7107c2e5aff3" 2025-12-04T08:28:55.2318138Z }, 2025-12-04T08:28:55.2318493Z { 2025-12-04T08:28:55.2318724Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:28:55.2319059Z "size": 704, 2025-12-04T08:28:55.2319349Z "digest": "sha256:aa9a10a37b3945f69568695fa4a9321333654a69dd44c55eff4e26bf9bffcad1" 2025-12-04T08:28:55.2319670Z }, 2025-12-04T08:28:55.2319829Z { 2025-12-04T08:28:55.2320059Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:28:55.2320344Z "size": 1218, 2025-12-04T08:28:55.2320632Z "digest": "sha256:55bac08f3e18897f61d3e7f28ada6547b05b3c36f559a882656834d43de61478" 2025-12-04T08:28:55.2321282Z }, 2025-12-04T08:28:55.2321421Z { 2025-12-04T08:28:55.2321651Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:28:55.2321936Z "size": 485, 2025-12-04T08:28:55.2322222Z "digest": "sha256:acf6468f6aadc3613e3b8fe73c016ce8d6f47f01e0cfee3a1015d0335c24d691" 2025-12-04T08:28:55.2322531Z }, 2025-12-04T08:28:55.2322673Z { 2025-12-04T08:28:55.2322902Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:28:55.2323188Z "size": 110362453, 2025-12-04T08:28:55.2323485Z "digest": "sha256:c0c31a0e69b798875a8d2c05d01a6861b2fe623e1970bf7faf822e7df6691d2c" 2025-12-04T08:28:55.2323800Z }, 2025-12-04T08:28:55.2323930Z { 2025-12-04T08:28:55.2324150Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:28:55.2324429Z "size": 4961, 2025-12-04T08:28:55.2324702Z "digest": "sha256:0f7d0dc70d2d944cc1c8a592f051d3f28d2b062c79166713fb7cc9225b4c97c0" 2025-12-04T08:28:55.2325022Z }, 2025-12-04T08:28:55.2325297Z { 2025-12-04T08:28:55.2325514Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:28:55.2325771Z "size": 1756, 2025-12-04T08:28:55.2326047Z "digest": "sha256:58c4d15d3bc18a789eb0d1a13f68df0b1d6cea66697fe63adf11b53fec74ba18" 2025-12-04T08:28:55.2326348Z }, 2025-12-04T08:28:55.2326477Z { 2025-12-04T08:28:55.2326691Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:28:55.2326957Z "size": 724, 2025-12-04T08:28:55.2327223Z "digest": "sha256:2f1cc47b61e1ebefa16f8ee12c5a1df77b109a84a8e8f425e7d65ec51cafdb93" 2025-12-04T08:28:55.2327527Z }, 2025-12-04T08:28:55.2327665Z { 2025-12-04T08:28:55.2327872Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:28:55.2328138Z "size": 543, 2025-12-04T08:28:55.2328399Z "digest": "sha256:d2472a74103d877e6532d6409a2ac9d54db6672b5d3002be011fdbbd8d360c49" 2025-12-04T08:28:55.2328680Z }, 2025-12-04T08:28:55.2328811Z { 2025-12-04T08:28:55.2329023Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:28:55.2329285Z "size": 3353192867, 2025-12-04T08:28:55.2329560Z "digest": "sha256:d1fba72936888ea8f8e4c7ad015bfcf3285017b95f3d72795d5b6a0b0ded36fc" 2025-12-04T08:28:55.2329855Z }, 2025-12-04T08:28:55.2329989Z { 2025-12-04T08:28:55.2330204Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:28:55.2330560Z "size": 32, 2025-12-04T08:28:55.2330840Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2025-12-04T08:28:55.2331130Z }, 2025-12-04T08:28:55.2331265Z { 2025-12-04T08:28:55.2331476Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:28:55.2331727Z "size": 397, 2025-12-04T08:28:55.2331995Z "digest": "sha256:6a9b03ce41a77be16366fba2c6ca0aabefe42ee4879ab5d088b312970e830770" 2025-12-04T08:28:55.2332294Z }, 2025-12-04T08:28:55.2332420Z { 2025-12-04T08:28:55.2332637Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:28:55.2332902Z "size": 237420, 2025-12-04T08:28:55.2333170Z "digest": "sha256:3519ac15be7978a14201f5d431b6472ed30382cde8628c6401c9db98ccd1a871" 2025-12-04T08:28:55.2333455Z }, 2025-12-04T08:28:55.2333589Z { 2025-12-04T08:28:55.2333802Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:28:55.2334055Z "size": 231, 2025-12-04T08:28:55.2334322Z "digest": "sha256:dabb51b819a06823e5a1c8d4f9e29b689f38b19ee1e90da40075bd1593b8d0fb" 2025-12-04T08:28:55.2334617Z }, 2025-12-04T08:28:55.2334745Z { 2025-12-04T08:28:55.2334955Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:28:55.2335252Z "size": 2967944, 2025-12-04T08:28:55.2335523Z "digest": "sha256:8f36833a24d02d47bf18bdc573adbb45afb8f5f06886da9bd671a1a33e3007bd" 2025-12-04T08:28:55.2335817Z }, 2025-12-04T08:28:55.2335949Z { 2025-12-04T08:28:55.2336152Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:28:55.2336415Z "size": 1472, 2025-12-04T08:28:55.2336687Z "digest": "sha256:ab53c5a853a52de13f11fbb026aaf469754adbe52b02e96f4fbdc05916585244" 2025-12-04T08:28:55.2336990Z }, 2025-12-04T08:28:55.2337130Z { 2025-12-04T08:28:55.2337353Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:28:55.2337624Z "size": 482, 2025-12-04T08:28:55.2337899Z "digest": "sha256:025a0e5e6ac19cb2d27c9423a5d019a34ea7d4004d426d6b890693ccfe9f54d3" 2025-12-04T08:28:55.2338213Z }, 2025-12-04T08:28:55.2338354Z { 2025-12-04T08:28:55.2338574Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:28:55.2338848Z "size": 200, 2025-12-04T08:28:55.2339115Z "digest": "sha256:ce3394c8f2109f4538f6e7f2c39f685d0ba028aacac33ddacbb8ef9fd349702e" 2025-12-04T08:28:55.2339403Z }, 2025-12-04T08:28:55.2339537Z { 2025-12-04T08:28:55.2339750Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:28:55.2339999Z "size": 608, 2025-12-04T08:28:55.2340319Z "digest": "sha256:a5c3888c3a0c5812efbfe307d18de826363900fa8d8097c1ee84a72630aa067b" 2025-12-04T08:28:55.2340612Z }, 2025-12-04T08:28:55.2340738Z { 2025-12-04T08:28:55.2366583Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:28:55.2366913Z "size": 225, 2025-12-04T08:28:55.2367228Z "digest": "sha256:ed902a3a4e3b916aa869b612baf9cd36745b671b6781d19e3b4c3fadc2f513f5" 2025-12-04T08:28:55.2367586Z }, 2025-12-04T08:28:55.2367742Z { 2025-12-04T08:28:55.2367982Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:28:55.2368272Z "size": 829, 2025-12-04T08:28:55.2368573Z "digest": "sha256:b4e1efca22beb475459a14f4a571a2512c2b1ed4b24e66bd3f82f1681dc5b1bc" 2025-12-04T08:28:55.2369025Z }, 2025-12-04T08:28:55.2369242Z { 2025-12-04T08:28:55.2369528Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:28:55.2369849Z "size": 32, 2025-12-04T08:28:55.2370156Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2025-12-04T08:28:55.2370484Z }, 2025-12-04T08:28:55.2370636Z { 2025-12-04T08:28:55.2370869Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:28:55.2371153Z "size": 104, 2025-12-04T08:28:55.2371453Z "digest": "sha256:79fcfd297d9fcfe2810595fc2c6ba503293b59046bc192bb0620a64d9bdff778" 2025-12-04T08:28:55.2371790Z }, 2025-12-04T08:28:55.2372106Z { 2025-12-04T08:28:55.2372387Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:28:55.2372671Z "size": 1495, 2025-12-04T08:28:55.2372967Z "digest": "sha256:7bd89c134b49f83cfb0725df3615556b60af380b932186c5abd001232b4597cf" 2025-12-04T08:28:55.2373302Z }, 2025-12-04T08:28:55.2373448Z { 2025-12-04T08:28:55.2373682Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:28:55.2373974Z "size": 458790783, 2025-12-04T08:28:55.2374319Z "digest": "sha256:b7d40b4fd1b9375cfd45d5c593ec0cccc21c80d667ea79f4effb09f04b0705b0" 2025-12-04T08:28:55.2374675Z }, 2025-12-04T08:28:55.2374826Z { 2025-12-04T08:28:55.2375067Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:28:55.2375352Z "size": 163, 2025-12-04T08:28:55.2375657Z "digest": "sha256:dd1cdd87320d29d4ca09686ab00b76a396efcc6f3ea6d0bfd1f7922e46336ca6" 2025-12-04T08:28:55.2375996Z }, 2025-12-04T08:28:55.2376158Z { 2025-12-04T08:28:55.2376406Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:28:55.2376691Z "size": 347, 2025-12-04T08:28:55.2376983Z "digest": "sha256:c21a6e1cd03a718ebfb927ab114ebbbee0b57a93908d0265d24b02cf34c0fee3" 2025-12-04T08:28:55.2377312Z }, 2025-12-04T08:28:55.2377461Z { 2025-12-04T08:28:55.2377687Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:28:55.2377971Z "size": 32, 2025-12-04T08:28:55.2378261Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2025-12-04T08:28:55.2378591Z }, 2025-12-04T08:28:55.2378735Z { 2025-12-04T08:28:55.2378973Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:28:55.2379258Z "size": 106, 2025-12-04T08:28:55.2379549Z "digest": "sha256:0aa3ea0b275426eac8eff6b26ba17c168ae70d1baa5a7845e61d41bd01a9ff43" 2025-12-04T08:28:55.2379875Z }, 2025-12-04T08:28:55.2380024Z { 2025-12-04T08:28:55.2380252Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:28:55.2401224Z "size": 426, 2025-12-04T08:28:55.2401595Z "digest": "sha256:74e0bdbb05d31425d7a2f40ff0cb2423abaf90f1ed56c4b69f43e8d92c569379" 2025-12-04T08:28:55.2401911Z }, 2025-12-04T08:28:55.2402045Z { 2025-12-04T08:28:55.2402298Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:28:55.2402581Z "size": 19309400, 2025-12-04T08:28:55.2402862Z "digest": "sha256:ba3aee7dcf03efbd1e7b1611c8f65f385c94b2572d15c099a1535f27b72509f7" 2025-12-04T08:28:55.2403160Z }, 2025-12-04T08:28:55.2403379Z { 2025-12-04T08:28:55.2403597Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:28:55.2403877Z "size": 108, 2025-12-04T08:28:55.2404143Z "digest": "sha256:8c8d59b8759d40399b07b019a46fc05b9ccabe2b9276f49eea203fd6908ff334" 2025-12-04T08:28:55.2404431Z }, 2025-12-04T08:28:55.2404580Z { 2025-12-04T08:28:55.2404790Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:28:55.2405051Z "size": 827, 2025-12-04T08:28:55.2405306Z "digest": "sha256:3c960865867b66327e316bf682adabd1867dd9a41cc0aee1507ee8e35fa614a9" 2025-12-04T08:28:55.2405591Z }, 2025-12-04T08:28:55.2405715Z { 2025-12-04T08:28:55.2405922Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:28:55.2406175Z "size": 724, 2025-12-04T08:28:55.2406441Z "digest": "sha256:2f1cc47b61e1ebefa16f8ee12c5a1df77b109a84a8e8f425e7d65ec51cafdb93" 2025-12-04T08:28:55.2406730Z }, 2025-12-04T08:28:55.2406855Z { 2025-12-04T08:28:55.2407064Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:28:55.2407311Z "size": 149, 2025-12-04T08:28:55.2407592Z "digest": "sha256:074db79e3832af094ddad6e5c5649347b62e8ba0365229feced3ed0263a0c611" 2025-12-04T08:28:55.2407879Z }, 2025-12-04T08:28:55.2408011Z { 2025-12-04T08:28:55.2408231Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:28:55.2408501Z "size": 137, 2025-12-04T08:28:55.2408844Z "digest": "sha256:226748f4ff23ba2aef33619c3a639fb44865d5f4515c4b45d4196f4e0e77b187" 2025-12-04T08:28:55.2409160Z }, 2025-12-04T08:28:55.2409293Z { 2025-12-04T08:28:55.2409513Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:28:55.2409775Z "size": 140, 2025-12-04T08:28:55.2410049Z "digest": "sha256:75d817336f1df7579f5b4be29625c53d594cb78c776f5454cc69438bb1bb0dc9" 2025-12-04T08:28:55.2410353Z }, 2025-12-04T08:28:55.2410488Z { 2025-12-04T08:28:55.2410714Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:28:55.2410999Z "size": 18890247472, 2025-12-04T08:28:55.2411285Z "digest": "sha256:850870a17b94a4d7c5e55b287cc0b1ddd0982845e11f055765dc358277cdfc54" 2025-12-04T08:28:55.2411637Z }, 2025-12-04T08:28:55.2411769Z { 2025-12-04T08:28:55.2411989Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:28:55.2412261Z "size": 222, 2025-12-04T08:28:55.2412532Z "digest": "sha256:e6225129924f0a66acaf35efd792901351da58116629962669d9c3acaadbab6e" 2025-12-04T08:28:55.2412841Z }, 2025-12-04T08:28:55.2412975Z { 2025-12-04T08:28:55.2413204Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:28:55.2413477Z "size": 255, 2025-12-04T08:28:55.2413761Z "digest": "sha256:5cd6b9ec5c3c4a1cf2e2e7a1794aaae173eb5a291b9899bb733e5f36d866fab0" 2025-12-04T08:28:55.2414080Z }, 2025-12-04T08:28:55.2414222Z { 2025-12-04T08:28:55.2414451Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:28:55.2414732Z "size": 32, 2025-12-04T08:28:55.2415021Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2025-12-04T08:28:55.2415337Z }, 2025-12-04T08:28:55.2415471Z { 2025-12-04T08:28:55.2415701Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:28:55.2415981Z "size": 106, 2025-12-04T08:28:55.2416265Z "digest": "sha256:613d2b1f3db875d1d695081b44aefbf94eeb7815d794b28dd17ec623d3f9dad5" 2025-12-04T08:28:55.2416577Z }, 2025-12-04T08:28:55.2416721Z { 2025-12-04T08:28:55.2416945Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:28:55.2417231Z "size": 312293443, 2025-12-04T08:28:55.2417525Z "digest": "sha256:388169fffe8a06396f4e8d6fc51d07a83e08f04836995ae28c2e13960cb89fab" 2025-12-04T08:28:55.2417839Z }, 2025-12-04T08:28:55.2417971Z { 2025-12-04T08:28:55.2418195Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:28:55.2418488Z "size": 6628443345, 2025-12-04T08:28:55.2418837Z "digest": "sha256:6ca498e78b7b821a815bc7c14f057d52f7d9ce8dda8bca0945e6f522c7d208e1" 2025-12-04T08:28:55.2419158Z }, 2025-12-04T08:28:55.2419299Z { 2025-12-04T08:28:55.2419513Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:28:55.2419785Z "size": 129, 2025-12-04T08:28:55.2420066Z "digest": "sha256:952b753ec7be39e1967f95afb87a880712b6425610adcda2afe3b2211cc8de12" 2025-12-04T08:28:55.2420369Z }, 2025-12-04T08:28:55.2420513Z { 2025-12-04T08:28:55.2420897Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:28:55.2421190Z "size": 880, 2025-12-04T08:28:55.2421480Z "digest": "sha256:5b6f6de4bdd1c96aa8704c326fb27fa7465ebf388f0748ee56d9f521bb6b6697" 2025-12-04T08:28:55.2421804Z }, 2025-12-04T08:28:55.2421948Z { 2025-12-04T08:28:55.2422167Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:28:55.2422449Z "size": 724, 2025-12-04T08:28:55.2422748Z "digest": "sha256:2f1cc47b61e1ebefa16f8ee12c5a1df77b109a84a8e8f425e7d65ec51cafdb93" 2025-12-04T08:28:55.2423066Z }, 2025-12-04T08:28:55.2423211Z { 2025-12-04T08:28:55.2423436Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:28:55.2423708Z "size": 139, 2025-12-04T08:28:55.2423996Z "digest": "sha256:fa23d9952f3cea91af7b5ffce93e64d90d98aefb838aa8a1fe4a6b40cd0eab91" 2025-12-04T08:28:55.2424321Z }, 2025-12-04T08:28:55.2424581Z { 2025-12-04T08:28:55.2424796Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:28:55.2425061Z "size": 32, 2025-12-04T08:28:55.2425331Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2025-12-04T08:28:55.2425621Z }, 2025-12-04T08:28:55.2425756Z { 2025-12-04T08:28:55.2425969Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:28:55.2426224Z "size": 161, 2025-12-04T08:28:55.2426503Z "digest": "sha256:c2aa9ea4d09ac3edb41e48cdc892fafe72a5cddc98ebdf67d978c6f8d63cd7d2" 2025-12-04T08:28:55.2426806Z }, 2025-12-04T08:28:55.2426933Z { 2025-12-04T08:28:55.2427142Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:28:55.2427405Z "size": 1011, 2025-12-04T08:28:55.2427672Z "digest": "sha256:a9bbc9e426d367e15c7f0c1faae6417526e219fa8188ae17b9e6c2c3b8083bdc" 2025-12-04T08:28:55.2427969Z }, 2025-12-04T08:28:55.2428102Z { 2025-12-04T08:28:55.2428308Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:28:55.2428565Z "size": 724, 2025-12-04T08:28:55.2428832Z "digest": "sha256:2f1cc47b61e1ebefa16f8ee12c5a1df77b109a84a8e8f425e7d65ec51cafdb93" 2025-12-04T08:28:55.2429128Z }, 2025-12-04T08:28:55.2429253Z { 2025-12-04T08:28:55.2429462Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:28:55.2429716Z "size": 135, 2025-12-04T08:28:55.2429971Z "digest": "sha256:82f2371f6dc2c2797baa60e672fd78bb04909cd0248c52cae058a7fd8f215a4f" 2025-12-04T08:28:55.2430263Z }, 2025-12-04T08:28:55.2430398Z { 2025-12-04T08:28:55.2430603Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:28:55.2430860Z "size": 32, 2025-12-04T08:28:55.2431124Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2025-12-04T08:28:55.2431408Z }, 2025-12-04T08:28:55.2431541Z { 2025-12-04T08:28:55.2431754Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:28:55.2432008Z "size": 158, 2025-12-04T08:28:55.2432269Z "digest": "sha256:c87301921afac5ba5745add04b44f57bf31c551bf48a2574d4424dab64c25ce2" 2025-12-04T08:28:55.2432562Z }, 2025-12-04T08:28:55.2432699Z { 2025-12-04T08:28:55.2432901Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:28:55.2433155Z "size": 603, 2025-12-04T08:28:55.2433424Z "digest": "sha256:384d566a822f4defa0ecf9c7d37b6d7a5fc5eaab415fe1d3cc02b190ddc19e71" 2025-12-04T08:28:55.2433715Z }, 2025-12-04T08:28:55.2433851Z { 2025-12-04T08:28:55.2434132Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:28:55.2434385Z "size": 724, 2025-12-04T08:28:55.2434653Z "digest": "sha256:2f1cc47b61e1ebefa16f8ee12c5a1df77b109a84a8e8f425e7d65ec51cafdb93" 2025-12-04T08:28:55.2434978Z }, 2025-12-04T08:28:55.2435111Z { 2025-12-04T08:28:55.2435335Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:28:55.2435608Z "size": 155, 2025-12-04T08:28:55.2435877Z "digest": "sha256:32ddcdd5dd6920acb12ca95671a164346308ec7743afbe8a333142733c42bafd" 2025-12-04T08:28:55.2436195Z }, 2025-12-04T08:28:55.2436339Z { 2025-12-04T08:28:55.2436562Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:28:55.2436827Z "size": 32, 2025-12-04T08:28:55.2437106Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2025-12-04T08:28:55.2437422Z }, 2025-12-04T08:28:55.2437555Z { 2025-12-04T08:28:55.2437787Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:28:55.2438070Z "size": 188, 2025-12-04T08:28:55.2438508Z "digest": "sha256:a116a32ceaa15bbd55b0f4281fb0040de221c62e29659b23dd71ecd2382a969a" 2025-12-04T08:28:55.2438841Z }, 2025-12-04T08:28:55.2438989Z { 2025-12-04T08:28:55.2439213Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:28:55.2439498Z "size": 1371, 2025-12-04T08:28:55.2439842Z "digest": "sha256:f6fbbbe6067e383cadb530c3f53b113e9826843fb2348ea1f9a000a4790fa822" 2025-12-04T08:28:55.2440137Z }, 2025-12-04T08:28:55.2440265Z { 2025-12-04T08:28:55.2440474Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:28:55.2440732Z "size": 32, 2025-12-04T08:28:55.2440988Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2025-12-04T08:28:55.2441282Z }, 2025-12-04T08:28:55.2441415Z { 2025-12-04T08:28:55.2441621Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:28:55.2441883Z "size": 137, 2025-12-04T08:28:55.2442141Z "digest": "sha256:648018658875ac3356b277abe3d98df9a8b7a0345f131598c07f5e9f9318b740" 2025-12-04T08:28:55.2442419Z }, 2025-12-04T08:28:55.2442553Z { 2025-12-04T08:28:55.2442763Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:28:55.2443012Z "size": 528, 2025-12-04T08:28:55.2443282Z "digest": "sha256:5e2f7eac20ad5128eebe0ba7dbd08111d28e7f65c26c7fea7cd6dc7a2c0725b9" 2025-12-04T08:28:55.2443586Z }, 2025-12-04T08:28:55.2443720Z { 2025-12-04T08:28:55.2443925Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:28:55.2444185Z "size": 32, 2025-12-04T08:28:55.2444449Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2025-12-04T08:28:55.2444734Z }, 2025-12-04T08:28:55.2444868Z { 2025-12-04T08:28:55.2445078Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:28:55.2445328Z "size": 104, 2025-12-04T08:28:55.2445592Z "digest": "sha256:0d1243af7593a042ff443492a4cac70f817dc8c56ac45a0c138d102f40c1cc07" 2025-12-04T08:28:55.2445885Z }, 2025-12-04T08:28:55.2446011Z { 2025-12-04T08:28:55.2446222Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:28:55.2446477Z "size": 435, 2025-12-04T08:28:55.2446735Z "digest": "sha256:1f6d5b941ea1f6f07c190f5ca1d4a5172e4d696ad597ae58dcd37b297f60b556" 2025-12-04T08:28:55.2447032Z }, 2025-12-04T08:28:55.2447171Z { 2025-12-04T08:28:55.2447408Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:28:55.2447673Z "size": 32, 2025-12-04T08:28:55.2447951Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2025-12-04T08:28:55.2448270Z }, 2025-12-04T08:28:55.2448395Z { 2025-12-04T08:28:55.2448606Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:28:55.2448863Z "size": 107, 2025-12-04T08:28:55.2449197Z "digest": "sha256:4444f70d73cefec9f161242e9def76078af5abc7a3ad3e5d6e7d4f1efed07939" 2025-12-04T08:28:55.2449505Z }, 2025-12-04T08:28:55.2449640Z { 2025-12-04T08:28:55.2449853Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:28:55.2450106Z "size": 1896, 2025-12-04T08:28:55.2450386Z "digest": "sha256:bf45550ddbad2524852930466f963e432429f2b86f7f6ccf0415fc56980004ce" 2025-12-04T08:28:55.2450692Z }, 2025-12-04T08:28:55.2450830Z { 2025-12-04T08:28:55.2451055Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:28:55.2451335Z "size": 245588924, 2025-12-04T08:28:55.2451618Z "digest": "sha256:30fff4f4bad325a3a2d91463ca6702c411392e5b0646a5cd4f47fcae56c55639" 2025-12-04T08:28:55.2451923Z }, 2025-12-04T08:28:55.2452063Z { 2025-12-04T08:28:55.2452276Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:28:55.2452545Z "size": 106, 2025-12-04T08:28:55.2452827Z "digest": "sha256:cca4db1ba155a3afd1fd645cd3223e9a4de9c785f8559387f21da187672d9e9e" 2025-12-04T08:28:55.2453129Z }, 2025-12-04T08:28:55.2453270Z { 2025-12-04T08:28:55.2453495Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:28:55.2453767Z "size": 165, 2025-12-04T08:28:55.2454039Z "digest": "sha256:f0ffa379f4eba2f3ddc323f45f226f39aed5a7e93880884a97bd4d6a91864dc1" 2025-12-04T08:28:55.2454350Z }, 2025-12-04T08:28:55.2454490Z { 2025-12-04T08:28:55.2454758Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:28:55.2455031Z "size": 7942, 2025-12-04T08:28:55.2455307Z "digest": "sha256:df645e678f95db120f96e4c4c66400d315ed9214d9085216b417348cf682d43a" 2025-12-04T08:28:55.2455603Z }, 2025-12-04T08:28:55.2455750Z { 2025-12-04T08:28:55.2455971Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:28:55.2456237Z "size": 8076, 2025-12-04T08:28:55.2456519Z "digest": "sha256:52ad6c17d10308710fb2dc855f911246b7a369ccf5c3142cb3ad7fb182f708b8" 2025-12-04T08:28:55.2456841Z }, 2025-12-04T08:28:55.2456975Z { 2025-12-04T08:28:55.2457202Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:28:55.2457483Z "size": 302, 2025-12-04T08:28:55.2457767Z "digest": "sha256:34408374e32c70dbcfd7e0516edaa8698c25e6e3dbcd920978f2b3a801bbe659" 2025-12-04T08:28:55.2458069Z }, 2025-12-04T08:28:55.2458211Z { 2025-12-04T08:28:55.2458433Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:28:55.2458700Z "size": 32, 2025-12-04T08:28:55.2458978Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2025-12-04T08:28:55.2459298Z }, 2025-12-04T08:28:55.2459430Z { 2025-12-04T08:28:55.2459649Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:28:55.2459923Z "size": 108, 2025-12-04T08:28:55.2460198Z "digest": "sha256:27803b661d9e2cb1ce49c0a100ce824fbcbdd1f3a3ceda4ed5affbcb07ac3be4" 2025-12-04T08:28:55.2460513Z }, 2025-12-04T08:28:55.2460651Z { 2025-12-04T08:28:55.2460883Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:28:55.2461149Z "size": 54145699, 2025-12-04T08:28:55.2461440Z "digest": "sha256:07dafc893dea952d7677124b42e90b29b167fabe81e8a9d8b3fb4b0aa11b66ec" 2025-12-04T08:28:55.2461749Z }, 2025-12-04T08:28:55.2461884Z { 2025-12-04T08:28:55.2462110Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:28:55.2462385Z "size": 32, 2025-12-04T08:28:55.2462660Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2025-12-04T08:28:55.2462975Z } 2025-12-04T08:28:55.2463119Z ] 2025-12-04T08:28:55.2463255Z } 2025-12-04T08:28:55.2463426Z + exit 0 2025-12-04T08:28:55.2498555Z ##[group]Run set -eux 2025-12-04T08:28:55.2498782Z set -eux 2025-12-04T08:28:55.2499078Z # It's ok if this steps fails, it would then be an anonymous user like what we used to have 2025-12-04T08:28:55.2500004Z aws secretsmanager get-secret-value --secret-id docker_hub_readonly_token | jq --raw-output '.SecretString' | jq -r .docker_hub_readonly_token | docker login --username pytorchbot --password-stdin || true 2025-12-04T08:28:55.2505638Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T08:28:55.2505909Z env: 2025-12-04T08:28:55.2506084Z GIT_DEFAULT_BRANCH: main 2025-12-04T08:28:55.2506274Z ##[endgroup] 2025-12-04T08:28:55.2535708Z + aws secretsmanager get-secret-value --secret-id docker_hub_readonly_token 2025-12-04T08:28:55.2541857Z + docker login --username pytorchbot --password-stdin 2025-12-04T08:28:55.2542829Z + jq --raw-output .SecretString 2025-12-04T08:28:55.2543109Z + jq -r .docker_hub_readonly_token 2025-12-04T08:28:55.7328878Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2025-12-04T08:28:55.7329841Z Configure a credential helper to remove this warning. See 2025-12-04T08:28:55.7330314Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2025-12-04T08:28:55.7330616Z 2025-12-04T08:28:55.7330693Z Login Succeeded 2025-12-04T08:28:55.7417508Z ##[group]Run tag=${ECR_DOCKER_IMAGE##*:} 2025-12-04T08:28:55.7417782Z tag=${ECR_DOCKER_IMAGE##*:} 2025-12-04T08:28:55.7418060Z echo "docker pull ghcr.io/pytorch/ci-image:${tag/:/-}" 2025-12-04T08:28:55.7423541Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T08:28:55.7423800Z env: 2025-12-04T08:28:55.7424095Z GIT_DEFAULT_BRANCH: main 2025-12-04T08:28:55.7424661Z ECR_DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-f0cd68561080d537ef3d3d6f81b25a6416ad600a 2025-12-04T08:28:55.7425221Z ##[endgroup] 2025-12-04T08:28:55.7449274Z docker pull ghcr.io/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-f0cd68561080d537ef3d3d6f81b25a6416ad600a 2025-12-04T08:28:55.7497968Z ##[group]Run pytorch/test-infra/.github/actions/pull-docker-image@main 2025-12-04T08:28:55.7498262Z with: 2025-12-04T08:28:55.7498816Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-f0cd68561080d537ef3d3d6f81b25a6416ad600a 2025-12-04T08:28:55.7499471Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-12-04T08:28:55.7499743Z env: 2025-12-04T08:28:55.7499906Z GIT_DEFAULT_BRANCH: main 2025-12-04T08:28:55.7500100Z ##[endgroup] 2025-12-04T08:28:55.7524213Z ##[group]Run set -x 2025-12-04T08:28:55.7524433Z set -x 2025-12-04T08:28:55.7524597Z set +e 2025-12-04T08:28:55.7524763Z  2025-12-04T08:28:55.7524924Z login() { 2025-12-04T08:28:55.7525248Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2025-12-04T08:28:55.7525585Z } 2025-12-04T08:28:55.7525743Z  2025-12-04T08:28:55.7525937Z retry () { 2025-12-04T08:28:55.7526126Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2025-12-04T08:28:55.7526345Z } 2025-12-04T08:28:55.7526500Z  2025-12-04T08:28:55.7526667Z retry login "${DOCKER_REGISTRY}" 2025-12-04T08:28:55.7526877Z  2025-12-04T08:28:55.7527182Z IMAGE_SIZE=$(docker manifest inspect "${DOCKER_IMAGE}" | jq '[.layers[].size, .config.size] | add / 1024 / 1024') 2025-12-04T08:28:55.7527616Z echo "Compressed size of image in MB: ${IMAGE_SIZE}" 2025-12-04T08:28:55.7527859Z  2025-12-04T08:28:55.7528009Z set -e 2025-12-04T08:28:55.7528236Z # ignore output since only exit code is used for conditional 2025-12-04T08:28:55.7528552Z # only pull docker image if it's not available locally 2025-12-04T08:28:55.7528883Z if ! docker inspect --type=image "${DOCKER_IMAGE}" >/dev/null 2>/dev/null; then 2025-12-04T08:28:55.7529203Z  retry docker pull "${DOCKER_IMAGE}" 2025-12-04T08:28:55.7529416Z fi 2025-12-04T08:28:55.7533446Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T08:28:55.7533692Z env: 2025-12-04T08:28:55.7533855Z GIT_DEFAULT_BRANCH: main 2025-12-04T08:28:55.7534398Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-f0cd68561080d537ef3d3d6f81b25a6416ad600a 2025-12-04T08:28:55.7535003Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-12-04T08:28:55.7535259Z ##[endgroup] 2025-12-04T08:28:55.7556232Z + set +e 2025-12-04T08:28:55.7556866Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-12-04T08:28:55.7557214Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-12-04T08:28:55.7559313Z + aws ecr get-login-password --region us-east-1 2025-12-04T08:28:55.7559779Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-12-04T08:28:56.1917467Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2025-12-04T08:28:56.1917936Z Configure a credential helper to remove this warning. See 2025-12-04T08:28:56.1918648Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2025-12-04T08:28:56.1918920Z 2025-12-04T08:28:56.1919003Z Login Succeeded 2025-12-04T08:28:56.1940677Z ++ docker manifest inspect 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-f0cd68561080d537ef3d3d6f81b25a6416ad600a 2025-12-04T08:28:56.1941351Z ++ jq '[.layers[].size, .config.size] | add / 1024 / 1024' 2025-12-04T08:28:56.4065698Z + IMAGE_SIZE=29010.187264442444 2025-12-04T08:28:56.4066056Z + echo 'Compressed size of image in MB: 29010.187264442444' 2025-12-04T08:28:56.4066331Z + set -e 2025-12-04T08:28:56.4067465Z + docker inspect --type=image 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-f0cd68561080d537ef3d3d6f81b25a6416ad600a 2025-12-04T08:28:56.4068159Z Compressed size of image in MB: 29010.187264442444 2025-12-04T08:28:56.4308670Z + retry docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-f0cd68561080d537ef3d3d6f81b25a6416ad600a 2025-12-04T08:28:56.4309719Z + docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-f0cd68561080d537ef3d3d6f81b25a6416ad600a 2025-12-04T08:28:56.6976249Z pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-f0cd68561080d537ef3d3d6f81b25a6416ad600a: Pulling from pytorch/ci-image 2025-12-04T08:28:56.6984145Z 63e5bc7682b8: Pulling fs layer 2025-12-04T08:28:56.6984455Z 6dc15eca5138: Pulling fs layer 2025-12-04T08:28:56.6984668Z 459f4df18f07: Pulling fs layer 2025-12-04T08:28:56.6984872Z 821085416919: Pulling fs layer 2025-12-04T08:28:56.6985073Z 3738646b3d92: Pulling fs layer 2025-12-04T08:28:56.6985271Z aa9a10a37b39: Pulling fs layer 2025-12-04T08:28:56.6985517Z 55bac08f3e18: Pulling fs layer 2025-12-04T08:28:56.6985724Z acf6468f6aad: Pulling fs layer 2025-12-04T08:28:56.6985919Z c0c31a0e69b7: Pulling fs layer 2025-12-04T08:28:56.6986242Z 0f7d0dc70d2d: Pulling fs layer 2025-12-04T08:28:56.6986456Z 58c4d15d3bc1: Pulling fs layer 2025-12-04T08:28:56.6986648Z 2f1cc47b61e1: Pulling fs layer 2025-12-04T08:28:56.6986848Z d2472a74103d: Pulling fs layer 2025-12-04T08:28:56.6987045Z d1fba7293688: Pulling fs layer 2025-12-04T08:28:56.6987246Z 4f4fb700ef54: Pulling fs layer 2025-12-04T08:28:56.6987443Z 6a9b03ce41a7: Pulling fs layer 2025-12-04T08:28:56.6987664Z 3519ac15be79: Pulling fs layer 2025-12-04T08:28:56.6987871Z dabb51b819a0: Pulling fs layer 2025-12-04T08:28:56.6988062Z 8f36833a24d0: Pulling fs layer 2025-12-04T08:28:56.6988265Z ab53c5a853a5: Pulling fs layer 2025-12-04T08:28:56.6988468Z 025a0e5e6ac1: Pulling fs layer 2025-12-04T08:28:56.6988719Z ce3394c8f210: Pulling fs layer 2025-12-04T08:28:56.6988910Z a5c3888c3a0c: Pulling fs layer 2025-12-04T08:28:56.6989113Z ed902a3a4e3b: Pulling fs layer 2025-12-04T08:28:56.6989318Z b4e1efca22be: Pulling fs layer 2025-12-04T08:28:56.6989511Z 79fcfd297d9f: Pulling fs layer 2025-12-04T08:28:56.6989707Z 7bd89c134b49: Pulling fs layer 2025-12-04T08:28:56.6989904Z b7d40b4fd1b9: Pulling fs layer 2025-12-04T08:28:56.6990091Z dd1cdd87320d: Pulling fs layer 2025-12-04T08:28:56.6990288Z c21a6e1cd03a: Pulling fs layer 2025-12-04T08:28:56.6990486Z 0aa3ea0b2754: Pulling fs layer 2025-12-04T08:28:56.6990671Z 58c4d15d3bc1: Waiting 2025-12-04T08:28:56.6990860Z 74e0bdbb05d3: Pulling fs layer 2025-12-04T08:28:56.6991060Z 2f1cc47b61e1: Waiting 2025-12-04T08:28:56.6991245Z ba3aee7dcf03: Pulling fs layer 2025-12-04T08:28:56.6991438Z 8c8d59b8759d: Pulling fs layer 2025-12-04T08:28:56.6991637Z 3c960865867b: Pulling fs layer 2025-12-04T08:28:56.6991825Z d2472a74103d: Waiting 2025-12-04T08:28:56.6991991Z d1fba7293688: Waiting 2025-12-04T08:28:56.6992171Z 074db79e3832: Pulling fs layer 2025-12-04T08:28:56.6992369Z 226748f4ff23: Pulling fs layer 2025-12-04T08:28:56.6992562Z 75d817336f1d: Pulling fs layer 2025-12-04T08:28:56.6992756Z 4f4fb700ef54: Waiting 2025-12-04T08:28:56.6992952Z aa9a10a37b39: Waiting 2025-12-04T08:28:56.6993118Z 6a9b03ce41a7: Waiting 2025-12-04T08:28:56.6993294Z 850870a17b94: Pulling fs layer 2025-12-04T08:28:56.6993481Z 55bac08f3e18: Waiting 2025-12-04T08:28:56.6993649Z e6225129924f: Pulling fs layer 2025-12-04T08:28:56.6993843Z 5cd6b9ec5c3c: Pulling fs layer 2025-12-04T08:28:56.6994033Z 8f36833a24d0: Waiting 2025-12-04T08:28:56.6994198Z acf6468f6aad: Waiting 2025-12-04T08:28:56.6994380Z 613d2b1f3db8: Pulling fs layer 2025-12-04T08:28:56.6994810Z 388169fffe8a: Pulling fs layer 2025-12-04T08:28:56.6994993Z ab53c5a853a5: Waiting 2025-12-04T08:28:56.6995175Z 6ca498e78b7b: Pulling fs layer 2025-12-04T08:28:56.6995366Z 025a0e5e6ac1: Waiting 2025-12-04T08:28:56.6995543Z 952b753ec7be: Pulling fs layer 2025-12-04T08:28:56.6995747Z 5b6f6de4bdd1: Pulling fs layer 2025-12-04T08:28:56.6995939Z ce3394c8f210: Waiting 2025-12-04T08:28:56.6996224Z fa23d9952f3c: Pulling fs layer 2025-12-04T08:28:56.6996408Z a5c3888c3a0c: Waiting 2025-12-04T08:28:56.6996589Z c2aa9ea4d09a: Pulling fs layer 2025-12-04T08:28:56.6996787Z a9bbc9e426d3: Pulling fs layer 2025-12-04T08:28:56.6996976Z 82f2371f6dc2: Pulling fs layer 2025-12-04T08:28:56.6997167Z dabb51b819a0: Waiting 2025-12-04T08:28:56.6997341Z c0c31a0e69b7: Waiting 2025-12-04T08:28:56.6997505Z 850870a17b94: Waiting 2025-12-04T08:28:56.6997686Z c87301921afa: Pulling fs layer 2025-12-04T08:28:56.6997877Z e6225129924f: Waiting 2025-12-04T08:28:56.6998050Z 384d566a822f: Pulling fs layer 2025-12-04T08:28:56.6998405Z 5cd6b9ec5c3c: Waiting 2025-12-04T08:28:56.6998588Z b4e1efca22be: Waiting 2025-12-04T08:28:56.6998835Z 0f7d0dc70d2d: Waiting 2025-12-04T08:28:56.6999019Z 613d2b1f3db8: Waiting 2025-12-04T08:28:56.6999203Z 79fcfd297d9f: Waiting 2025-12-04T08:28:56.6999376Z 388169fffe8a: Waiting 2025-12-04T08:28:56.6999559Z 7bd89c134b49: Waiting 2025-12-04T08:28:56.7000086Z 6ca498e78b7b: Waiting 2025-12-04T08:28:56.7000338Z 32ddcdd5dd69: Pulling fs layer 2025-12-04T08:28:56.7000538Z 74e0bdbb05d3: Waiting 2025-12-04T08:28:56.7000775Z a116a32ceaa1: Pulling fs layer 2025-12-04T08:28:56.7000981Z f6fbbbe6067e: Pulling fs layer 2025-12-04T08:28:56.7001221Z 952b753ec7be: Waiting 2025-12-04T08:28:56.7001400Z 8c8d59b8759d: Waiting 2025-12-04T08:28:56.7001580Z ba3aee7dcf03: Waiting 2025-12-04T08:28:56.7001751Z 3c960865867b: Waiting 2025-12-04T08:28:56.7001932Z 648018658875: Pulling fs layer 2025-12-04T08:28:56.7002135Z 5e2f7eac20ad: Pulling fs layer 2025-12-04T08:28:56.7002330Z fa23d9952f3c: Waiting 2025-12-04T08:28:56.7002517Z 5b6f6de4bdd1: Waiting 2025-12-04T08:28:56.7002700Z 0d1243af7593: Pulling fs layer 2025-12-04T08:28:56.7002896Z 1f6d5b941ea1: Pulling fs layer 2025-12-04T08:28:56.7003100Z 4444f70d73ce: Pulling fs layer 2025-12-04T08:28:56.7003296Z 821085416919: Waiting 2025-12-04T08:28:56.7003462Z 3738646b3d92: Waiting 2025-12-04T08:28:56.7003644Z bf45550ddbad: Pulling fs layer 2025-12-04T08:28:56.7003841Z 074db79e3832: Waiting 2025-12-04T08:28:56.7004011Z 82f2371f6dc2: Waiting 2025-12-04T08:28:56.7004195Z 30fff4f4bad3: Pulling fs layer 2025-12-04T08:28:56.7004392Z 226748f4ff23: Waiting 2025-12-04T08:28:56.7004569Z cca4db1ba155: Pulling fs layer 2025-12-04T08:28:56.7004774Z f0ffa379f4eb: Pulling fs layer 2025-12-04T08:28:56.7004973Z 75d817336f1d: Waiting 2025-12-04T08:28:56.7005149Z ed902a3a4e3b: Waiting 2025-12-04T08:28:56.7005320Z c87301921afa: Waiting 2025-12-04T08:28:56.7005494Z c21a6e1cd03a: Waiting 2025-12-04T08:28:56.7005677Z df645e678f95: Pulling fs layer 2025-12-04T08:28:56.7005872Z 52ad6c17d103: Pulling fs layer 2025-12-04T08:28:56.7006071Z b7d40b4fd1b9: Waiting 2025-12-04T08:28:56.7006246Z dd1cdd87320d: Waiting 2025-12-04T08:28:56.7006423Z 34408374e32c: Pulling fs layer 2025-12-04T08:28:56.7006622Z 27803b661d9e: Pulling fs layer 2025-12-04T08:28:56.7006826Z 07dafc893dea: Pulling fs layer 2025-12-04T08:28:56.7007014Z f0ffa379f4eb: Waiting 2025-12-04T08:28:56.7007192Z 384d566a822f: Waiting 2025-12-04T08:28:56.7007366Z df645e678f95: Waiting 2025-12-04T08:28:56.7007532Z 52ad6c17d103: Waiting 2025-12-04T08:28:56.7007703Z 1f6d5b941ea1: Waiting 2025-12-04T08:28:56.7007875Z c2aa9ea4d09a: Waiting 2025-12-04T08:28:56.7008038Z 34408374e32c: Waiting 2025-12-04T08:28:56.7008210Z 4444f70d73ce: Waiting 2025-12-04T08:28:56.7008381Z 27803b661d9e: Waiting 2025-12-04T08:28:56.7008548Z a9bbc9e426d3: Waiting 2025-12-04T08:28:56.7008724Z 07dafc893dea: Waiting 2025-12-04T08:28:56.7008897Z 0d1243af7593: Waiting 2025-12-04T08:28:56.7009071Z bf45550ddbad: Waiting 2025-12-04T08:28:56.7009244Z cca4db1ba155: Waiting 2025-12-04T08:28:56.7009505Z 30fff4f4bad3: Waiting 2025-12-04T08:28:56.7009683Z 5e2f7eac20ad: Waiting 2025-12-04T08:28:56.7009854Z f6fbbbe6067e: Waiting 2025-12-04T08:28:56.7010030Z a116a32ceaa1: Waiting 2025-12-04T08:28:56.7010336Z 0aa3ea0b2754: Waiting 2025-12-04T08:28:56.7010509Z 32ddcdd5dd69: Waiting 2025-12-04T08:28:56.7010799Z 648018658875: Waiting 2025-12-04T08:28:56.7849357Z 6dc15eca5138: Verifying Checksum 2025-12-04T08:28:56.7849987Z 6dc15eca5138: Download complete 2025-12-04T08:28:56.8682052Z 821085416919: Download complete 2025-12-04T08:28:56.9725490Z 3738646b3d92: Verifying Checksum 2025-12-04T08:28:56.9725788Z 3738646b3d92: Download complete 2025-12-04T08:28:57.0333355Z aa9a10a37b39: Verifying Checksum 2025-12-04T08:28:57.0333864Z aa9a10a37b39: Download complete 2025-12-04T08:28:57.0623449Z 63e5bc7682b8: Verifying Checksum 2025-12-04T08:28:57.0623834Z 63e5bc7682b8: Download complete 2025-12-04T08:28:57.1207931Z 55bac08f3e18: Verifying Checksum 2025-12-04T08:28:57.1208254Z 55bac08f3e18: Download complete 2025-12-04T08:28:57.1451447Z acf6468f6aad: Verifying Checksum 2025-12-04T08:28:57.1451745Z acf6468f6aad: Download complete 2025-12-04T08:28:57.2290050Z 0f7d0dc70d2d: Download complete 2025-12-04T08:28:57.3256189Z 58c4d15d3bc1: Download complete 2025-12-04T08:28:57.4070921Z 2f1cc47b61e1: Download complete 2025-12-04T08:28:57.4971923Z d2472a74103d: Verifying Checksum 2025-12-04T08:28:57.4974833Z d2472a74103d: Download complete 2025-12-04T08:28:58.1904478Z 63e5bc7682b8: Pull complete 2025-12-04T08:28:58.2058712Z 6dc15eca5138: Pull complete 2025-12-04T08:28:58.2881416Z c0c31a0e69b7: Verifying Checksum 2025-12-04T08:28:58.2881902Z c0c31a0e69b7: Download complete 2025-12-04T08:28:58.2977779Z 4f4fb700ef54: Verifying Checksum 2025-12-04T08:28:58.2978102Z 4f4fb700ef54: Download complete 2025-12-04T08:28:58.3714187Z 6a9b03ce41a7: Download complete 2025-12-04T08:28:58.4688878Z 3519ac15be79: Download complete 2025-12-04T08:28:58.5479479Z dabb51b819a0: Verifying Checksum 2025-12-04T08:28:58.5479786Z dabb51b819a0: Download complete 2025-12-04T08:28:58.6719014Z 8f36833a24d0: Verifying Checksum 2025-12-04T08:28:58.6719320Z 8f36833a24d0: Download complete 2025-12-04T08:28:58.7636947Z ab53c5a853a5: Download complete 2025-12-04T08:28:58.8355790Z 025a0e5e6ac1: Verifying Checksum 2025-12-04T08:28:58.8356182Z 025a0e5e6ac1: Download complete 2025-12-04T08:28:58.9188840Z ce3394c8f210: Verifying Checksum 2025-12-04T08:28:58.9189130Z ce3394c8f210: Download complete 2025-12-04T08:28:59.0242029Z a5c3888c3a0c: Verifying Checksum 2025-12-04T08:28:59.0242411Z a5c3888c3a0c: Download complete 2025-12-04T08:28:59.1177328Z ed902a3a4e3b: Verifying Checksum 2025-12-04T08:28:59.1177703Z ed902a3a4e3b: Download complete 2025-12-04T08:28:59.2175693Z b4e1efca22be: Download complete 2025-12-04T08:28:59.2973260Z 79fcfd297d9f: Download complete 2025-12-04T08:28:59.3887923Z 7bd89c134b49: Download complete 2025-12-04T08:28:59.9051605Z 459f4df18f07: Verifying Checksum 2025-12-04T08:28:59.9051920Z 459f4df18f07: Download complete 2025-12-04T08:28:59.9944741Z dd1cdd87320d: Download complete 2025-12-04T08:29:00.0812157Z c21a6e1cd03a: Verifying Checksum 2025-12-04T08:29:00.0812661Z c21a6e1cd03a: Download complete 2025-12-04T08:29:00.1890373Z 0aa3ea0b2754: Download complete 2025-12-04T08:29:00.2785559Z 74e0bdbb05d3: Verifying Checksum 2025-12-04T08:29:00.2785965Z 74e0bdbb05d3: Download complete 2025-12-04T08:29:00.5265950Z ba3aee7dcf03: Verifying Checksum 2025-12-04T08:29:00.5266527Z ba3aee7dcf03: Download complete 2025-12-04T08:29:00.6099085Z 8c8d59b8759d: Verifying Checksum 2025-12-04T08:29:00.6099398Z 8c8d59b8759d: Download complete 2025-12-04T08:29:00.7121215Z 3c960865867b: Verifying Checksum 2025-12-04T08:29:00.7121506Z 3c960865867b: Download complete 2025-12-04T08:29:00.7899813Z 074db79e3832: Download complete 2025-12-04T08:29:00.8808930Z 226748f4ff23: Verifying Checksum 2025-12-04T08:29:00.8809273Z 226748f4ff23: Download complete 2025-12-04T08:29:00.9502068Z 75d817336f1d: Verifying Checksum 2025-12-04T08:29:00.9502428Z 75d817336f1d: Download complete 2025-12-04T08:29:04.0458740Z b7d40b4fd1b9: Verifying Checksum 2025-12-04T08:29:04.0459059Z b7d40b4fd1b9: Download complete 2025-12-04T08:29:04.1174541Z e6225129924f: Verifying Checksum 2025-12-04T08:29:04.1174844Z e6225129924f: Download complete 2025-12-04T08:29:04.2094531Z 5cd6b9ec5c3c: Download complete 2025-12-04T08:29:04.2932363Z 613d2b1f3db8: Verifying Checksum 2025-12-04T08:29:04.2932814Z 613d2b1f3db8: Download complete 2025-12-04T08:29:07.4586294Z 388169fffe8a: Verifying Checksum 2025-12-04T08:29:07.4587446Z 388169fffe8a: Download complete 2025-12-04T08:29:09.9659920Z 459f4df18f07: Pull complete 2025-12-04T08:29:10.2379905Z 821085416919: Pull complete 2025-12-04T08:29:10.4903434Z 3738646b3d92: Pull complete 2025-12-04T08:29:10.7554597Z aa9a10a37b39: Pull complete 2025-12-04T08:29:11.0446054Z 55bac08f3e18: Pull complete 2025-12-04T08:29:11.3353112Z acf6468f6aad: Pull complete 2025-12-04T08:29:14.5661446Z c0c31a0e69b7: Pull complete 2025-12-04T08:29:14.8493984Z 0f7d0dc70d2d: Pull complete 2025-12-04T08:29:15.1347247Z 58c4d15d3bc1: Pull complete 2025-12-04T08:29:15.4119292Z 2f1cc47b61e1: Pull complete 2025-12-04T08:29:15.6541896Z d2472a74103d: Pull complete 2025-12-04T08:29:48.7934147Z d1fba7293688: Verifying Checksum 2025-12-04T08:29:48.7934432Z d1fba7293688: Download complete 2025-12-04T08:29:48.8900098Z 952b753ec7be: Verifying Checksum 2025-12-04T08:29:48.8902183Z 952b753ec7be: Download complete 2025-12-04T08:29:49.0199587Z 5b6f6de4bdd1: Verifying Checksum 2025-12-04T08:29:49.0199963Z 5b6f6de4bdd1: Download complete 2025-12-04T08:29:49.1289879Z fa23d9952f3c: Verifying Checksum 2025-12-04T08:29:49.1290181Z fa23d9952f3c: Download complete 2025-12-04T08:29:49.2196071Z c2aa9ea4d09a: Verifying Checksum 2025-12-04T08:29:49.2196385Z c2aa9ea4d09a: Download complete 2025-12-04T08:29:49.3133029Z a9bbc9e426d3: Verifying Checksum 2025-12-04T08:29:49.3133342Z a9bbc9e426d3: Download complete 2025-12-04T08:29:49.4043648Z 82f2371f6dc2: Verifying Checksum 2025-12-04T08:29:49.4044044Z 82f2371f6dc2: Download complete 2025-12-04T08:29:49.4906592Z c87301921afa: Verifying Checksum 2025-12-04T08:29:49.4906902Z c87301921afa: Download complete 2025-12-04T08:29:49.5888225Z 384d566a822f: Verifying Checksum 2025-12-04T08:29:49.5888512Z 384d566a822f: Download complete 2025-12-04T08:29:49.6593336Z 32ddcdd5dd69: Verifying Checksum 2025-12-04T08:29:49.6596024Z 32ddcdd5dd69: Download complete 2025-12-04T08:29:49.7512902Z a116a32ceaa1: Verifying Checksum 2025-12-04T08:29:49.7516494Z a116a32ceaa1: Download complete 2025-12-04T08:29:49.9260670Z 648018658875: Download complete 2025-12-04T08:29:50.0312493Z 5e2f7eac20ad: Verifying Checksum 2025-12-04T08:29:50.0312808Z 5e2f7eac20ad: Download complete 2025-12-04T08:29:50.1172776Z 0d1243af7593: Download complete 2025-12-04T08:29:50.2115193Z 1f6d5b941ea1: Verifying Checksum 2025-12-04T08:29:50.2115512Z 1f6d5b941ea1: Download complete 2025-12-04T08:29:50.3002847Z 4444f70d73ce: Download complete 2025-12-04T08:29:50.3889254Z bf45550ddbad: Verifying Checksum 2025-12-04T08:29:50.3889586Z bf45550ddbad: Download complete 2025-12-04T08:29:56.8244257Z 30fff4f4bad3: Verifying Checksum 2025-12-04T08:29:56.8245885Z 30fff4f4bad3: Download complete 2025-12-04T08:29:56.9201420Z cca4db1ba155: Verifying Checksum 2025-12-04T08:29:56.9201734Z cca4db1ba155: Download complete 2025-12-04T08:29:56.9969482Z f0ffa379f4eb: Verifying Checksum 2025-12-04T08:29:56.9973836Z f0ffa379f4eb: Download complete 2025-12-04T08:29:57.0687871Z df645e678f95: Verifying Checksum 2025-12-04T08:29:57.0691305Z df645e678f95: Download complete 2025-12-04T08:29:57.1450236Z 52ad6c17d103: Verifying Checksum 2025-12-04T08:29:57.1450717Z 52ad6c17d103: Download complete 2025-12-04T08:29:57.2384621Z 34408374e32c: Verifying Checksum 2025-12-04T08:29:57.2384926Z 34408374e32c: Download complete 2025-12-04T08:29:57.3308177Z 27803b661d9e: Verifying Checksum 2025-12-04T08:29:57.3309663Z 27803b661d9e: Download complete 2025-12-04T08:29:59.3744753Z 07dafc893dea: Verifying Checksum 2025-12-04T08:29:59.3745250Z 07dafc893dea: Download complete 2025-12-04T08:31:09.6855026Z 6ca498e78b7b: Verifying Checksum 2025-12-04T08:31:09.6857773Z 6ca498e78b7b: Download complete 2025-12-04T08:32:58.0944932Z 850870a17b94: Verifying Checksum 2025-12-04T08:32:58.0945364Z 850870a17b94: Download complete 2025-12-04T08:34:07.4055781Z d1fba7293688: Pull complete 2025-12-04T08:34:07.6845861Z 4f4fb700ef54: Pull complete 2025-12-04T08:34:07.7631417Z 6a9b03ce41a7: Pull complete 2025-12-04T08:34:07.8903111Z 3519ac15be79: Pull complete 2025-12-04T08:34:08.0897949Z dabb51b819a0: Pull complete 2025-12-04T08:34:08.2698335Z 8f36833a24d0: Pull complete 2025-12-04T08:34:08.3043684Z ab53c5a853a5: Pull complete 2025-12-04T08:34:08.3420159Z 025a0e5e6ac1: Pull complete 2025-12-04T08:34:08.3763117Z ce3394c8f210: Pull complete 2025-12-04T08:34:08.4049167Z a5c3888c3a0c: Pull complete 2025-12-04T08:34:08.4341852Z ed902a3a4e3b: Pull complete 2025-12-04T08:34:08.4689163Z b4e1efca22be: Pull complete 2025-12-04T08:34:08.5834320Z 79fcfd297d9f: Pull complete 2025-12-04T08:34:08.7789640Z 7bd89c134b49: Pull complete 2025-12-04T08:34:23.9353512Z b7d40b4fd1b9: Pull complete 2025-12-04T08:34:24.1271802Z dd1cdd87320d: Pull complete 2025-12-04T08:34:24.3125004Z c21a6e1cd03a: Pull complete 2025-12-04T08:34:24.7794082Z 0aa3ea0b2754: Pull complete 2025-12-04T08:34:24.9802488Z 74e0bdbb05d3: Pull complete 2025-12-04T08:34:25.4705967Z ba3aee7dcf03: Pull complete 2025-12-04T08:34:25.5036805Z 8c8d59b8759d: Pull complete 2025-12-04T08:34:25.5433615Z 3c960865867b: Pull complete 2025-12-04T08:34:25.7431154Z 074db79e3832: Pull complete 2025-12-04T08:34:25.8494892Z 226748f4ff23: Pull complete 2025-12-04T08:34:26.1558228Z 75d817336f1d: Pull complete 2025-12-04T08:40:27.7435322Z 850870a17b94: Pull complete 2025-12-04T08:40:28.0034494Z e6225129924f: Pull complete 2025-12-04T08:40:28.0795169Z 5cd6b9ec5c3c: Pull complete 2025-12-04T08:40:28.3519281Z 613d2b1f3db8: Pull complete 2025-12-04T08:40:34.3351833Z 388169fffe8a: Pull complete 2025-12-04T08:43:31.4491355Z 6ca498e78b7b: Pull complete 2025-12-04T08:43:31.4752758Z 952b753ec7be: Pull complete 2025-12-04T08:43:31.5019334Z 5b6f6de4bdd1: Pull complete 2025-12-04T08:43:31.5535575Z fa23d9952f3c: Pull complete 2025-12-04T08:43:31.6038701Z c2aa9ea4d09a: Pull complete 2025-12-04T08:43:31.6340650Z a9bbc9e426d3: Pull complete 2025-12-04T08:43:31.6877108Z 82f2371f6dc2: Pull complete 2025-12-04T08:43:31.7381700Z c87301921afa: Pull complete 2025-12-04T08:43:31.7633456Z 384d566a822f: Pull complete 2025-12-04T08:43:31.8159082Z 32ddcdd5dd69: Pull complete 2025-12-04T08:43:31.8695487Z a116a32ceaa1: Pull complete 2025-12-04T08:43:31.8952352Z f6fbbbe6067e: Pull complete 2025-12-04T08:43:31.9454391Z 648018658875: Pull complete 2025-12-04T08:43:31.9708176Z 5e2f7eac20ad: Pull complete 2025-12-04T08:43:32.0231669Z 0d1243af7593: Pull complete 2025-12-04T08:43:32.0477099Z 1f6d5b941ea1: Pull complete 2025-12-04T08:43:32.1003629Z 4444f70d73ce: Pull complete 2025-12-04T08:43:32.1275718Z bf45550ddbad: Pull complete 2025-12-04T08:43:41.4448289Z 30fff4f4bad3: Pull complete 2025-12-04T08:43:41.4759801Z cca4db1ba155: Pull complete 2025-12-04T08:43:41.5564403Z f0ffa379f4eb: Pull complete 2025-12-04T08:43:41.6439721Z df645e678f95: Pull complete 2025-12-04T08:43:41.7029631Z 52ad6c17d103: Pull complete 2025-12-04T08:43:41.7447940Z 34408374e32c: Pull complete 2025-12-04T08:43:41.8111670Z 27803b661d9e: Pull complete 2025-12-04T08:43:44.0654816Z 07dafc893dea: Pull complete 2025-12-04T08:43:44.2018839Z Digest: sha256:b178ee928adfcab963ebbc6ea05ea1a0f0c605bb095e9076f010bf92d150869d 2025-12-04T08:43:44.2175197Z Status: Downloaded newer image for 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-f0cd68561080d537ef3d3d6f81b25a6416ad600a 2025-12-04T08:43:44.2270294Z 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-f0cd68561080d537ef3d3d6f81b25a6416ad600a 2025-12-04T08:43:44.2323282Z ##[group]Run echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2025-12-04T08:43:44.2324163Z echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2025-12-04T08:43:44.2331732Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T08:43:44.2332019Z env: 2025-12-04T08:43:44.2332209Z GIT_DEFAULT_BRANCH: main 2025-12-04T08:43:44.2332406Z ##[endgroup] 2025-12-04T08:43:44.2600527Z ##[group]Run pytorch/test-infra/.github/actions/setup-nvidia@main 2025-12-04T08:43:44.2600834Z with: 2025-12-04T08:43:44.2601015Z driver-version: 580.82.07 2025-12-04T08:43:44.2601204Z env: 2025-12-04T08:43:44.2601376Z GIT_DEFAULT_BRANCH: main 2025-12-04T08:43:44.2601573Z ##[endgroup] 2025-12-04T08:43:44.2624606Z ##[group]Run echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2025-12-04T08:43:44.2625358Z echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2025-12-04T08:43:44.2630846Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T08:43:44.2631094Z env: 2025-12-04T08:43:44.2631256Z GIT_DEFAULT_BRANCH: main 2025-12-04T08:43:44.2631434Z ##[endgroup] 2025-12-04T08:43:44.2719114Z ##[group]Run set -euo pipefail 2025-12-04T08:43:44.2719389Z set -euo pipefail 2025-12-04T08:43:44.2719646Z  2025-12-04T08:43:44.2719809Z has_gpu=false 2025-12-04T08:43:44.2720023Z devices="" 2025-12-04T08:43:44.2720223Z  2025-12-04T08:43:44.2720433Z if command -v nvidia-smi >/dev/null 2>&1; then 2025-12-04T08:43:44.2720968Z  if nvidia-smi -L >/tmp/nvidia_devices 2>/dev/null; then 2025-12-04T08:43:44.2721257Z  has_gpu=true 2025-12-04T08:43:44.2721525Z  devices=$(cat /tmp/nvidia_devices) 2025-12-04T08:43:44.2721763Z  fi 2025-12-04T08:43:44.2721933Z fi 2025-12-04T08:43:44.2722100Z  2025-12-04T08:43:44.2722271Z if [ "$has_gpu" = false ]; then 2025-12-04T08:43:44.2722573Z  if ls /dev/nvidia* >/tmp/nvidia_devices 2>/dev/null; then 2025-12-04T08:43:44.2722846Z  has_gpu=true 2025-12-04T08:43:44.2723069Z  devices=$(cat /tmp/nvidia_devices) 2025-12-04T08:43:44.2723291Z  fi 2025-12-04T08:43:44.2723456Z fi 2025-12-04T08:43:44.2723620Z  2025-12-04T08:43:44.2723849Z if [ "$has_gpu" = false ] && command -v lspci >/dev/null 2>&1; then 2025-12-04T08:43:44.2724209Z  if lspci | grep -i 'nvidia' >/tmp/nvidia_devices 2>/dev/null; then 2025-12-04T08:43:44.2724509Z  has_gpu=true 2025-12-04T08:43:44.2724721Z  devices=$(cat /tmp/nvidia_devices) 2025-12-04T08:43:44.2724950Z  fi 2025-12-04T08:43:44.2725117Z fi 2025-12-04T08:43:44.2725270Z  2025-12-04T08:43:44.2725502Z printf 'HAS_NVIDIA=%s\n' "$has_gpu" >> "$GITHUB_OUTPUT" 2025-12-04T08:43:44.2725947Z printf 'DETECTED_DEVICES<> "$GITHUB_OUTPUT" 2025-12-04T08:43:44.2730990Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T08:43:44.2731261Z env: 2025-12-04T08:43:44.2731442Z GIT_DEFAULT_BRANCH: main 2025-12-04T08:43:44.2731642Z ##[endgroup] 2025-12-04T08:43:44.4049600Z ##[group]Run if [ "${HAS_NVIDIA}" = "true" ]; then 2025-12-04T08:43:44.4049903Z if [ "${HAS_NVIDIA}" = "true" ]; then 2025-12-04T08:43:44.4050161Z  echo "HAS_NVIDIA_GPU=true" >> "${GITHUB_ENV}" 2025-12-04T08:43:44.4050513Z  echo "GPU_FLAG=--gpus all -e NVIDIA_DRIVER_CAPABILITIES=all" >> "${GITHUB_ENV}" 2025-12-04T08:43:44.4050821Z else 2025-12-04T08:43:44.4051023Z  echo "HAS_NVIDIA_GPU=false" >> "${GITHUB_ENV}" 2025-12-04T08:43:44.4051262Z fi 2025-12-04T08:43:44.4056259Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T08:43:44.4056577Z env: 2025-12-04T08:43:44.4056736Z GIT_DEFAULT_BRANCH: main 2025-12-04T08:43:44.4057032Z HAS_NVIDIA: false 2025-12-04T08:43:44.4057197Z ##[endgroup] 2025-12-04T08:43:44.4138404Z Prepare all required actions 2025-12-04T08:43:44.4211242Z ##[group]Run ./.github/actions/get-workflow-job-id 2025-12-04T08:43:44.4211497Z with: 2025-12-04T08:43:44.4212122Z github-token: *** 2025-12-04T08:43:44.4212302Z env: 2025-12-04T08:43:44.4212465Z GIT_DEFAULT_BRANCH: main 2025-12-04T08:43:44.4212656Z HAS_NVIDIA_GPU: false 2025-12-04T08:43:44.4212832Z ##[endgroup] 2025-12-04T08:43:44.4243011Z ##[group]Run set -eux 2025-12-04T08:43:44.4243222Z set -eux 2025-12-04T08:43:44.4243544Z python3 .github/scripts/get_workflow_job_id.py "${GITHUB_RUN_ID}" "${RUNNER_NAME}" 2025-12-04T08:43:44.4248662Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T08:43:44.4248899Z env: 2025-12-04T08:43:44.4249057Z GIT_DEFAULT_BRANCH: main 2025-12-04T08:43:44.4249245Z HAS_NVIDIA_GPU: false 2025-12-04T08:43:44.4249555Z GITHUB_TOKEN: *** 2025-12-04T08:43:44.4249713Z ##[endgroup] 2025-12-04T08:43:44.4277999Z + python3 .github/scripts/get_workflow_job_id.py 19921726389 i-02dcdd5317aed8049 2025-12-04T08:43:46.6774112Z Setting output job-id=57113856426 2025-12-04T08:43:46.6774731Z Setting output job-name=inductor-cpu-test / test (dynamic_cpu_inductor_huggingface, 1, 1, linux.2xlarge.amx) 2025-12-04T08:43:46.6912782Z ##[group]Run python3 -m pip install psutil==5.9.8 dataclasses_json==0.6.7 nvidia-ml-py==11.525.84 2025-12-04T08:43:46.6913256Z python3 -m pip install psutil==5.9.8 dataclasses_json==0.6.7 nvidia-ml-py==11.525.84 2025-12-04T08:43:46.6913843Z python3 -m tools.stats.monitor --log-interval "$MONITOR_LOG_INTERVAL" --data-collect-interval "$MONITOR_DATA_COLLECT_INTERVAL" > usage_log.txt 2>&1 & 2025-12-04T08:43:46.6914371Z echo "monitor-script-pid=${!}" >> "${GITHUB_OUTPUT}" 2025-12-04T08:43:46.6920374Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T08:43:46.6920622Z env: 2025-12-04T08:43:46.6920951Z GIT_DEFAULT_BRANCH: main 2025-12-04T08:43:46.6921163Z HAS_NVIDIA_GPU: false 2025-12-04T08:43:46.6921342Z JOB_ID: 57113856426 2025-12-04T08:43:46.6921651Z JOB_NAME: inductor-cpu-test / test (dynamic_cpu_inductor_huggingface, 1, 1, linux.2xlarge.amx) 2025-12-04T08:43:46.6921985Z WORKFLOW_NAME: inductor 2025-12-04T08:43:46.6922219Z WORKFLOW_RUN_ID: 19921726389 2025-12-04T08:43:46.6922410Z MONITOR_LOG_INTERVAL: 5 2025-12-04T08:43:46.6922602Z MONITOR_DATA_COLLECT_INTERVAL: 1 2025-12-04T08:43:46.6922804Z ##[endgroup] 2025-12-04T08:43:48.0291150Z Defaulting to user installation because normal site-packages is not writeable 2025-12-04T08:43:48.3098164Z Collecting psutil==5.9.8 2025-12-04T08:43:48.3266415Z Downloading psutil-5.9.8-cp36-abi3-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (288 kB) 2025-12-04T08:43:48.3894631Z Collecting dataclasses_json==0.6.7 2025-12-04T08:43:48.3934470Z Downloading dataclasses_json-0.6.7-py3-none-any.whl (28 kB) 2025-12-04T08:43:48.4197360Z Collecting nvidia-ml-py==11.525.84 2025-12-04T08:43:48.4235762Z Downloading nvidia_ml_py-11.525.84-py3-none-any.whl (34 kB) 2025-12-04T08:43:48.5108165Z Collecting marshmallow<4.0.0,>=3.18.0 2025-12-04T08:43:48.5148820Z Downloading marshmallow-3.26.1-py3-none-any.whl (50 kB) 2025-12-04T08:43:48.5369989Z Collecting typing-inspect<1,>=0.4.0 2025-12-04T08:43:48.5413665Z Downloading typing_inspect-0.9.0-py3-none-any.whl (8.8 kB) 2025-12-04T08:43:48.5855784Z Collecting packaging>=17.0 2025-12-04T08:43:48.5894395Z Downloading packaging-25.0-py3-none-any.whl (66 kB) 2025-12-04T08:43:48.6127958Z Collecting mypy-extensions>=0.3.0 2025-12-04T08:43:48.6165090Z Downloading mypy_extensions-1.1.0-py3-none-any.whl (5.0 kB) 2025-12-04T08:43:48.6554078Z Collecting typing-extensions>=3.7.4 2025-12-04T08:43:48.6595821Z Downloading typing_extensions-4.15.0-py3-none-any.whl (44 kB) 2025-12-04T08:43:48.7352191Z Installing collected packages: typing-extensions, packaging, mypy-extensions, typing-inspect, marshmallow, psutil, nvidia-ml-py, dataclasses-json 2025-12-04T08:43:48.9870637Z Successfully installed dataclasses-json-0.6.7 marshmallow-3.26.1 mypy-extensions-1.1.0 nvidia-ml-py-11.525.84 packaging-25.0 psutil-5.9.8 typing-extensions-4.15.0 typing-inspect-0.9.0 2025-12-04T08:43:49.1516852Z Prepare all required actions 2025-12-04T08:43:49.1517211Z Getting action download info 2025-12-04T08:43:49.3356383Z Download action repository 'seemethere/download-artifact-s3@v4' (SHA:1da556a7aa0a088e3153970611f6c432d58e80e6) 2025-12-04T08:43:49.5975332Z Download action repository 'actions/download-artifact@v4' (SHA:d3f86a106a0bac45b974a628896c90dbdf5c8093) 2025-12-04T08:43:49.9900653Z ##[group]Run ./.github/actions/download-build-artifacts 2025-12-04T08:43:49.9900966Z with: 2025-12-04T08:43:49.9901164Z name: linux-jammy-py3.10-gcc11-build 2025-12-04T08:43:49.9901400Z s3-bucket: gha-artifacts 2025-12-04T08:43:49.9901583Z env: 2025-12-04T08:43:49.9901748Z GIT_DEFAULT_BRANCH: main 2025-12-04T08:43:49.9901941Z HAS_NVIDIA_GPU: false 2025-12-04T08:43:49.9902133Z ##[endgroup] 2025-12-04T08:43:49.9941664Z ##[group]Run seemethere/download-artifact-s3@v4 2025-12-04T08:43:49.9941920Z with: 2025-12-04T08:43:49.9942113Z name: linux-jammy-py3.10-gcc11-build 2025-12-04T08:43:49.9942393Z s3-bucket: gha-artifacts 2025-12-04T08:43:49.9942596Z region: us-east-1 2025-12-04T08:43:49.9942761Z env: 2025-12-04T08:43:49.9942931Z GIT_DEFAULT_BRANCH: main 2025-12-04T08:43:49.9943128Z HAS_NVIDIA_GPU: false 2025-12-04T08:43:49.9943311Z ##[endgroup] 2025-12-04T08:43:50.3604231Z (node:44969) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2025-12-04T08:43:50.3604624Z 2025-12-04T08:43:50.3604774Z Please migrate your code to use AWS SDK for JavaScript (v3). 2025-12-04T08:43:50.3605164Z For more information, check the migration guide at https://a.co/7PzMCcy 2025-12-04T08:43:50.3605555Z (Use `node --trace-warnings ...` to show where the warning was created) 2025-12-04T08:43:50.5948674Z Found 1 objects with prefix pytorch/pytorch/19921726389/linux-jammy-py3.10-gcc11-build/ 2025-12-04T08:43:50.5953498Z Starting download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2025-12-04T08:43:55.3028154Z Finished download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2025-12-04T08:43:55.3033765Z Artifact download has finished successfully 2025-12-04T08:43:55.3201777Z ##[group]Run unzip -o artifacts.zip 2025-12-04T08:43:55.3202037Z unzip -o artifacts.zip 2025-12-04T08:43:55.3207313Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T08:43:55.3207581Z env: 2025-12-04T08:43:55.3207745Z GIT_DEFAULT_BRANCH: main 2025-12-04T08:43:55.3207947Z HAS_NVIDIA_GPU: false 2025-12-04T08:43:55.3208138Z ##[endgroup] 2025-12-04T08:43:55.3270889Z Archive: artifacts.zip 2025-12-04T08:43:55.3271251Z creating: dist/ 2025-12-04T08:43:56.4934275Z inflating: dist/torch-2.10.0a0+gitffd9b0f-cp310-cp310-linux_x86_64.whl 2025-12-04T08:43:56.4935692Z creating: dist/vision/ 2025-12-04T08:43:56.5014430Z inflating: dist/vision/torchvision-0.25.0a0+617079d-cp310-cp310-linux_x86_64.whl 2025-12-04T08:43:56.5014848Z creating: dist/audio/ 2025-12-04T08:43:56.5042179Z inflating: dist/audio/torchaudio-2.10.0a0+e90a398-cp310-cp310-linux_x86_64.whl 2025-12-04T08:43:56.5042589Z creating: dist/ao/ 2025-12-04T08:43:56.5083426Z inflating: dist/ao/torchao-0.7.0+git51c87b6e-py3-none-any.whl 2025-12-04T08:43:56.5206822Z inflating: dist/.ninja_log 2025-12-04T08:43:56.5207158Z creating: build/custom_test_artifacts/ 2025-12-04T08:43:56.5207494Z creating: build/custom_test_artifacts/custom-op-build/ 2025-12-04T08:43:56.5207867Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/ 2025-12-04T08:43:56.5208330Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/pkgRedirects/ 2025-12-04T08:43:56.5210212Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeConfigureLog.yaml 2025-12-04T08:43:56.5211067Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/ 2025-12-04T08:43:56.5211533Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CMakeSystem.cmake 2025-12-04T08:43:56.5212204Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CompilerIdC/ 2025-12-04T08:43:56.5212700Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CompilerIdC/tmp/ 2025-12-04T08:43:56.5213659Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CompilerIdC/CMakeCCompilerId.c 2025-12-04T08:43:56.5215196Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CompilerIdC/a.out 2025-12-04T08:43:56.5215723Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CMakeCCompiler.cmake 2025-12-04T08:43:56.5216232Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CompilerIdCXX/ 2025-12-04T08:43:56.5216715Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CompilerIdCXX/tmp/ 2025-12-04T08:43:56.5218442Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CompilerIdCXX/CMakeCXXCompilerId.cpp 2025-12-04T08:43:56.5219446Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CompilerIdCXX/a.out 2025-12-04T08:43:56.5220210Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CMakeCXXCompiler.cmake 2025-12-04T08:43:56.5225592Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CMakeDetermineCompilerABI_C.bin 2025-12-04T08:43:56.5226261Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CMakeDetermineCompilerABI_CXX.bin 2025-12-04T08:43:56.5226793Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeScratch/ 2025-12-04T08:43:56.5227251Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/cmake.check_cache 2025-12-04T08:43:56.5227706Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/ 2025-12-04T08:43:56.5228204Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.ts 2025-12-04T08:43:56.5228747Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.make 2025-12-04T08:43:56.5229263Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/depend.make 2025-12-04T08:43:56.5229737Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/link.txt 2025-12-04T08:43:56.5230233Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/cmake_clean.cmake 2025-12-04T08:43:56.5230745Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/build.make 2025-12-04T08:43:56.5231276Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/DependInfo.cmake 2025-12-04T08:43:56.5231768Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/flags.make 2025-12-04T08:43:56.5232261Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/progress.make 2025-12-04T08:43:56.5251663Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o.d 2025-12-04T08:43:56.5442316Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o 2025-12-04T08:43:56.5445005Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/ 2025-12-04T08:43:56.5445550Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.ts 2025-12-04T08:43:56.5446123Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.make 2025-12-04T08:43:56.5446668Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/depend.make 2025-12-04T08:43:56.5447168Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/link.txt 2025-12-04T08:43:56.5448070Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/cmake_clean.cmake 2025-12-04T08:43:56.5448783Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/build.make 2025-12-04T08:43:56.5449291Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/DependInfo.cmake 2025-12-04T08:43:56.5449804Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/flags.make 2025-12-04T08:43:56.5450319Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/progress.make 2025-12-04T08:43:56.5464503Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o.d 2025-12-04T08:43:56.5542682Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o 2025-12-04T08:43:56.5543376Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeDirectoryInformation.cmake 2025-12-04T08:43:56.5543878Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/TargetDirectories.txt 2025-12-04T08:43:56.5544353Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/progress.marks 2025-12-04T08:43:56.5544795Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile2 2025-12-04T08:43:56.5545228Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile.cmake 2025-12-04T08:43:56.5545686Z inflating: build/custom_test_artifacts/custom-op-build/CMakeCache.txt 2025-12-04T08:43:56.5546798Z inflating: build/custom_test_artifacts/custom-op-build/Makefile 2025-12-04T08:43:56.5547290Z inflating: build/custom_test_artifacts/custom-op-build/cmake_install.cmake 2025-12-04T08:43:56.5709792Z inflating: build/custom_test_artifacts/custom-op-build/libcustom_ops.so 2025-12-04T08:43:56.5764154Z inflating: build/custom_test_artifacts/custom-op-build/test_custom_ops 2025-12-04T08:43:56.5764699Z creating: build/custom_test_artifacts/jit-hook-build/ 2025-12-04T08:43:56.5765060Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/ 2025-12-04T08:43:56.5765444Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/pkgRedirects/ 2025-12-04T08:43:56.5765892Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeConfigureLog.yaml 2025-12-04T08:43:56.5766372Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/ 2025-12-04T08:43:56.5766811Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CMakeSystem.cmake 2025-12-04T08:43:56.5767261Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CompilerIdC/ 2025-12-04T08:43:56.5767732Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CompilerIdC/tmp/ 2025-12-04T08:43:56.5768242Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CompilerIdC/CMakeCCompilerId.c 2025-12-04T08:43:56.5774068Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CompilerIdC/a.out 2025-12-04T08:43:56.5777505Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CMakeCCompiler.cmake 2025-12-04T08:43:56.5781345Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CompilerIdCXX/ 2025-12-04T08:43:56.5781951Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CompilerIdCXX/tmp/ 2025-12-04T08:43:56.5782507Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CompilerIdCXX/CMakeCXXCompilerId.cpp 2025-12-04T08:43:56.5783085Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CompilerIdCXX/a.out 2025-12-04T08:43:56.5783592Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CMakeCXXCompiler.cmake 2025-12-04T08:43:56.5784135Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CMakeDetermineCompilerABI_C.bin 2025-12-04T08:43:56.5785000Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CMakeDetermineCompilerABI_CXX.bin 2025-12-04T08:43:56.5785622Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeScratch/ 2025-12-04T08:43:56.5786064Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/cmake.check_cache 2025-12-04T08:43:56.5786521Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/ 2025-12-04T08:43:56.5787018Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.ts 2025-12-04T08:43:56.5787596Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.make 2025-12-04T08:43:56.5788131Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/depend.make 2025-12-04T08:43:56.5788681Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/link.txt 2025-12-04T08:43:56.5789223Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/cmake_clean.cmake 2025-12-04T08:43:56.5789743Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/build.make 2025-12-04T08:43:56.5790293Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/DependInfo.cmake 2025-12-04T08:43:56.5790835Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/flags.make 2025-12-04T08:43:56.5791332Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/progress.make 2025-12-04T08:43:56.5800686Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o.d 2025-12-04T08:43:56.5865346Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o 2025-12-04T08:43:56.5866013Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeDirectoryInformation.cmake 2025-12-04T08:43:56.5866587Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/TargetDirectories.txt 2025-12-04T08:43:56.5867059Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/progress.marks 2025-12-04T08:43:56.5867484Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile2 2025-12-04T08:43:56.5867913Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile.cmake 2025-12-04T08:43:56.5868317Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeCache.txt 2025-12-04T08:43:56.5868701Z inflating: build/custom_test_artifacts/jit-hook-build/Makefile 2025-12-04T08:43:56.5869071Z inflating: build/custom_test_artifacts/jit-hook-build/cmake_install.cmake 2025-12-04T08:43:56.5903553Z inflating: build/custom_test_artifacts/jit-hook-build/test_jit_hooks 2025-12-04T08:43:56.5904703Z creating: build/custom_test_artifacts/custom-backend-build/ 2025-12-04T08:43:56.5905150Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/ 2025-12-04T08:43:56.5905589Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/pkgRedirects/ 2025-12-04T08:43:56.5907129Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeConfigureLog.yaml 2025-12-04T08:43:56.5907602Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/ 2025-12-04T08:43:56.5908065Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CMakeSystem.cmake 2025-12-04T08:43:56.5908551Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CompilerIdC/ 2025-12-04T08:43:56.5909029Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CompilerIdC/tmp/ 2025-12-04T08:43:56.5910022Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CompilerIdC/CMakeCCompilerId.c 2025-12-04T08:43:56.5911929Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CompilerIdC/a.out 2025-12-04T08:43:56.5912758Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CMakeCCompiler.cmake 2025-12-04T08:43:56.5913411Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CompilerIdCXX/ 2025-12-04T08:43:56.5913901Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CompilerIdCXX/tmp/ 2025-12-04T08:43:56.5915525Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CompilerIdCXX/CMakeCXXCompilerId.cpp 2025-12-04T08:43:56.5917206Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CompilerIdCXX/a.out 2025-12-04T08:43:56.5917772Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CMakeCXXCompiler.cmake 2025-12-04T08:43:56.5919065Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CMakeDetermineCompilerABI_C.bin 2025-12-04T08:43:56.5921109Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CMakeDetermineCompilerABI_CXX.bin 2025-12-04T08:43:56.5921670Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeScratch/ 2025-12-04T08:43:56.5922135Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/cmake.check_cache 2025-12-04T08:43:56.5922613Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/ 2025-12-04T08:43:56.5923141Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.ts 2025-12-04T08:43:56.5923760Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.make 2025-12-04T08:43:56.5924323Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/depend.make 2025-12-04T08:43:56.5924851Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/link.txt 2025-12-04T08:43:56.5925390Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/cmake_clean.cmake 2025-12-04T08:43:56.5926040Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/build.make 2025-12-04T08:43:56.5931431Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/DependInfo.cmake 2025-12-04T08:43:56.5932071Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/flags.make 2025-12-04T08:43:56.5932647Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/progress.make 2025-12-04T08:43:56.5933265Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o.d 2025-12-04T08:43:56.6048943Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o 2025-12-04T08:43:56.6049593Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/ 2025-12-04T08:43:56.6050137Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.ts 2025-12-04T08:43:56.6050724Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.make 2025-12-04T08:43:56.6051289Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/depend.make 2025-12-04T08:43:56.6051817Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/link.txt 2025-12-04T08:43:56.6052461Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/cmake_clean.cmake 2025-12-04T08:43:56.6053038Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/build.make 2025-12-04T08:43:56.6053614Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/DependInfo.cmake 2025-12-04T08:43:56.6054488Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/flags.make 2025-12-04T08:43:56.6055130Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/progress.make 2025-12-04T08:43:56.6069655Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o.d 2025-12-04T08:43:56.6123574Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o 2025-12-04T08:43:56.6126341Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeDirectoryInformation.cmake 2025-12-04T08:43:56.6126927Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/TargetDirectories.txt 2025-12-04T08:43:56.6127458Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/progress.marks 2025-12-04T08:43:56.6127951Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile2 2025-12-04T08:43:56.6128410Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile.cmake 2025-12-04T08:43:56.6128857Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeCache.txt 2025-12-04T08:43:56.6129273Z inflating: build/custom_test_artifacts/custom-backend-build/Makefile 2025-12-04T08:43:56.6129695Z inflating: build/custom_test_artifacts/custom-backend-build/cmake_install.cmake 2025-12-04T08:43:56.6226782Z inflating: build/custom_test_artifacts/custom-backend-build/libcustom_backend.so 2025-12-04T08:43:56.6266833Z inflating: build/custom_test_artifacts/custom-backend-build/test_custom_backend 2025-12-04T08:43:56.6267235Z creating: build/lib/ 2025-12-04T08:43:56.6349764Z inflating: build/lib/libprotobuf-lite.a 2025-12-04T08:43:56.6773286Z inflating: build/lib/libprotobuf.a 2025-12-04T08:43:56.7257795Z inflating: build/lib/libprotoc.a 2025-12-04T08:43:56.7270502Z inflating: build/lib/libpthreadpool.a 2025-12-04T08:43:56.7271079Z inflating: build/lib/libcpuinfo.a 2025-12-04T08:43:56.7282499Z inflating: build/lib/libcpuinfo_internals.a 2025-12-04T08:43:56.7282793Z inflating: build/lib/libclog.a 2025-12-04T08:43:56.7299710Z inflating: build/lib/libpytorch_qnnpack.a 2025-12-04T08:43:56.7300071Z inflating: build/lib/libnnpack_reference_layers.a 2025-12-04T08:43:56.7486846Z inflating: build/lib/libmicrokernels-prod.a 2025-12-04T08:43:56.7503004Z inflating: build/lib/libnnpack.a 2025-12-04T08:43:56.8378905Z inflating: build/lib/libmicrokernels-all.a 2025-12-04T08:43:56.8448711Z inflating: build/lib/libgtest.a 2025-12-04T08:43:56.8464642Z inflating: build/lib/libgmock.a 2025-12-04T08:43:56.8465476Z inflating: build/lib/libgmock_main.a 2025-12-04T08:43:56.8465787Z inflating: build/lib/libgtest_main.a 2025-12-04T08:43:56.8557908Z inflating: build/lib/libXNNPACK.a 2025-12-04T08:43:56.8631618Z inflating: build/lib/libbenchmark.a 2025-12-04T08:43:56.8631977Z inflating: build/lib/libbenchmark_main.a 2025-12-04T08:43:56.8639460Z inflating: build/lib/libittnotify.a 2025-12-04T08:43:56.8639784Z inflating: build/lib/libjitprofiling.a 2025-12-04T08:43:56.8705962Z inflating: build/lib/libasmjit.a 2025-12-04T08:43:56.9824682Z inflating: build/lib/libfbgemm.a 2025-12-04T08:43:56.9853883Z inflating: build/lib/libtensorpipe_uv.a 2025-12-04T08:43:57.0374921Z inflating: build/lib/libtensorpipe.a 2025-12-04T08:43:57.0493650Z inflating: build/lib/libgloo.a 2025-12-04T08:43:57.0541020Z inflating: build/lib/libonnx_proto.a 2025-12-04T08:43:57.1232764Z inflating: build/lib/libonnx.a 2025-12-04T08:43:58.0775071Z inflating: build/lib/libdnnl.a 2025-12-04T08:43:58.0792054Z inflating: build/lib/libfmt.a 2025-12-04T08:43:58.1059126Z inflating: build/lib/libkineto.a 2025-12-04T08:43:58.1172572Z inflating: build/lib/libc10.so 2025-12-04T08:43:58.1174530Z inflating: build/lib/libtorch_global_deps.so 2025-12-04T08:44:00.9691575Z inflating: build/lib/libtorch_cpu.so 2025-12-04T08:44:00.9692245Z inflating: build/lib/libtorch.so 2025-12-04T08:44:00.9757484Z inflating: build/lib/libtorchbind_test.so 2025-12-04T08:44:00.9773137Z inflating: build/lib/libjitbackend_test.so 2025-12-04T08:44:00.9796287Z inflating: build/lib/libbackend_with_compiler.so 2025-12-04T08:44:00.9818257Z inflating: build/lib/libaoti_custom_ops.so 2025-12-04T08:44:00.9821616Z inflating: build/lib/libshm.so 2025-12-04T08:44:01.1978041Z inflating: build/lib/libtorch_python.so 2025-12-04T08:44:01.2011463Z inflating: build/lib/libnnapi_backend.so 2025-12-04T08:44:01.2014425Z creating: build/bin/ 2025-12-04T08:44:01.2014649Z creating: build/bin/CMakeFiles/ 2025-12-04T08:44:01.2014920Z inflating: build/bin/cmake_install.cmake 2025-12-04T08:44:01.2015199Z inflating: build/bin/CTestTestfile.cmake 2025-12-04T08:44:01.2451750Z inflating: build/bin/protoc-3.13.0.0 2025-12-04T08:44:01.2889267Z inflating: build/bin/protoc 2025-12-04T08:44:01.2949958Z inflating: build/bin/c10_AllocatorConfig_test 2025-12-04T08:44:01.2999818Z inflating: build/bin/c10_CompileTimeFunctionPointer_test 2025-12-04T08:44:01.3053884Z inflating: build/bin/c10_DeviceGuard_test 2025-12-04T08:44:01.3116341Z inflating: build/bin/c10_DispatchKeySet_test 2025-12-04T08:44:01.3172085Z inflating: build/bin/c10_Device_test 2025-12-04T08:44:01.3225422Z inflating: build/bin/c10_StreamGuard_test 2025-12-04T08:44:01.3285084Z inflating: build/bin/c10_Scalar_test 2025-12-04T08:44:01.3344540Z inflating: build/bin/c10_SizesAndStrides_test 2025-12-04T08:44:01.3402009Z inflating: build/bin/c10_SymInt_test 2025-12-04T08:44:01.3459746Z inflating: build/bin/c10_InlineDeviceGuard_test 2025-12-04T08:44:01.3514867Z inflating: build/bin/c10_InlineStreamGuard_test 2025-12-04T08:44:01.3586690Z inflating: build/bin/c10_cow_test 2025-12-04T08:44:01.3637692Z inflating: build/bin/c10_ArrayRef_test 2025-12-04T08:44:01.3689046Z inflating: build/bin/c10_ConstexprCrc_test 2025-12-04T08:44:01.3746227Z inflating: build/bin/c10_Bitset_test 2025-12-04T08:44:01.3796273Z inflating: build/bin/c10_DeadlockDetection_test 2025-12-04T08:44:01.3857448Z inflating: build/bin/c10_Enumerate_test 2025-12-04T08:44:01.3913758Z inflating: build/bin/c10_IntrusiveList_test 2025-12-04T08:44:01.3971031Z inflating: build/bin/c10_Half_test 2025-12-04T08:44:01.4027380Z inflating: build/bin/c10_LeftRight_test 2025-12-04T08:44:01.4084721Z inflating: build/bin/c10_NetworkFlow_test 2025-12-04T08:44:01.4138781Z inflating: build/bin/c10_Semaphore_test 2025-12-04T08:44:01.4192119Z inflating: build/bin/c10_Synchronized_test 2025-12-04T08:44:01.4253109Z inflating: build/bin/c10_ThreadLocal_test 2025-12-04T08:44:01.4304468Z inflating: build/bin/c10_TypeIndex_test 2025-12-04T08:44:01.4356974Z inflating: build/bin/c10_accumulate_test 2025-12-04T08:44:01.4416074Z inflating: build/bin/c10_bfloat16_test 2025-12-04T08:44:01.4468368Z inflating: build/bin/c10_bit_cast_test 2025-12-04T08:44:01.4523736Z inflating: build/bin/c10_complex_math_test 2025-12-04T08:44:01.4582696Z inflating: build/bin/c10_complex_test 2025-12-04T08:44:01.4630920Z inflating: build/bin/c10_error_test 2025-12-04T08:44:01.4683861Z inflating: build/bin/c10_exception_test 2025-12-04T08:44:01.4738587Z inflating: build/bin/c10_flags_test 2025-12-04T08:44:01.4793694Z inflating: build/bin/c10_generic_math_test 2025-12-04T08:44:01.4845188Z inflating: build/bin/c10_irange_test 2025-12-04T08:44:01.4902088Z inflating: build/bin/c10_lazy_test 2025-12-04T08:44:01.5059134Z inflating: build/bin/c10_intrusive_ptr_test 2025-12-04T08:44:01.5112191Z inflating: build/bin/c10_nofatal_test 2025-12-04T08:44:01.5171418Z inflating: build/bin/c10_logging_test 2025-12-04T08:44:01.5249074Z inflating: build/bin/c10_optional_test 2025-12-04T08:44:01.5304188Z inflating: build/bin/c10_registry_test 2025-12-04T08:44:01.5451923Z inflating: build/bin/c10_small_vector_test 2025-12-04T08:44:01.5513962Z inflating: build/bin/c10_ordered_preserving_dict_test 2025-12-04T08:44:01.5576781Z inflating: build/bin/c10_string_util_test 2025-12-04T08:44:01.5628951Z inflating: build/bin/c10_ssize_test 2025-12-04T08:44:01.5677718Z inflating: build/bin/c10_string_view_test 2025-12-04T08:44:01.5723962Z inflating: build/bin/c10_intrusive_ptr_benchmark 2025-12-04T08:44:01.5781349Z inflating: build/bin/c10_tempfile_test 2025-12-04T08:44:01.5840765Z inflating: build/bin/c10_typeid_test 2025-12-04T08:44:01.6400566Z inflating: build/bin/vec_test_all_types_DEFAULT 2025-12-04T08:44:01.6983382Z inflating: build/bin/vec_test_all_types_AVX512 2025-12-04T08:44:01.7555938Z inflating: build/bin/vec_test_all_types_AVX2 2025-12-04T08:44:01.7660017Z inflating: build/bin/test_aoti_abi_check 2025-12-04T08:44:01.7713984Z inflating: build/bin/test_vec_half_DEFAULT 2025-12-04T08:44:01.7768928Z inflating: build/bin/test_vec_half_AVX2 2025-12-04T08:44:01.7828118Z inflating: build/bin/test_vec_half_AVX512 2025-12-04T08:44:01.7883647Z inflating: build/bin/FileStoreTest 2025-12-04T08:44:01.7941116Z inflating: build/bin/BackoffTest 2025-12-04T08:44:01.8000673Z inflating: build/bin/static_runtime_bench 2025-12-04T08:44:01.8262180Z inflating: build/bin/static_runtime_test 2025-12-04T08:44:01.8342296Z inflating: build/bin/Dict_test 2025-12-04T08:44:01.8397564Z inflating: build/bin/Dimname_test 2025-12-04T08:44:01.8461775Z inflating: build/bin/MaybeOwned_test 2025-12-04T08:44:01.8524487Z inflating: build/bin/NamedTensor_test 2025-12-04T08:44:01.8589749Z inflating: build/bin/apply_utils_test 2025-12-04T08:44:01.8654431Z inflating: build/bin/atest 2025-12-04T08:44:01.8722803Z inflating: build/bin/basic 2025-12-04T08:44:01.8782080Z inflating: build/bin/broadcast_test 2025-12-04T08:44:01.8838670Z inflating: build/bin/cpu_allocator_test 2025-12-04T08:44:01.8903324Z inflating: build/bin/cpu_generator_test 2025-12-04T08:44:01.8960636Z inflating: build/bin/cpu_profiling_allocator_test 2025-12-04T08:44:01.9058187Z inflating: build/bin/cpu_rng_test 2025-12-04T08:44:01.9110894Z inflating: build/bin/dlconvertor_test 2025-12-04T08:44:01.9170849Z inflating: build/bin/extension_backend_test 2025-12-04T08:44:01.9228667Z inflating: build/bin/half_test 2025-12-04T08:44:01.9329758Z inflating: build/bin/ivalue_test 2025-12-04T08:44:01.9381010Z inflating: build/bin/lazy_tensor_test 2025-12-04T08:44:01.9433523Z inflating: build/bin/math_kernel_test 2025-12-04T08:44:01.9493192Z inflating: build/bin/memory_format_test 2025-12-04T08:44:01.9549995Z inflating: build/bin/memory_overlapping_test 2025-12-04T08:44:01.9602577Z inflating: build/bin/mobile_memory_cleanup 2025-12-04T08:44:01.9660865Z inflating: build/bin/native_test 2025-12-04T08:44:01.9713180Z inflating: build/bin/operator_name_test 2025-12-04T08:44:01.9767908Z inflating: build/bin/operators_test 2025-12-04T08:44:01.9823080Z inflating: build/bin/packedtensoraccessor_test 2025-12-04T08:44:01.9891094Z inflating: build/bin/pow_test 2025-12-04T08:44:01.9948957Z inflating: build/bin/quantized_test 2025-12-04T08:44:01.9996816Z inflating: build/bin/reduce_ops_test 2025-12-04T08:44:02.0051444Z inflating: build/bin/reportMemoryUsage_test 2025-12-04T08:44:02.0109447Z inflating: build/bin/scalar_tensor_test 2025-12-04T08:44:02.0170116Z inflating: build/bin/scalar_test 2025-12-04T08:44:02.0223118Z inflating: build/bin/StorageUtils_test 2025-12-04T08:44:02.0272429Z inflating: build/bin/stride_properties_test 2025-12-04T08:44:02.0352598Z inflating: build/bin/tensor_iterator_test 2025-12-04T08:44:02.0406928Z inflating: build/bin/test_parallel 2025-12-04T08:44:02.0467453Z inflating: build/bin/type_ptr_test 2025-12-04T08:44:02.0519513Z inflating: build/bin/thread_init_test 2025-12-04T08:44:02.0583935Z inflating: build/bin/type_test 2025-12-04T08:44:02.0640534Z inflating: build/bin/undefined_tensor_test 2025-12-04T08:44:02.0697033Z inflating: build/bin/verify_api_visibility 2025-12-04T08:44:02.0767368Z inflating: build/bin/legacy_vmap_test 2025-12-04T08:44:02.0819914Z inflating: build/bin/weakref_test 2025-12-04T08:44:02.0872688Z inflating: build/bin/wrapdim_test 2025-12-04T08:44:02.0924904Z inflating: build/bin/xla_tensor_test 2025-12-04T08:44:02.0983414Z inflating: build/bin/IListRef_test 2025-12-04T08:44:02.1090210Z inflating: build/bin/List_test 2025-12-04T08:44:02.1162626Z inflating: build/bin/KernelFunction_test 2025-12-04T08:44:02.1281564Z inflating: build/bin/kernel_function_legacy_test 2025-12-04T08:44:02.1379989Z inflating: build/bin/kernel_function_test 2025-12-04T08:44:02.1509073Z inflating: build/bin/kernel_lambda_legacy_test 2025-12-04T08:44:02.1617369Z inflating: build/bin/kernel_lambda_test 2025-12-04T08:44:02.1677014Z inflating: build/bin/kernel_stackbased_test 2025-12-04T08:44:02.1773999Z inflating: build/bin/make_boxed_from_unboxed_functor_test 2025-12-04T08:44:02.1832600Z inflating: build/bin/CppSignature_test 2025-12-04T08:44:02.1889469Z inflating: build/bin/backend_fallback_test 2025-12-04T08:44:02.1944271Z inflating: build/bin/op_allowlist_test 2025-12-04T08:44:02.2240488Z inflating: build/bin/op_registration_test 2025-12-04T08:44:02.2306079Z inflating: build/bin/inline_container_test 2025-12-04T08:44:02.2649694Z inflating: build/bin/test_lazy 2025-12-04T08:44:02.2711391Z inflating: build/bin/TCPStoreTest 2025-12-04T08:44:02.2765764Z inflating: build/bin/HashStoreTest 2025-12-04T08:44:02.3836160Z inflating: build/bin/test_jit 2025-12-04T08:44:02.3899162Z inflating: build/bin/test_aoti_inference 2025-12-04T08:44:02.3899516Z inflating: build/bin/example_allreduce 2025-12-04T08:44:02.3975186Z inflating: build/bin/ProcessGroupGlooTest 2025-12-04T08:44:02.4035573Z inflating: build/bin/test_dist_autograd 2025-12-04T08:44:02.4109973Z inflating: build/bin/test_cpp_rpc 2025-12-04T08:44:02.5253834Z inflating: build/bin/test_api 2025-12-04T08:44:02.5254140Z inflating: build/bin/parallel_benchmark 2025-12-04T08:44:02.5258890Z inflating: build/bin/torch_shm_manager 2025-12-04T08:44:02.5259323Z creating: .additional_ci_files/ 2025-12-04T08:44:02.5319142Z inflating: .additional_ci_files/test-times.json 2025-12-04T08:44:02.5542678Z inflating: .additional_ci_files/test-class-times.json 2025-12-04T08:44:02.5651777Z ##[group]Run rm artifacts.zip 2025-12-04T08:44:02.5652004Z rm artifacts.zip 2025-12-04T08:44:02.5657223Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T08:44:02.5657507Z env: 2025-12-04T08:44:02.5657675Z GIT_DEFAULT_BRANCH: main 2025-12-04T08:44:02.5657875Z HAS_NVIDIA_GPU: false 2025-12-04T08:44:02.5658087Z ##[endgroup] 2025-12-04T08:44:02.6015388Z ##[group]Run df -H 2025-12-04T08:44:02.6015583Z df -H 2025-12-04T08:44:02.6020059Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T08:44:02.6020323Z env: 2025-12-04T08:44:02.6020475Z GIT_DEFAULT_BRANCH: main 2025-12-04T08:44:02.6020659Z HAS_NVIDIA_GPU: false 2025-12-04T08:44:02.6021121Z ##[endgroup] 2025-12-04T08:44:02.6423026Z Filesystem Size Used Avail Use% Mounted on 2025-12-04T08:44:02.6423566Z devtmpfs 4.2M 0 4.2M 0% /dev 2025-12-04T08:44:02.6423907Z tmpfs 17G 0 17G 0% /dev/shm 2025-12-04T08:44:02.6424249Z tmpfs 6.7G 676k 6.7G 1% /run 2025-12-04T08:44:02.6424563Z /dev/nvme0n1p1 215G 72G 144G 34% / 2025-12-04T08:44:02.6424865Z tmpfs 17G 13k 17G 1% /tmp 2025-12-04T08:44:02.6425199Z /dev/nvme0n1p128 11M 1.4M 9.2M 13% /boot/efi 2025-12-04T08:44:02.6482209Z Prepare all required actions 2025-12-04T08:44:02.6483154Z Getting action download info 2025-12-04T08:44:02.8142196Z ##[group]Run ./.github/actions/download-td-artifacts 2025-12-04T08:44:02.8142489Z with: 2025-12-04T08:44:02.8142672Z env: 2025-12-04T08:44:02.8142888Z GIT_DEFAULT_BRANCH: main 2025-12-04T08:44:02.8143110Z HAS_NVIDIA_GPU: false 2025-12-04T08:44:02.8143312Z ##[endgroup] 2025-12-04T08:44:02.8237692Z ##[group]Run seemethere/download-artifact-s3@v4 2025-12-04T08:44:02.8237951Z with: 2025-12-04T08:44:02.8238323Z name: td_results 2025-12-04T08:44:02.8238531Z s3-bucket: gha-artifacts 2025-12-04T08:44:02.8238744Z region: us-east-1 2025-12-04T08:44:02.8238913Z env: 2025-12-04T08:44:02.8239088Z GIT_DEFAULT_BRANCH: main 2025-12-04T08:44:02.8239291Z HAS_NVIDIA_GPU: false 2025-12-04T08:44:02.8239486Z ##[endgroup] 2025-12-04T08:44:03.4542867Z (node:44988) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2025-12-04T08:44:03.4543515Z 2025-12-04T08:44:03.4543684Z Please migrate your code to use AWS SDK for JavaScript (v3). 2025-12-04T08:44:03.4544115Z For more information, check the migration guide at https://a.co/7PzMCcy 2025-12-04T08:44:03.4544517Z (Use `node --trace-warnings ...` to show where the warning was created) 2025-12-04T08:44:03.5403412Z Found 0 objects with prefix pytorch/pytorch/19921726389/td_results/ 2025-12-04T08:44:03.5410054Z Artifact download has finished successfully 2025-12-04T08:44:03.5692678Z ##[group]Run mkdir -p .additional_ci_files 2025-12-04T08:44:03.5692967Z mkdir -p .additional_ci_files 2025-12-04T08:44:03.5693279Z mv td_results.json .additional_ci_files/td_results.json || true 2025-12-04T08:44:03.5699489Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T08:44:03.5699755Z env: 2025-12-04T08:44:03.5699937Z GIT_DEFAULT_BRANCH: main 2025-12-04T08:44:03.5700136Z HAS_NVIDIA_GPU: false 2025-12-04T08:44:03.5700318Z ##[endgroup] 2025-12-04T08:44:03.5939675Z mv: cannot stat 'td_results.json': No such file or directory 2025-12-04T08:44:03.6008288Z ##[group]Run .github/scripts/parse_ref.py 2025-12-04T08:44:03.6008584Z .github/scripts/parse_ref.py 2025-12-04T08:44:03.6013614Z shell: /usr/bin/bash -e {0} 2025-12-04T08:44:03.6013820Z env: 2025-12-04T08:44:03.6013987Z GIT_DEFAULT_BRANCH: main 2025-12-04T08:44:03.6014187Z HAS_NVIDIA_GPU: false 2025-12-04T08:44:03.6014368Z ##[endgroup] 2025-12-04T08:44:03.6218866Z Setting output branch=main 2025-12-04T08:44:03.6317183Z Prepare all required actions 2025-12-04T08:44:03.6317540Z Getting action download info 2025-12-04T08:44:03.7860401Z ##[group]Run ./.github/actions/filter-test-configs 2025-12-04T08:44:03.7860693Z with: 2025-12-04T08:44:03.7861250Z github-token: *** 2025-12-04T08:44:03.7863345Z test-matrix: {"include": [{"config": "cpu_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.2xlarge.amx"}, {"config": "cpu_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.2xlarge.amx"}, {"config": "dynamic_cpu_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.2xlarge.amx"}, {"config": "dynamic_cpu_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.2xlarge.amx"}, {"config": "dynamic_cpu_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.2xlarge.amx"}, {"config": "dynamic_cpu_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.2xlarge.amx"}, {"config": "dynamic_cpu_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.2xlarge.amx"}, {"config": "inductor_torchbench_cpu_smoketest_perf", "shard": 1, "num_shards": 1, "runner": "linux.24xl.spr-metal"}]} 2025-12-04T08:44:03.7865446Z job-name: inductor-cpu-test / test (dynamic_cpu_inductor_huggingface, 1, 1, linux.2xlarge.amx) 2025-12-04T08:44:03.7865840Z env: 2025-12-04T08:44:03.7866029Z GIT_DEFAULT_BRANCH: main 2025-12-04T08:44:03.7866270Z HAS_NVIDIA_GPU: false 2025-12-04T08:44:03.7866478Z ##[endgroup] 2025-12-04T08:44:03.7905610Z ##[group]Run nick-fields/retry@v3.0.0 2025-12-04T08:44:03.7905866Z with: 2025-12-04T08:44:03.7906053Z shell: bash 2025-12-04T08:44:03.7906383Z timeout_minutes: 10 2025-12-04T08:44:03.7906581Z max_attempts: 5 2025-12-04T08:44:03.7906786Z retry_wait_seconds: 30 2025-12-04T08:44:03.7907377Z command: set -eux # PyYAML 6.0 doesn't work with MacOS x86 anymore # This must run on Python-3.7 (AmazonLinux2) so can't use request=3.32.2 python3 -m pip install requests==2.27.1 pyyaml==6.0.2 2025-12-04T08:44:03.7908010Z polling_interval_seconds: 1 2025-12-04T08:44:03.7908241Z warning_on_retry: true 2025-12-04T08:44:03.7908467Z continue_on_error: false 2025-12-04T08:44:03.7908687Z env: 2025-12-04T08:44:03.7908868Z GIT_DEFAULT_BRANCH: main 2025-12-04T08:44:03.7909093Z HAS_NVIDIA_GPU: false 2025-12-04T08:44:03.7909478Z GITHUB_TOKEN: *** 2025-12-04T08:44:03.7909681Z ##[endgroup] 2025-12-04T08:44:03.8882349Z + python3 -m pip install requests==2.27.1 pyyaml==6.0.2 2025-12-04T08:44:04.0796213Z Defaulting to user installation because normal site-packages is not writeable 2025-12-04T08:44:04.2184442Z Collecting requests==2.27.1 2025-12-04T08:44:04.2324215Z Downloading requests-2.27.1-py2.py3-none-any.whl (63 kB) 2025-12-04T08:44:04.3704337Z Collecting pyyaml==6.0.2 2025-12-04T08:44:04.3741737Z Downloading PyYAML-6.0.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (737 kB) 2025-12-04T08:44:04.6738342Z Collecting charset-normalizer~=2.0.0 2025-12-04T08:44:04.6773576Z Downloading charset_normalizer-2.0.12-py3-none-any.whl (39 kB) 2025-12-04T08:44:04.6831995Z Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/lib/python3.9/site-packages (from requests==2.27.1) (1.25.10) 2025-12-04T08:44:04.7203243Z Collecting certifi>=2017.4.17 2025-12-04T08:44:04.7249431Z Downloading certifi-2025.11.12-py3-none-any.whl (159 kB) 2025-12-04T08:44:04.7313826Z Requirement already satisfied: idna<4,>=2.5 in /usr/lib/python3.9/site-packages (from requests==2.27.1) (2.10) 2025-12-04T08:44:04.7955345Z Installing collected packages: charset-normalizer, certifi, requests, pyyaml 2025-12-04T08:44:04.9076535Z Successfully installed certifi-2025.11.12 charset-normalizer-2.0.12 pyyaml-6.0.2 requests-2.27.1 2025-12-04T08:44:05.8651292Z Command completed after 1 attempt(s). 2025-12-04T08:44:05.8702204Z ##[group]Run set -x 2025-12-04T08:44:05.8702416Z set -x 2025-12-04T08:44:05.8702593Z  2025-12-04T08:44:05.8702878Z # Use relative path here as this could be checked out anywhere, not necessarily 2025-12-04T08:44:05.8703371Z # in runner workspace 2025-12-04T08:44:05.8703670Z python3 "${GITHUB_ACTION_PATH}/../../scripts/parse_ref.py" 2025-12-04T08:44:05.8709233Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T08:44:05.8709505Z env: 2025-12-04T08:44:05.8709678Z GIT_DEFAULT_BRANCH: main 2025-12-04T08:44:05.8709884Z HAS_NVIDIA_GPU: false 2025-12-04T08:44:05.8710081Z ##[endgroup] 2025-12-04T08:44:05.8732939Z + python3 /home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/filter-test-configs/../../scripts/parse_ref.py 2025-12-04T08:44:05.8887736Z Setting output branch=main 2025-12-04T08:44:05.8942108Z ##[group]Run echo "Workflow: ${GITHUB_WORKFLOW}" 2025-12-04T08:44:05.8942419Z echo "Workflow: ${GITHUB_WORKFLOW}" 2025-12-04T08:44:05.8942676Z echo "Job name: ${JOB_NAME}" 2025-12-04T08:44:05.8942897Z  2025-12-04T08:44:05.8943164Z # Use relative path here as this could be checked out anywhere, not necessarily 2025-12-04T08:44:05.8943512Z # in runner workspace 2025-12-04T08:44:05.8943996Z python3 "${GITHUB_ACTION_PATH}/../../scripts/filter_test_configs.py" \ 2025-12-04T08:44:05.8944354Z  --workflow "${GITHUB_WORKFLOW}" \ 2025-12-04T08:44:05.8944592Z  --job-name "${JOB_NAME}" \ 2025-12-04T08:44:05.8946434Z  --test-matrix "{"include": [{"config": "cpu_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.2xlarge.amx"}, {"config": "cpu_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.2xlarge.amx"}, {"config": "dynamic_cpu_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.2xlarge.amx"}, {"config": "dynamic_cpu_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.2xlarge.amx"}, {"config": "dynamic_cpu_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.2xlarge.amx"}, {"config": "dynamic_cpu_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.2xlarge.amx"}, {"config": "dynamic_cpu_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.2xlarge.amx"}, {"config": "inductor_torchbench_cpu_smoketest_perf", "shard": 1, "num_shards": 1, "runner": "linux.24xl.spr-metal"}]}" \ 2025-12-04T08:44:05.8948536Z  --selected-test-configs "" \ 2025-12-04T08:44:05.8948798Z  --pr-number "${PR_NUMBER}" \ 2025-12-04T08:44:05.8949040Z  --tag "${TAG}" \ 2025-12-04T08:44:05.8949271Z  --event-name "${EVENT_NAME}" \ 2025-12-04T08:44:05.8949507Z  --schedule "${SCHEDULE}" \ 2025-12-04T08:44:05.8949744Z  --branch "${HEAD_BRANCH}" 2025-12-04T08:44:05.8954923Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T08:44:05.8955189Z env: 2025-12-04T08:44:05.8955372Z GIT_DEFAULT_BRANCH: main 2025-12-04T08:44:05.8955580Z HAS_NVIDIA_GPU: false 2025-12-04T08:44:05.8955983Z GITHUB_TOKEN: *** 2025-12-04T08:44:05.8956303Z JOB_NAME: inductor-cpu-test / test (dynamic_cpu_inductor_huggingface, 1, 1, linux.2xlarge.amx) 2025-12-04T08:44:05.8956663Z PR_NUMBER: 2025-12-04T08:44:05.8956866Z TAG: 2025-12-04T08:44:05.8957027Z EVENT_NAME: push 2025-12-04T08:44:05.8957213Z SCHEDULE: 2025-12-04T08:44:05.8957385Z HEAD_BRANCH: main 2025-12-04T08:44:05.8957558Z ##[endgroup] 2025-12-04T08:44:05.8981652Z Workflow: inductor 2025-12-04T08:44:05.8983440Z Job name: inductor-cpu-test / test (dynamic_cpu_inductor_huggingface, 1, 1, linux.2xlarge.amx) 2025-12-04T08:44:06.1098604Z Setting output keep-going=True 2025-12-04T08:44:06.1098933Z Setting output ci-verbose-test-logs=False 2025-12-04T08:44:06.1099206Z Setting output ci-test-showlocals=False 2025-12-04T08:44:06.1099493Z Setting output ci-no-test-timeout=False 2025-12-04T08:44:06.1099722Z Setting output ci-no-td=False 2025-12-04T08:44:06.1099957Z Setting output ci-td-distributed=False 2025-12-04T08:44:06.1100200Z Setting output is-unstable=False 2025-12-04T08:44:06.1100425Z Setting output reenabled-issues= 2025-12-04T08:44:06.1102582Z Setting output test-matrix={"include": [{"config": "cpu_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.2xlarge.amx"}, {"config": "cpu_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.2xlarge.amx"}, {"config": "dynamic_cpu_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.2xlarge.amx"}, {"config": "dynamic_cpu_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.2xlarge.amx"}, {"config": "dynamic_cpu_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.2xlarge.amx"}, {"config": "dynamic_cpu_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.2xlarge.amx"}, {"config": "dynamic_cpu_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.2xlarge.amx"}, {"config": "inductor_torchbench_cpu_smoketest_perf", "shard": 1, "num_shards": 1, "runner": "linux.24xl.spr-metal"}]} 2025-12-04T08:44:06.1104709Z Setting output is-test-matrix-empty=False 2025-12-04T08:44:06.1230247Z ##[group]Run echo "Filtered matrix:" 2025-12-04T08:44:06.1230534Z echo "Filtered matrix:" 2025-12-04T08:44:06.1232389Z echo "{"include": [{"config": "cpu_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.2xlarge.amx"}, {"config": "cpu_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.2xlarge.amx"}, {"config": "dynamic_cpu_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.2xlarge.amx"}, {"config": "dynamic_cpu_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.2xlarge.amx"}, {"config": "dynamic_cpu_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.2xlarge.amx"}, {"config": "dynamic_cpu_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.2xlarge.amx"}, {"config": "dynamic_cpu_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.2xlarge.amx"}, {"config": "inductor_torchbench_cpu_smoketest_perf", "shard": 1, "num_shards": 1, "runner": "linux.24xl.spr-metal"}]}" 2025-12-04T08:44:06.1234708Z  2025-12-04T08:44:06.1234885Z echo 2025-12-04T08:44:06.1235107Z echo "Is the current job unstable? False" 2025-12-04T08:44:06.1235356Z  2025-12-04T08:44:06.1235524Z echo 2025-12-04T08:44:06.1235726Z echo "Is keep-going label set? True" 2025-12-04T08:44:06.1235954Z  2025-12-04T08:44:06.1236116Z echo 2025-12-04T08:44:06.1236301Z echo "Reenabled issues? " 2025-12-04T08:44:06.1241719Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T08:44:06.1242004Z env: 2025-12-04T08:44:06.1242190Z GIT_DEFAULT_BRANCH: main 2025-12-04T08:44:06.1242391Z HAS_NVIDIA_GPU: false 2025-12-04T08:44:06.1242601Z ##[endgroup] 2025-12-04T08:44:06.1268552Z Filtered matrix: 2025-12-04T08:44:06.1270394Z {include: [{config: cpu_inductor_torchbench, shard: 1, num_shards: 2, runner: linux.2xlarge.amx}, {config: cpu_inductor_torchbench, shard: 2, num_shards: 2, runner: linux.2xlarge.amx}, {config: dynamic_cpu_inductor_huggingface, shard: 1, num_shards: 1, runner: linux.2xlarge.amx}, {config: dynamic_cpu_inductor_timm, shard: 1, num_shards: 2, runner: linux.2xlarge.amx}, {config: dynamic_cpu_inductor_timm, shard: 2, num_shards: 2, runner: linux.2xlarge.amx}, {config: dynamic_cpu_inductor_torchbench, shard: 1, num_shards: 2, runner: linux.2xlarge.amx}, {config: dynamic_cpu_inductor_torchbench, shard: 2, num_shards: 2, runner: linux.2xlarge.amx}, {config: inductor_torchbench_cpu_smoketest_perf, shard: 1, num_shards: 1, runner: linux.24xl.spr-metal}]} 2025-12-04T08:44:06.1272103Z 2025-12-04T08:44:06.1272198Z Is the current job unstable? False 2025-12-04T08:44:06.1272360Z 2025-12-04T08:44:06.1272448Z Is keep-going label set? True 2025-12-04T08:44:06.1272597Z 2025-12-04T08:44:06.1272684Z Reenabled issues? 2025-12-04T08:44:06.1302619Z ##[group]Run echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2025-12-04T08:44:06.1303027Z echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2025-12-04T08:44:06.1307494Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T08:44:06.1307810Z env: 2025-12-04T08:44:06.1308024Z GIT_DEFAULT_BRANCH: main 2025-12-04T08:44:06.1308223Z HAS_NVIDIA_GPU: false 2025-12-04T08:44:06.1308416Z JOB_TIMEOUT: 240 2025-12-04T08:44:06.1308595Z ##[endgroup] 2025-12-04T08:44:06.1358482Z ##[group]Run env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2025-12-04T08:44:06.1358867Z env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2025-12-04T08:44:06.1359186Z env | grep '^CI' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2025-12-04T08:44:06.1363438Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T08:44:06.1363708Z env: 2025-12-04T08:44:06.1363896Z GIT_DEFAULT_BRANCH: main 2025-12-04T08:44:06.1364098Z HAS_NVIDIA_GPU: false 2025-12-04T08:44:06.1364279Z ##[endgroup] 2025-12-04T08:44:06.1452167Z ##[group]Run set -x 2025-12-04T08:44:06.1452431Z set -x 2025-12-04T08:44:06.1452602Z  2025-12-04T08:44:06.1452802Z if [[ $TEST_CONFIG == 'multigpu' ]]; then 2025-12-04T08:44:06.1453073Z  TEST_COMMAND=.ci/pytorch/multigpu-test.sh 2025-12-04T08:44:06.1453353Z elif [[ $BUILD_ENVIRONMENT == *onnx* ]]; then 2025-12-04T08:44:06.1453609Z  TEST_COMMAND=.ci/onnx/test.sh 2025-12-04T08:44:06.1453817Z else 2025-12-04T08:44:06.1454010Z  TEST_COMMAND=.ci/pytorch/test.sh 2025-12-04T08:44:06.1454229Z fi 2025-12-04T08:44:06.1454375Z  2025-12-04T08:44:06.1454565Z # Leaving 1GB for the runner and other things 2025-12-04T08:44:06.1454962Z TOTAL_AVAILABLE_MEMORY_IN_GB=$(awk '/MemTotal/ { printf "%.3f \n", $2/1024/1024 - 1 }' /proc/meminfo) 2025-12-04T08:44:06.1457582Z # https://docs.docker.com/engine/containers/resource_constraints/#--memory-swap-details, the 3GB swap 2025-12-04T08:44:06.1458136Z # comes from https://github.com/pytorch/test-infra/pull/6058 2025-12-04T08:44:06.1458564Z TOTAL_MEMORY_WITH_SWAP=$(("${TOTAL_AVAILABLE_MEMORY_IN_GB%.*}" + 3)) 2025-12-04T08:44:06.1458893Z  2025-12-04T08:44:06.1459138Z if [[ ${BUILD_ENVIRONMENT} == *"s390x"* ]]; then 2025-12-04T08:44:06.1459427Z  SHM_OPTS= 2025-12-04T08:44:06.1459653Z  JENKINS_USER= 2025-12-04T08:44:06.1459934Z  # ensure that docker container cleanly exits in 12 hours 2025-12-04T08:44:06.1460298Z  # if for some reason cleanup action doesn't stop container 2025-12-04T08:44:06.1460596Z  # when job is cancelled 2025-12-04T08:44:06.1460855Z  DOCKER_SHELL_CMD="sleep 12h" 2025-12-04T08:44:06.1461120Z  USED_IMAGE="${DOCKER_IMAGE_S390X}" 2025-12-04T08:44:06.1461406Z else 2025-12-04T08:44:06.1461627Z  SHM_OPTS="--shm-size=${SHM_SIZE}" 2025-12-04T08:44:06.1461904Z  JENKINS_USER="--user jenkins" 2025-12-04T08:44:06.1462172Z  DOCKER_SHELL_CMD= 2025-12-04T08:44:06.1462390Z  USED_IMAGE="${DOCKER_IMAGE}" 2025-12-04T08:44:06.1462585Z fi 2025-12-04T08:44:06.1462740Z  2025-12-04T08:44:06.1462977Z # detached container should get cleaned up by teardown_ec2_linux 2025-12-04T08:44:06.1463331Z # TODO: Stop building test binaries as part of the build phase 2025-12-04T08:44:06.1463717Z # Used for GPU_FLAG, SHM_OPTS, JENKINS_USER and DOCKER_SHELL_CMD since that doesn't play nice 2025-12-04T08:44:06.1464069Z # shellcheck disable=SC2086,SC2090 2025-12-04T08:44:06.1464307Z container_name=$(docker run \ 2025-12-04T08:44:06.1464519Z  ${GPU_FLAG:-} \ 2025-12-04T08:44:06.1464741Z  ${SCCACHE_SERVER_PORT_DOCKER_FLAG:-} \ 2025-12-04T08:44:06.1464989Z  -e BUILD_ENVIRONMENT \ 2025-12-04T08:44:06.1465204Z  -e PR_NUMBER \ 2025-12-04T08:44:06.1465402Z  -e GITHUB_ACTIONS \ 2025-12-04T08:44:06.1465613Z  -e GITHUB_REPOSITORY \ 2025-12-04T08:44:06.1465828Z  -e GITHUB_WORKFLOW \ 2025-12-04T08:44:06.1466028Z  -e GITHUB_JOB \ 2025-12-04T08:44:06.1466225Z  -e GITHUB_RUN_ID \ 2025-12-04T08:44:06.1466429Z  -e GITHUB_RUN_NUMBER \ 2025-12-04T08:44:06.1466634Z  -e GITHUB_RUN_ATTEMPT \ 2025-12-04T08:44:06.1466847Z  -e JOB_ID \ 2025-12-04T08:44:06.1467041Z  -e JOB_NAME \ 2025-12-04T08:44:06.1467233Z  -e BASE_SHA \ 2025-12-04T08:44:06.1467413Z  -e BRANCH \ 2025-12-04T08:44:06.1467593Z  -e SHA1 \ 2025-12-04T08:44:06.1467781Z  -e AWS_DEFAULT_REGION \ 2025-12-04T08:44:06.1467985Z  -e IN_WHEEL_TEST \ 2025-12-04T08:44:06.1468185Z  -e SHARD_NUMBER \ 2025-12-04T08:44:06.1468384Z  -e TEST_CONFIG \ 2025-12-04T08:44:06.1468577Z  -e NUM_TEST_SHARDS \ 2025-12-04T08:44:06.1468934Z  -e REENABLED_ISSUES \ 2025-12-04T08:44:06.1469167Z  -e CONTINUE_THROUGH_ERROR \ 2025-12-04T08:44:06.1469400Z  -e VERBOSE_TEST_LOGS \ 2025-12-04T08:44:06.1469616Z  -e TEST_SHOWLOCALS \ 2025-12-04T08:44:06.1469826Z  -e NO_TEST_TIMEOUT \ 2025-12-04T08:44:06.1470030Z  -e NO_TD \ 2025-12-04T08:44:06.1470220Z  -e TD_DISTRIBUTED \ 2025-12-04T08:44:06.1470431Z  -e PR_LABELS \ 2025-12-04T08:44:06.1470686Z  -e MAX_JOBS="$(nproc --ignore=2)" \ 2025-12-04T08:44:06.1470937Z  -e SCCACHE_BUCKET \ 2025-12-04T08:44:06.1471145Z  -e SCCACHE_REGION \ 2025-12-04T08:44:06.1471349Z  -e XLA_CUDA \ 2025-12-04T08:44:06.1471555Z  -e XLA_CLANG_CACHE_S3_BUCKET_NAME \ 2025-12-04T08:44:06.1471815Z  -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK \ 2025-12-04T08:44:06.1472137Z  -e PYTORCH_TEST_RERUN_DISABLED_TESTS \ 2025-12-04T08:44:06.1472400Z  -e SKIP_SCCACHE_INITIALIZATION=1 \ 2025-12-04T08:44:06.1472647Z  -e HUGGING_FACE_HUB_TOKEN \ 2025-12-04T08:44:06.1472894Z  -e VLLM_TEST_HUGGING_FACE_TOKEN \ 2025-12-04T08:44:06.1473146Z  -e SCRIBE_GRAPHQL_ACCESS_TOKEN \ 2025-12-04T08:44:06.1473365Z  -e DASHBOARD_TAG \ 2025-12-04T08:44:06.1473575Z  -e ARTIFACTS_FILE_SUFFIX \ 2025-12-04T08:44:06.1473831Z  --memory="${TOTAL_AVAILABLE_MEMORY_IN_GB%.*}g" \ 2025-12-04T08:44:06.1474112Z  --memory-swap="${TOTAL_MEMORY_WITH_SWAP}g" \ 2025-12-04T08:44:06.1474405Z  --env-file="/tmp/github_env_${GITHUB_RUN_ID}" \ 2025-12-04T08:44:06.1474690Z  --security-opt seccomp=unconfined \ 2025-12-04T08:44:06.1474932Z  --cap-add=SYS_PTRACE \ 2025-12-04T08:44:06.1475153Z  --ipc=host \ 2025-12-04T08:44:06.1475347Z  ${SHM_OPTS} \ 2025-12-04T08:44:06.1475540Z  --tty \ 2025-12-04T08:44:06.1475713Z  --detach \ 2025-12-04T08:44:06.1475918Z  --name="${container_name}" \ 2025-12-04T08:44:06.1476147Z  ${JENKINS_USER} \ 2025-12-04T08:44:06.1476396Z  -v "${GITHUB_WORKSPACE}:/var/lib/jenkins/workspace" \ 2025-12-04T08:44:06.1476676Z  -w /var/lib/jenkins/workspace \ 2025-12-04T08:44:06.1476906Z  "${USED_IMAGE}" \ 2025-12-04T08:44:06.1477100Z  ${DOCKER_SHELL_CMD} 2025-12-04T08:44:06.1477296Z ) 2025-12-04T08:44:06.1477540Z echo "DOCKER_CONTAINER_ID=${container_name}" >> "${GITHUB_ENV}" 2025-12-04T08:44:06.1477824Z  2025-12-04T08:44:06.1478020Z if [[ ${BUILD_ENVIRONMENT} == *"s390x"* ]]; then 2025-12-04T08:44:06.1478658Z  docker exec -t "${container_name}" sh -c "python3 -m pip install -r .ci/docker/requirements-ci.txt" 2025-12-04T08:44:06.1479018Z fi 2025-12-04T08:44:06.1479182Z  2025-12-04T08:44:06.1479534Z docker exec -t "${container_name}" sh -c "python3 -m pip install $(echo dist/*.whl)[opt-einsum] && ${TEST_COMMAND}" 2025-12-04T08:44:06.1484526Z shell: /usr/bin/bash -e {0} 2025-12-04T08:44:06.1484724Z env: 2025-12-04T08:44:06.1484902Z GIT_DEFAULT_BRANCH: main 2025-12-04T08:44:06.1485107Z HAS_NVIDIA_GPU: false 2025-12-04T08:44:06.1485347Z BUILD_ENVIRONMENT: linux-jammy-py3.10-gcc11-build 2025-12-04T08:44:06.1485594Z PR_NUMBER: 2025-12-04T08:44:06.1485787Z GITHUB_REPOSITORY: pytorch/pytorch 2025-12-04T08:44:06.1486014Z GITHUB_WORKFLOW: inductor 2025-12-04T08:44:06.1486197Z GITHUB_JOB: test 2025-12-04T08:44:06.1486376Z GITHUB_RUN_ID: 19921726389 2025-12-04T08:44:06.1486577Z GITHUB_RUN_NUMBER: 168680 2025-12-04T08:44:06.1486762Z GITHUB_RUN_ATTEMPT: 1 2025-12-04T08:44:06.1486948Z JOB_ID: 57113856426 2025-12-04T08:44:06.1487277Z JOB_NAME: inductor-cpu-test / test (dynamic_cpu_inductor_huggingface, 1, 1, linux.2xlarge.amx) 2025-12-04T08:44:06.1487625Z BRANCH: main 2025-12-04T08:44:06.1487838Z SHA1: ffd9b0fb4355e97af82fc42cf185c3ffa0fc0a32 2025-12-04T08:44:06.1488211Z BASE_SHA: ffd9b0fb4355e97af82fc42cf185c3ffa0fc0a32 2025-12-04T08:44:06.1488490Z TEST_CONFIG: dynamic_cpu_inductor_huggingface 2025-12-04T08:44:06.1488726Z SHARD_NUMBER: 1 2025-12-04T08:44:06.1488910Z NUM_TEST_SHARDS: 1 2025-12-04T08:44:06.1489092Z EXTRA_FLAGS: 2025-12-04T08:44:06.1489273Z OP_BENCHMARK_TESTS: 2025-12-04T08:44:06.1489461Z REENABLED_ISSUES: 2025-12-04T08:44:06.1489652Z CONTINUE_THROUGH_ERROR: True 2025-12-04T08:44:06.1489854Z VERBOSE_TEST_LOGS: False 2025-12-04T08:44:06.1490056Z TEST_SHOWLOCALS: False 2025-12-04T08:44:06.1490257Z NO_TEST_TIMEOUT: False 2025-12-04T08:44:06.1490434Z NO_TD: False 2025-12-04T08:44:06.1490609Z TD_DISTRIBUTED: False 2025-12-04T08:44:06.1490841Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2025-12-04T08:44:06.1491104Z SCCACHE_REGION: us-east-1 2025-12-04T08:44:06.1491294Z SHM_SIZE: 1g 2025-12-04T08:44:06.1491906Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-f0cd68561080d537ef3d3d6f81b25a6416ad600a 2025-12-04T08:44:06.1492878Z DOCKER_IMAGE_S390X: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-f0cd68561080d537ef3d3d6f81b25a6416ad600a 2025-12-04T08:44:06.1493463Z XLA_CUDA: 2025-12-04T08:44:06.1493722Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2025-12-04T08:44:06.1494048Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK: 0 2025-12-04T08:44:06.1494286Z PYTORCH_TEST_RERUN_DISABLED_TESTS: 0 2025-12-04T08:44:06.1494496Z DASHBOARD_TAG: 2025-12-04T08:44:06.1494879Z VLLM_TEST_HUGGING_FACE_TOKEN: *** 2025-12-04T08:44:06.1495181Z HUGGING_FACE_HUB_TOKEN: *** 2025-12-04T08:44:06.1495478Z SCRIBE_GRAPHQL_ACCESS_TOKEN: *** 2025-12-04T08:44:06.1495844Z ARTIFACTS_FILE_SUFFIX: test-dynamic_cpu_inductor_huggingface-1-1-linux.2xlarge.amx_57113856426 2025-12-04T08:44:06.1496207Z ##[endgroup] 2025-12-04T08:44:06.1519408Z + [[ dynamic_cpu_inductor_huggingface == \m\u\l\t\i\g\p\u ]] 2025-12-04T08:44:06.1519769Z + [[ linux-jammy-py3.10-gcc11-build == *onnx* ]] 2025-12-04T08:44:06.1520051Z + TEST_COMMAND=.ci/pytorch/test.sh 2025-12-04T08:44:06.1522545Z ++ awk '/MemTotal/ { printf "%.3f \n", $2/1024/1024 - 1 }' /proc/meminfo 2025-12-04T08:44:06.1589349Z + TOTAL_AVAILABLE_MEMORY_IN_GB='29.809 ' 2025-12-04T08:44:06.1589710Z + TOTAL_MEMORY_WITH_SWAP=32 2025-12-04T08:44:06.1590086Z + [[ linux-jammy-py3.10-gcc11-build == *\s\3\9\0\x* ]] 2025-12-04T08:44:06.1590365Z + SHM_OPTS=--shm-size=1g 2025-12-04T08:44:06.1590580Z + JENKINS_USER='--user jenkins' 2025-12-04T08:44:06.1590794Z + DOCKER_SHELL_CMD= 2025-12-04T08:44:06.1591378Z + USED_IMAGE=308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-f0cd68561080d537ef3d3d6f81b25a6416ad600a 2025-12-04T08:44:06.1596659Z +++ nproc --ignore=2 2025-12-04T08:44:06.1621120Z ++ docker run -e BUILD_ENVIRONMENT -e PR_NUMBER -e GITHUB_ACTIONS -e GITHUB_REPOSITORY -e GITHUB_WORKFLOW -e GITHUB_JOB -e GITHUB_RUN_ID -e GITHUB_RUN_NUMBER -e GITHUB_RUN_ATTEMPT -e JOB_ID -e JOB_NAME -e BASE_SHA -e BRANCH -e SHA1 -e AWS_DEFAULT_REGION -e IN_WHEEL_TEST -e SHARD_NUMBER -e TEST_CONFIG -e NUM_TEST_SHARDS -e REENABLED_ISSUES -e CONTINUE_THROUGH_ERROR -e VERBOSE_TEST_LOGS -e TEST_SHOWLOCALS -e NO_TEST_TIMEOUT -e NO_TD -e TD_DISTRIBUTED -e PR_LABELS -e MAX_JOBS=6 -e SCCACHE_BUCKET -e SCCACHE_REGION -e XLA_CUDA -e XLA_CLANG_CACHE_S3_BUCKET_NAME -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK -e PYTORCH_TEST_RERUN_DISABLED_TESTS -e SKIP_SCCACHE_INITIALIZATION=1 -e HUGGING_FACE_HUB_TOKEN -e VLLM_TEST_HUGGING_FACE_TOKEN -e SCRIBE_GRAPHQL_ACCESS_TOKEN -e DASHBOARD_TAG -e ARTIFACTS_FILE_SUFFIX --memory=29g --memory-swap=32g --env-file=/tmp/github_env_19921726389 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --ipc=host --shm-size=1g --tty --detach --name= --user jenkins -v /home/ec2-user/actions-runner/_work/pytorch/pytorch:/var/lib/jenkins/workspace -w /var/lib/jenkins/workspace 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-f0cd68561080d537ef3d3d6f81b25a6416ad600a 2025-12-04T08:44:20.6101753Z + container_name=f7efe3f2eca996d8a9efcd5effd738cd22ce08409dea3a9ea34cb5639faf9c86 2025-12-04T08:44:20.6102323Z + echo DOCKER_CONTAINER_ID=f7efe3f2eca996d8a9efcd5effd738cd22ce08409dea3a9ea34cb5639faf9c86 2025-12-04T08:44:20.6102743Z + [[ linux-jammy-py3.10-gcc11-build == *\s\3\9\0\x* ]] 2025-12-04T08:44:20.6107522Z ++ echo dist/torch-2.10.0a0+gitffd9b0f-cp310-cp310-linux_x86_64.whl 2025-12-04T08:44:20.6109546Z + docker exec -t f7efe3f2eca996d8a9efcd5effd738cd22ce08409dea3a9ea34cb5639faf9c86 sh -c 'python3 -m pip install dist/torch-2.10.0a0+gitffd9b0f-cp310-cp310-linux_x86_64.whl[opt-einsum] && .ci/pytorch/test.sh' 2025-12-04T08:44:21.0609487Z Processing ./dist/torch-2.10.0a0+gitffd9b0f-cp310-cp310-linux_x86_64.whl (from torch==2.10.0a0+gitffd9b0f) 2025-12-04T08:44:21.2821506Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.10.0a0+gitffd9b0f->torch==2.10.0a0+gitffd9b0f) (3.18.0) 2025-12-04T08:44:21.2822467Z Requirement already satisfied: typing-extensions>=4.10.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.10.0a0+gitffd9b0f->torch==2.10.0a0+gitffd9b0f) (4.15.0) 2025-12-04T08:44:21.2824945Z Requirement already satisfied: sympy>=1.13.3 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.10.0a0+gitffd9b0f->torch==2.10.0a0+gitffd9b0f) (1.13.3) 2025-12-04T08:44:21.2829638Z Requirement already satisfied: networkx>=2.5.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.10.0a0+gitffd9b0f->torch==2.10.0a0+gitffd9b0f) (2.8.8) 2025-12-04T08:44:21.2834502Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.10.0a0+gitffd9b0f->torch==2.10.0a0+gitffd9b0f) (3.1.6) 2025-12-04T08:44:21.2835437Z Requirement already satisfied: fsspec>=0.8.5 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.10.0a0+gitffd9b0f->torch==2.10.0a0+gitffd9b0f) (2025.10.0) 2025-12-04T08:44:21.2854552Z Requirement already satisfied: opt-einsum>=3.3 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.10.0a0+gitffd9b0f->torch==2.10.0a0+gitffd9b0f) (3.3.0) 2025-12-04T08:44:21.3135023Z Requirement already satisfied: numpy>=1.7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from opt-einsum>=3.3->torch==2.10.0a0+gitffd9b0f->torch==2.10.0a0+gitffd9b0f) (1.22.4) 2025-12-04T08:44:21.3156179Z Requirement already satisfied: mpmath<1.4,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy>=1.13.3->torch==2.10.0a0+gitffd9b0f->torch==2.10.0a0+gitffd9b0f) (1.3.0) 2025-12-04T08:44:21.3205102Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch==2.10.0a0+gitffd9b0f->torch==2.10.0a0+gitffd9b0f) (3.0.3) 2025-12-04T08:44:22.1369608Z Installing collected packages: torch 2025-12-04T08:44:29.9665758Z ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. 2025-12-04T08:44:29.9666638Z dall-e 0.1 requires torchvision, which is not installed. 2025-12-04T08:44:29.9667023Z effdet 0.4.1 requires torchvision, which is not installed. 2025-12-04T08:44:29.9676188Z python-doctr 1.0.0 requires torchvision>=0.15.0, which is not installed. 2025-12-04T08:44:29.9676842Z pytorch-labs-segment-anything-fast 0.2 requires torchao, which is not installed. 2025-12-04T08:44:29.9677352Z pytorch-labs-segment-anything-fast 0.2 requires torchvision>=0.17.0.dev20231026, which is not installed. 2025-12-04T08:44:29.9677877Z timm 1.0.22 requires torchvision, which is not installed. 2025-12-04T08:44:29.9678329Z Successfully installed torch-2.10.0a0+gitffd9b0f 2025-12-04T08:44:30.0731556Z + export TERM=vt100 2025-12-04T08:44:30.0731812Z + TERM=vt100 2025-12-04T08:44:30.0737197Z ++ dirname .ci/pytorch/test.sh 2025-12-04T08:44:30.0758053Z + source .ci/pytorch/common.sh 2025-12-04T08:44:30.0769177Z +++ dirname .ci/pytorch/common.sh 2025-12-04T08:44:30.0775372Z ++ source .ci/pytorch/common_utils.sh 2025-12-04T08:44:30.0779566Z +++ declare -f -t trap_add 2025-12-04T08:44:30.0787689Z ++ set -ex -o pipefail 2025-12-04T08:44:30.0788051Z ++ [[ linux-jammy-py3.10-gcc11-build == *rocm* ]] 2025-12-04T08:44:30.0788409Z ++ BUILD_TEST_LIBTORCH=0 2025-12-04T08:44:30.0792088Z ++ dirname .ci/pytorch/test.sh 2025-12-04T08:44:30.0798419Z + source .ci/pytorch/common-build.sh 2025-12-04T08:44:30.0804338Z ++ [[ linux-jammy-py3.10-gcc11-build != *win-* ]] 2025-12-04T08:44:30.0806612Z ++++ dirname .ci/pytorch/common-build.sh 2025-12-04T08:44:30.0818516Z +++ cd .ci/pytorch 2025-12-04T08:44:30.0818930Z +++ pwd -P 2025-12-04T08:44:30.0820987Z ++ script_dir=/var/lib/jenkins/workspace/.ci/pytorch 2025-12-04T08:44:30.0821743Z ++ [[ linux-jammy-py3.10-gcc11-build == *-pch* ]] 2025-12-04T08:44:30.0821963Z ++ which sccache 2025-12-04T08:44:30.0841286Z ++ [[ -z ossci-compiler-cache-circleci-v2 ]] 2025-12-04T08:44:30.0844456Z ++ sccache --stop-server 2025-12-04T08:44:30.1006538Z ++ true 2025-12-04T08:44:30.1006777Z ++ rm -f /var/lib/jenkins/sccache_error.log 2025-12-04T08:44:30.1033212Z ++ trap_add sccache_epilogue EXIT 2025-12-04T08:44:30.1033535Z ++ trap_add_cmd=sccache_epilogue 2025-12-04T08:44:30.1033745Z ++ shift 2025-12-04T08:44:30.1033933Z ++ for trap_add_name in "$@" 2025-12-04T08:44:30.1036926Z ++++ trap -p EXIT 2025-12-04T08:44:30.1038894Z +++ eval 'extract_trap_cmd ' 2025-12-04T08:44:30.1039335Z ++++ extract_trap_cmd 2025-12-04T08:44:30.1039638Z ++++ printf '%s\n' '' 2025-12-04T08:44:30.1048033Z +++ printf '%s\n' sccache_epilogue 2025-12-04T08:44:30.1048631Z ++ trap -- ' 2025-12-04T08:44:30.1048862Z sccache_epilogue' EXIT 2025-12-04T08:44:30.1049059Z ++ [[ -n 1 ]] 2025-12-04T08:44:30.1049348Z ++ echo 'Skipping sccache server initialization, setting environment variables' 2025-12-04T08:44:30.1049796Z Skipping sccache server initialization, setting environment variables 2025-12-04T08:44:30.1050167Z ++ export SCCACHE_IDLE_TIMEOUT=0 2025-12-04T08:44:30.1050391Z ++ SCCACHE_IDLE_TIMEOUT=0 2025-12-04T08:44:30.1050665Z ++ export SCCACHE_ERROR_LOG=/var/lib/jenkins/sccache_error.log 2025-12-04T08:44:30.1050989Z ++ SCCACHE_ERROR_LOG=/var/lib/jenkins/sccache_error.log 2025-12-04T08:44:30.1061215Z ++ export RUST_LOG=sccache::server=error 2025-12-04T08:44:30.1061521Z ++ RUST_LOG=sccache::server=error 2025-12-04T08:44:30.1061749Z ++ sccache --zero-stats 2025-12-04T08:44:30.3272840Z Statistics zeroed. 2025-12-04T08:44:30.3277599Z ++ which ccache 2025-12-04T08:44:30.3298097Z + [[ linux-jammy-py3.10-gcc11-build != *rocm* ]] 2025-12-04T08:44:30.3298444Z + [[ linux-jammy-py3.10-gcc11-build != *s390x* ]] 2025-12-04T08:44:30.3298710Z + [[ -d /var/lib/jenkins/workspace ]] 2025-12-04T08:44:30.3300939Z ++ stat -c %u /var/lib/jenkins/workspace 2025-12-04T08:44:30.3362221Z + WORKSPACE_ORIGINAL_OWNER_ID=1000 2025-12-04T08:44:30.3362513Z + trap_add cleanup_workspace EXIT 2025-12-04T08:44:30.3362773Z + trap_add_cmd=cleanup_workspace 2025-12-04T08:44:30.3362987Z + shift 2025-12-04T08:44:30.3363207Z + for trap_add_name in "$@" 2025-12-04T08:44:30.3372493Z +++ trap -p EXIT 2025-12-04T08:44:30.3372776Z ++ eval 'extract_trap_cmd trap -- '\'' 2025-12-04T08:44:30.3373039Z sccache_epilogue'\'' EXIT' 2025-12-04T08:44:30.3373255Z +++ extract_trap_cmd trap -- ' 2025-12-04T08:44:30.3373476Z sccache_epilogue' EXIT 2025-12-04T08:44:30.3373674Z +++ printf '%s\n' ' 2025-12-04T08:44:30.3373853Z sccache_epilogue' 2025-12-04T08:44:30.3374060Z ++ printf '%s\n' cleanup_workspace 2025-12-04T08:44:30.3380393Z + trap -- ' 2025-12-04T08:44:30.3380630Z sccache_epilogue 2025-12-04T08:44:30.3381142Z cleanup_workspace' EXIT 2025-12-04T08:44:30.3381434Z + sudo chown -R jenkins /var/lib/jenkins/workspace 2025-12-04T08:44:30.8318735Z + git config --global --add safe.directory /var/lib/jenkins/workspace 2025-12-04T08:44:30.8509364Z + [[ linux-jammy-py3.10-gcc11-build == *cuda* ]] 2025-12-04T08:44:30.8510168Z + echo 'Environment variables:' 2025-12-04T08:44:30.8510439Z Environment variables: 2025-12-04T08:44:30.8510659Z + env 2025-12-04T08:44:30.8526751Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2025-12-04T08:44:30.8527089Z CONTINUE_THROUGH_ERROR=True 2025-12-04T08:44:30.8527359Z BUILD_ENVIRONMENT=linux-jammy-py3.10-gcc11-build 2025-12-04T08:44:30.8528150Z VLLM_TEST_HUGGING_FACE_TOKEN=*** 2025-12-04T08:44:30.8528379Z HOSTNAME=f7efe3f2eca9 2025-12-04T08:44:30.8528846Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_524bc399-029d-4ed7-ae9d-e1110b8a36d1 2025-12-04T08:44:30.8529270Z GITHUB_ACTION=__run_3 2025-12-04T08:44:30.8529512Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2025-12-04T08:44:30.8529761Z GITHUB_RUN_NUMBER=168680 2025-12-04T08:44:30.8529983Z TEST_CONFIG=dynamic_cpu_inductor_huggingface 2025-12-04T08:44:30.8530244Z GITHUB_REPOSITORY_OWNER_ID=21003710 2025-12-04T08:44:30.8530669Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2025-12-04T08:44:30.8530898Z SCCACHE_IDLE_TIMEOUT=0 2025-12-04T08:44:30.8531198Z SCRIBE_GRAPHQL_ACCESS_TOKEN=*** 2025-12-04T08:44:30.8531435Z GITHUB_TRIGGERING_ACTOR=pytorchmergebot 2025-12-04T08:44:30.8531668Z GITHUB_REF_TYPE=branch 2025-12-04T08:44:30.8531895Z BASE_SHA=ffd9b0fb4355e97af82fc42cf185c3ffa0fc0a32 2025-12-04T08:44:30.8532150Z XLA_CUDA= 2025-12-04T08:44:30.8532336Z NCCL_LIB_DIR=/usr/local/cuda/lib64/ 2025-12-04T08:44:30.8532630Z HUGGING_FACE_HUB_TOKEN=*** 2025-12-04T08:44:30.8532942Z *** 2025-12-04T08:44:30.8533121Z GITHUB_REPOSITORY_ID=65600975 2025-12-04T08:44:30.8533335Z GITHUB_ACTIONS=true 2025-12-04T08:44:30.8533557Z SCCACHE_ERROR_LOG=/var/lib/jenkins/sccache_error.log 2025-12-04T08:44:30.8533845Z SHA1=ffd9b0fb4355e97af82fc42cf185c3ffa0fc0a32 2025-12-04T08:44:30.8534125Z GITHUB_SHA=ffd9b0fb4355e97af82fc42cf185c3ffa0fc0a32 2025-12-04T08:44:30.8534498Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/inductor.yml@refs/heads/main 2025-12-04T08:44:30.8534842Z UCC_HOME=/usr 2025-12-04T08:44:30.8535017Z VERBOSE_TEST_LOGS=False 2025-12-04T08:44:30.8535206Z GITHUB_REF=refs/heads/main 2025-12-04T08:44:30.8535400Z SHARD_NUMBER=1 2025-12-04T08:44:30.8535575Z GITHUB_REF_PROTECTED=true 2025-12-04T08:44:30.8535762Z HOME=/var/lib/jenkins 2025-12-04T08:44:30.8535977Z GITHUB_API_URL=https://api.github.com 2025-12-04T08:44:30.8536220Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2025-12-04T08:44:30.8536431Z UCX_COMMIT= 2025-12-04T08:44:30.8536588Z USE_SYSTEM_NCCL=1 2025-12-04T08:44:30.8536763Z NUM_TEST_SHARDS=1 2025-12-04T08:44:30.8536934Z UCX_HOME=/usr 2025-12-04T08:44:30.8537325Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_524bc399-029d-4ed7-ae9d-e1110b8a36d1 2025-12-04T08:44:30.8537892Z JOB_NAME=inductor-cpu-test / test (dynamic_cpu_inductor_huggingface, 1, 1, linux.2xlarge.amx) 2025-12-04T08:44:30.8538440Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_524bc399-029d-4ed7-ae9d-e1110b8a36d1 2025-12-04T08:44:30.8538965Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2025-12-04T08:44:30.8539306Z GITHUB_EVENT_NAME=push 2025-12-04T08:44:30.8539494Z DASHBOARD_TAG= 2025-12-04T08:44:30.8539672Z GITHUB_RUN_ID=19921726389 2025-12-04T08:44:30.8539863Z INSTALLED_OPENBLAS= 2025-12-04T08:44:30.8540272Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_524bc399-029d-4ed7-ae9d-e1110b8a36d1 2025-12-04T08:44:30.8540745Z GITHUB_ACTOR=pytorchmergebot 2025-12-04T08:44:30.8540940Z PR_NUMBER= 2025-12-04T08:44:30.8541105Z DESIRED_CUDA= 2025-12-04T08:44:30.8541277Z GITHUB_RUN_ATTEMPT=1 2025-12-04T08:44:30.8541464Z ANACONDA_PYTHON_VERSION=3.10 2025-12-04T08:44:30.8541709Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2025-12-04T08:44:30.8541959Z TERM=vt100 2025-12-04T08:44:30.8542119Z INSTALLED_VISION=yes 2025-12-04T08:44:30.8542303Z BRANCH=main 2025-12-04T08:44:30.8542477Z SCCACHE_REGION=us-east-1 2025-12-04T08:44:30.8542682Z OPENSSL_ROOT_DIR=/opt/openssl 2025-12-04T08:44:30.8542899Z BUILD_AOT_INDUCTOR_TEST=yes 2025-12-04T08:44:30.8543215Z CUDA_PATH=/usr/local/cuda 2025-12-04T08:44:30.8543582Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2025-12-04T08:44:30.8543970Z GITHUB_SERVER_URL=https://github.com 2025-12-04T08:44:30.8544196Z UCC_COMMIT= 2025-12-04T08:44:30.8544365Z REENABLED_ISSUES= 2025-12-04T08:44:30.8544533Z DOCS=yes 2025-12-04T08:44:30.8544692Z SHLVL=1 2025-12-04T08:44:30.8544852Z MAX_JOBS=6 2025-12-04T08:44:30.8545012Z GITHUB_ACTOR_ID=97764156 2025-12-04T08:44:30.8545267Z GITHUB_WORKFLOW_SHA=ffd9b0fb4355e97af82fc42cf185c3ffa0fc0a32 2025-12-04T08:44:30.8545540Z GITHUB_REF_NAME=main 2025-12-04T08:44:30.8545812Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2025-12-04T08:44:30.8546111Z GITHUB_JOB=test 2025-12-04T08:44:30.8546290Z NO_TEST_TIMEOUT=False 2025-12-04T08:44:30.8546466Z TD_DISTRIBUTED=False 2025-12-04T08:44:30.8546725Z GITHUB_REPOSITORY=pytorch/pytorch 2025-12-04T08:44:30.8546945Z GITHUB_RETENTION_DAYS=90 2025-12-04T08:44:30.8547136Z OPENSSL_DIR=/opt/openssl 2025-12-04T08:44:30.8547334Z GITHUB_ACTION_REPOSITORY= 2025-12-04T08:44:30.8547854Z PATH=/opt/cache/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2025-12-04T08:44:30.8548370Z GITHUB_BASE_REF= 2025-12-04T08:44:30.8548538Z INSTALLED_ACL= 2025-12-04T08:44:30.8548865Z ARTIFACTS_FILE_SUFFIX=test-dynamic_cpu_inductor_huggingface-1-1-linux.2xlarge.amx_57113856426 2025-12-04T08:44:30.8549216Z CI=true 2025-12-04T08:44:30.8549383Z GITHUB_REPOSITORY_OWNER=pytorch 2025-12-04T08:44:30.8549646Z RUST_LOG=sccache::server=error 2025-12-04T08:44:30.8549847Z JOB_ID=57113856426 2025-12-04T08:44:30.8550012Z GITHUB_HEAD_REF= 2025-12-04T08:44:30.8550186Z GITHUB_ACTION_REF= 2025-12-04T08:44:30.8550405Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2025-12-04T08:44:30.8550654Z TEST_SHOWLOCALS=False 2025-12-04T08:44:30.8550843Z GITHUB_WORKFLOW=inductor 2025-12-04T08:44:30.8551044Z DEBIAN_FRONTEND=noninteractive 2025-12-04T08:44:30.8551463Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_524bc399-029d-4ed7-ae9d-e1110b8a36d1 2025-12-04T08:44:30.8551894Z NO_TD=False 2025-12-04T08:44:30.8552076Z SKIP_SCCACHE_INITIALIZATION=1 2025-12-04T08:44:30.8552306Z NCCL_INCLUDE_DIR=/usr/local/cuda/include/ 2025-12-04T08:44:30.8552523Z _=/usr/bin/env 2025-12-04T08:44:30.8552770Z ++ python -c 'import site; print(site.getsitepackages()[0])' 2025-12-04T08:44:30.8796155Z + TORCH_INSTALL_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch 2025-12-04T08:44:30.8796642Z + TORCH_BIN_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2025-12-04T08:44:30.8797033Z + TORCH_LIB_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib 2025-12-04T08:44:30.8797429Z + TORCH_TEST_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/test 2025-12-04T08:44:30.8797812Z + BUILD_DIR=build 2025-12-04T08:44:30.8798010Z + BUILD_RENAMED_DIR=build_renamed 2025-12-04T08:44:30.8798405Z + BUILD_BIN_DIR=build/bin 2025-12-04T08:44:30.8798617Z + SHARD_NUMBER=1 2025-12-04T08:44:30.8798834Z + NUM_TEST_SHARDS=1 2025-12-04T08:44:30.8799043Z + export TORCH_SERIALIZATION_DEBUG=1 2025-12-04T08:44:30.8799275Z + TORCH_SERIALIZATION_DEBUG=1 2025-12-04T08:44:30.8799489Z + export VALGRIND=ON 2025-12-04T08:44:30.8799688Z + VALGRIND=ON 2025-12-04T08:44:30.8799897Z + [[ linux-jammy-py3.10-gcc11-build == *clang9* ]] 2025-12-04T08:44:30.8800182Z + [[ linux-jammy-py3.10-gcc11-build == *xpu* ]] 2025-12-04T08:44:30.8800422Z + detect_cuda_arch 2025-12-04T08:44:30.8800624Z + [[ linux-jammy-py3.10-gcc11-build == *cuda* ]] 2025-12-04T08:44:30.8800897Z + [[ linux-jammy-py3.10-gcc11-build == *s390x* ]] 2025-12-04T08:44:30.8801135Z + [[ 0 == \1 ]] 2025-12-04T08:44:30.8801304Z + [[ True == \1 ]] 2025-12-04T08:44:30.8801503Z + [[ linux-jammy-py3.10-gcc11-build != *bazel* ]] 2025-12-04T08:44:30.8802598Z ++ realpath build/custom_test_artifacts 2025-12-04T08:44:30.8836329Z + CUSTOM_TEST_ARTIFACT_BUILD_DIR=/var/lib/jenkins/workspace/build/custom_test_artifacts 2025-12-04T08:44:30.8837268Z + [[ -n '' ]] 2025-12-04T08:44:30.8837478Z + echo 'Environment variables' 2025-12-04T08:44:30.8837706Z Environment variables 2025-12-04T08:44:30.8837890Z + env 2025-12-04T08:44:30.8870711Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2025-12-04T08:44:30.8871239Z CONTINUE_THROUGH_ERROR=True 2025-12-04T08:44:30.8871658Z BUILD_ENVIRONMENT=linux-jammy-py3.10-gcc11-build 2025-12-04T08:44:30.8872350Z VLLM_TEST_HUGGING_FACE_TOKEN=*** 2025-12-04T08:44:30.8872669Z HOSTNAME=f7efe3f2eca9 2025-12-04T08:44:30.8873125Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_524bc399-029d-4ed7-ae9d-e1110b8a36d1 2025-12-04T08:44:30.8873574Z GITHUB_ACTION=__run_3 2025-12-04T08:44:30.8873783Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2025-12-04T08:44:30.8874322Z GITHUB_RUN_NUMBER=168680 2025-12-04T08:44:30.8874564Z TEST_CONFIG=dynamic_cpu_inductor_huggingface 2025-12-04T08:44:30.8874828Z GITHUB_REPOSITORY_OWNER_ID=21003710 2025-12-04T08:44:30.8875087Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2025-12-04T08:44:30.8875330Z SCCACHE_IDLE_TIMEOUT=0 2025-12-04T08:44:30.8875675Z SCRIBE_GRAPHQL_ACCESS_TOKEN=*** 2025-12-04T08:44:30.8875908Z GITHUB_TRIGGERING_ACTOR=pytorchmergebot 2025-12-04T08:44:30.8876147Z GITHUB_REF_TYPE=branch 2025-12-04T08:44:30.8876381Z BASE_SHA=ffd9b0fb4355e97af82fc42cf185c3ffa0fc0a32 2025-12-04T08:44:30.8876631Z XLA_CUDA= 2025-12-04T08:44:30.8876816Z NCCL_LIB_DIR=/usr/local/cuda/lib64/ 2025-12-04T08:44:30.8877299Z HUGGING_FACE_HUB_TOKEN=*** 2025-12-04T08:44:30.8877580Z *** 2025-12-04T08:44:30.8877750Z GITHUB_REPOSITORY_ID=65600975 2025-12-04T08:44:30.8877964Z GITHUB_ACTIONS=true 2025-12-04T08:44:30.8878415Z SCCACHE_ERROR_LOG=/var/lib/jenkins/sccache_error.log 2025-12-04T08:44:30.8878706Z SHA1=ffd9b0fb4355e97af82fc42cf185c3ffa0fc0a32 2025-12-04T08:44:30.8878997Z GITHUB_SHA=ffd9b0fb4355e97af82fc42cf185c3ffa0fc0a32 2025-12-04T08:44:30.8879381Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/inductor.yml@refs/heads/main 2025-12-04T08:44:30.8879734Z UCC_HOME=/usr 2025-12-04T08:44:30.8879924Z TORCH_SERIALIZATION_DEBUG=1 2025-12-04T08:44:30.8880142Z VERBOSE_TEST_LOGS=False 2025-12-04T08:44:30.8880341Z GITHUB_REF=refs/heads/main 2025-12-04T08:44:30.8880544Z SHARD_NUMBER=1 2025-12-04T08:44:30.8880849Z GITHUB_REF_PROTECTED=true 2025-12-04T08:44:30.8881073Z HOME=/var/lib/jenkins 2025-12-04T08:44:30.8881289Z GITHUB_API_URL=https://api.github.com 2025-12-04T08:44:30.8881542Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2025-12-04T08:44:30.8881761Z UCX_COMMIT= 2025-12-04T08:44:30.8881926Z USE_SYSTEM_NCCL=1 2025-12-04T08:44:30.8882108Z NUM_TEST_SHARDS=1 2025-12-04T08:44:30.8882285Z UCX_HOME=/usr 2025-12-04T08:44:30.8882720Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_524bc399-029d-4ed7-ae9d-e1110b8a36d1 2025-12-04T08:44:30.8883322Z JOB_NAME=inductor-cpu-test / test (dynamic_cpu_inductor_huggingface, 1, 1, linux.2xlarge.amx) 2025-12-04T08:44:30.8883894Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_524bc399-029d-4ed7-ae9d-e1110b8a36d1 2025-12-04T08:44:30.8884447Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2025-12-04T08:44:30.8884786Z GITHUB_EVENT_NAME=push 2025-12-04T08:44:30.8884979Z DASHBOARD_TAG= 2025-12-04T08:44:30.8885163Z GITHUB_RUN_ID=19921726389 2025-12-04T08:44:30.8885355Z INSTALLED_OPENBLAS= 2025-12-04T08:44:30.8885778Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_524bc399-029d-4ed7-ae9d-e1110b8a36d1 2025-12-04T08:44:30.8886250Z GITHUB_ACTOR=pytorchmergebot 2025-12-04T08:44:30.8886456Z PR_NUMBER= 2025-12-04T08:44:30.8886617Z DESIRED_CUDA= 2025-12-04T08:44:30.8886793Z GITHUB_RUN_ATTEMPT=1 2025-12-04T08:44:30.8886979Z VALGRIND=ON 2025-12-04T08:44:30.8887154Z ANACONDA_PYTHON_VERSION=3.10 2025-12-04T08:44:30.8887409Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2025-12-04T08:44:30.8887660Z TERM=vt100 2025-12-04T08:44:30.8887922Z INSTALLED_VISION=yes 2025-12-04T08:44:30.8888113Z BRANCH=main 2025-12-04T08:44:30.8888290Z SCCACHE_REGION=us-east-1 2025-12-04T08:44:30.8888499Z OPENSSL_ROOT_DIR=/opt/openssl 2025-12-04T08:44:30.8888720Z BUILD_AOT_INDUCTOR_TEST=yes 2025-12-04T08:44:30.8888935Z CUDA_PATH=/usr/local/cuda 2025-12-04T08:44:30.8889298Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2025-12-04T08:44:30.8889717Z GITHUB_SERVER_URL=https://github.com 2025-12-04T08:44:30.8889948Z UCC_COMMIT= 2025-12-04T08:44:30.8890107Z REENABLED_ISSUES= 2025-12-04T08:44:30.8890285Z DOCS=yes 2025-12-04T08:44:30.8890449Z SHLVL=1 2025-12-04T08:44:30.8890604Z MAX_JOBS=6 2025-12-04T08:44:30.8890778Z GITHUB_ACTOR_ID=97764156 2025-12-04T08:44:30.8891040Z GITHUB_WORKFLOW_SHA=ffd9b0fb4355e97af82fc42cf185c3ffa0fc0a32 2025-12-04T08:44:30.8891319Z GITHUB_REF_NAME=main 2025-12-04T08:44:30.8891660Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2025-12-04T08:44:30.8892104Z GITHUB_JOB=test 2025-12-04T08:44:30.8892300Z NO_TEST_TIMEOUT=False 2025-12-04T08:44:30.8892485Z TD_DISTRIBUTED=False 2025-12-04T08:44:30.8892686Z GITHUB_REPOSITORY=pytorch/pytorch 2025-12-04T08:44:30.8892909Z GITHUB_RETENTION_DAYS=90 2025-12-04T08:44:30.8893104Z OPENSSL_DIR=/opt/openssl 2025-12-04T08:44:30.8893308Z GITHUB_ACTION_REPOSITORY= 2025-12-04T08:44:30.8893848Z PATH=/opt/cache/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2025-12-04T08:44:30.8894374Z GITHUB_BASE_REF= 2025-12-04T08:44:30.8894550Z INSTALLED_ACL= 2025-12-04T08:44:30.8894871Z ARTIFACTS_FILE_SUFFIX=test-dynamic_cpu_inductor_huggingface-1-1-linux.2xlarge.amx_57113856426 2025-12-04T08:44:30.8895226Z CI=true 2025-12-04T08:44:30.8895390Z GITHUB_REPOSITORY_OWNER=pytorch 2025-12-04T08:44:30.8895653Z RUST_LOG=sccache::server=error 2025-12-04T08:44:30.8895860Z JOB_ID=57113856426 2025-12-04T08:44:30.8896027Z GITHUB_HEAD_REF= 2025-12-04T08:44:30.8896200Z GITHUB_ACTION_REF= 2025-12-04T08:44:30.8896417Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2025-12-04T08:44:30.8896669Z TEST_SHOWLOCALS=False 2025-12-04T08:44:30.8896853Z GITHUB_WORKFLOW=inductor 2025-12-04T08:44:30.8897047Z DEBIAN_FRONTEND=noninteractive 2025-12-04T08:44:30.8897443Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_524bc399-029d-4ed7-ae9d-e1110b8a36d1 2025-12-04T08:44:30.8897857Z NO_TD=False 2025-12-04T08:44:30.8898029Z SKIP_SCCACHE_INITIALIZATION=1 2025-12-04T08:44:30.8898238Z NCCL_INCLUDE_DIR=/usr/local/cuda/include/ 2025-12-04T08:44:30.8898458Z _=/usr/bin/env 2025-12-04T08:44:30.8898631Z + echo 'Testing pytorch' 2025-12-04T08:44:30.8898816Z Testing pytorch 2025-12-04T08:44:30.8899176Z + export LANG=C.UTF-8 2025-12-04T08:44:30.8899368Z + LANG=C.UTF-8 2025-12-04T08:44:30.8971952Z + PR_NUMBER= 2025-12-04T08:44:30.8972285Z + [[ dynamic_cpu_inductor_huggingface == \d\e\f\a\u\l\t ]] 2025-12-04T08:44:30.8972641Z + [[ dynamic_cpu_inductor_huggingface == \d\i\s\t\r\i\b\u\t\e\d ]] 2025-12-04T08:44:30.8972963Z + [[ dynamic_cpu_inductor_huggingface == \s\l\o\w ]] 2025-12-04T08:44:30.8973264Z + [[ linux-jammy-py3.10-gcc11-build == *slow-gradcheck* ]] 2025-12-04T08:44:30.8973569Z + [[ linux-jammy-py3.10-gcc11-build == *cuda* ]] 2025-12-04T08:44:30.8973844Z + [[ linux-jammy-py3.10-gcc11-build == *rocm* ]] 2025-12-04T08:44:30.8974121Z + [[ linux-jammy-py3.10-gcc11-build == *xpu* ]] 2025-12-04T08:44:30.8974458Z + [[ dynamic_cpu_inductor_huggingface == *crossref* ]] 2025-12-04T08:44:30.8974739Z + [[ linux-jammy-py3.10-gcc11-build == *rocm* ]] 2025-12-04T08:44:30.8975009Z + [[ linux-jammy-py3.10-gcc11-build == *xpu* ]] 2025-12-04T08:44:30.8975276Z + [[ linux-jammy-py3.10-gcc11-build != *-bazel-* ]] 2025-12-04T08:44:30.8975545Z + pip_install ninja==1.10.2 2025-12-04T08:44:30.8975828Z + pip_install_pkg='python3 -m pip install --progress-bar off' 2025-12-04T08:44:30.8976157Z + python3 -m pip install --progress-bar off ninja==1.10.2 2025-12-04T08:44:31.2638937Z Collecting ninja==1.10.2 2025-12-04T08:44:31.2859447Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl.metadata (5.0 kB) 2025-12-04T08:44:31.2955440Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (108 kB) 2025-12-04T08:44:32.1101355Z Installing collected packages: ninja 2025-12-04T08:44:32.1101688Z Attempting uninstall: ninja 2025-12-04T08:44:32.1109229Z Found existing installation: ninja 1.11.1.4 2025-12-04T08:44:32.1126461Z Uninstalling ninja-1.11.1.4: 2025-12-04T08:44:32.1311911Z Successfully uninstalled ninja-1.11.1.4 2025-12-04T08:44:32.3885894Z Successfully installed ninja-1.10.2 2025-12-04T08:44:32.4995905Z + export PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2025-12-04T08:44:32.4997397Z + PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2025-12-04T08:44:32.4998052Z + [[ linux-jammy-py3.10-gcc11-build == *aarch64* ]] 2025-12-04T08:44:32.4998526Z + [[ linux-jammy-py3.10-gcc11-build == *asan* ]] 2025-12-04T08:44:32.4998815Z + [[ linux-jammy-py3.10-gcc11-build == *-debug* ]] 2025-12-04T08:44:32.4999096Z + [[ linux-jammy-py3.10-gcc11-build != *-bazel-* ]] 2025-12-04T08:44:32.4999507Z + echo 'We are not in debug mode: linux-jammy-py3.10-gcc11-build. Expect the assertion to pass' 2025-12-04T08:44:32.4999981Z We are not in debug mode: linux-jammy-py3.10-gcc11-build. Expect the assertion to pass 2025-12-04T08:44:32.5000305Z + cd test 2025-12-04T08:44:32.5000574Z + python -c 'import torch; torch._C._crash_if_debug_asserts_fail(424242)' 2025-12-04T08:44:32.9750061Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/__init__.py:65: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-12-04T08:44:32.9751105Z import pynvml # type: ignore[import] 2025-12-04T08:44:34.0274500Z + [[ dynamic_cpu_inductor_huggingface == \n\o\g\p\u\_\N\O\_\A\V\X\2 ]] 2025-12-04T08:44:34.0274949Z + [[ dynamic_cpu_inductor_huggingface == \n\o\g\p\u\_\A\V\X\5\1\2 ]] 2025-12-04T08:44:34.0275338Z + [[ dynamic_cpu_inductor_huggingface == \l\e\g\a\c\y\_\n\v\i\d\i\a\_\d\r\i\v\e\r ]] 2025-12-04T08:44:34.0275703Z + DYNAMO_BENCHMARK_FLAGS=() 2025-12-04T08:44:34.0275972Z + [[ dynamic_cpu_inductor_huggingface == *pr_time_benchmarks* ]] 2025-12-04T08:44:34.0276307Z + [[ dynamic_cpu_inductor_huggingface == *dynamo_eager* ]] 2025-12-04T08:44:34.0276604Z + [[ dynamic_cpu_inductor_huggingface == *aot_eager* ]] 2025-12-04T08:44:34.0276897Z + [[ dynamic_cpu_inductor_huggingface == *aot_inductor* ]] 2025-12-04T08:44:34.0277261Z + [[ dynamic_cpu_inductor_huggingface == *max_autotune_inductor* ]] 2025-12-04T08:44:34.0277606Z + [[ dynamic_cpu_inductor_huggingface == *inductor* ]] 2025-12-04T08:44:34.0277906Z + [[ dynamic_cpu_inductor_huggingface != *perf* ]] 2025-12-04T08:44:34.0278261Z + DYNAMO_BENCHMARK_FLAGS+=(--inductor) 2025-12-04T08:44:34.0278538Z + [[ dynamic_cpu_inductor_huggingface == *dynamic* ]] 2025-12-04T08:44:34.0278876Z + DYNAMO_BENCHMARK_FLAGS+=(--dynamic-shapes --dynamic-batch-only) 2025-12-04T08:44:34.0279217Z + [[ dynamic_cpu_inductor_huggingface == *cpu* ]] 2025-12-04T08:44:34.0279468Z + DYNAMO_BENCHMARK_FLAGS+=(--device cpu) 2025-12-04T08:44:34.0480953Z + [[ linux-jammy-py3.10-gcc11-build == *libtorch* ]] 2025-12-04T08:44:34.0481330Z + [[ linux-jammy-py3.10-gcc11-build == *-bazel-* ]] 2025-12-04T08:44:34.0485385Z + cd test 2025-12-04T08:44:34.0485744Z + python -c 'import torch; print(torch.__config__.show())' 2025-12-04T08:44:34.3534054Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/__init__.py:65: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-12-04T08:44:34.3535552Z import pynvml # type: ignore[import] 2025-12-04T08:44:35.0532552Z PyTorch built with: 2025-12-04T08:44:35.0532815Z - GCC 11.4 2025-12-04T08:44:35.0532992Z - C++ Version: 201703 2025-12-04T08:44:35.0533402Z - Intel(R) oneAPI Math Kernel Library Version 2024.2-Product Build 20240605 for Intel(R) 64 architecture applications 2025-12-04T08:44:35.0533892Z - Intel(R) MKL-DNN v3.7.1 (Git Hash 8d263e693366ef8db40acc569cc7d8edf644556d) 2025-12-04T08:44:35.0534205Z - OpenMP 201511 (a.k.a. OpenMP 4.5) 2025-12-04T08:44:35.0534465Z - LAPACK is enabled (usually provided by MKL) 2025-12-04T08:44:35.0534710Z - NNPACK is enabled 2025-12-04T08:44:35.0534921Z - CPU capability usage: AVX512 2025-12-04T08:44:35.0538022Z - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, COMMIT_SHA=ffd9b0fb4355e97af82fc42cf185c3ffa0fc0a32, CXX_COMPILER=/opt/cache/bin/c++, CXX_FLAGS= -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOCUPTI -DLIBKINETO_NOROCTRACER -DLIBKINETO_NOXPUPTI=ON -DUSE_FBGEMM -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -DC10_NODEPRECATED -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=range-loop-construct -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-unknown-pragmas -Wno-unused-parameter -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=old-style-cast -faligned-new -Werror -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, TORCH_VERSION=2.10.0, USE_CUDA=OFF, USE_CUDNN=OFF, USE_CUSPARSELT=OFF, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=OFF, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, USE_ROCM_KERNEL_ASSERT=OFF, USE_XCCL=OFF, USE_XPU=OFF, 2025-12-04T08:44:35.0541527Z 2025-12-04T08:44:35.2890927Z + cd test 2025-12-04T08:44:35.2891264Z + python -c 'import torch; print(torch.__config__.parallel_info())' 2025-12-04T08:44:35.5551138Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/__init__.py:65: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-12-04T08:44:35.5552064Z import pynvml # type: ignore[import] 2025-12-04T08:44:36.2538487Z ATen/Parallel: 2025-12-04T08:44:36.2538812Z at::get_num_threads() : 4 2025-12-04T08:44:36.2554501Z at::get_num_interop_threads() : 4 2025-12-04T08:44:36.2554828Z OpenMP 201511 (a.k.a. OpenMP 4.5) 2025-12-04T08:44:36.2555055Z omp_get_max_threads() : 4 2025-12-04T08:44:36.2555462Z Intel(R) oneAPI Math Kernel Library Version 2024.2-Product Build 20240605 for Intel(R) 64 architecture applications 2025-12-04T08:44:36.2555909Z mkl_get_max_threads() : 4 2025-12-04T08:44:36.2556213Z Intel(R) MKL-DNN v3.7.1 (Git Hash 8d263e693366ef8db40acc569cc7d8edf644556d) 2025-12-04T08:44:36.2556531Z std::thread::hardware_concurrency() : 8 2025-12-04T08:44:36.2556773Z Environment variables: 2025-12-04T08:44:36.2556977Z OMP_NUM_THREADS : [not set] 2025-12-04T08:44:36.2557179Z MKL_NUM_THREADS : [not set] 2025-12-04T08:44:36.2557424Z ATen parallel backend: OpenMP 2025-12-04T08:44:36.2557563Z 2025-12-04T08:44:36.5299338Z + [[ dynamic_cpu_inductor_huggingface == *numpy_2* ]] 2025-12-04T08:44:36.5300782Z + [[ linux-jammy-py3.10-gcc11-build == *aarch64* ]] 2025-12-04T08:44:36.5301095Z + [[ dynamic_cpu_inductor_huggingface == *backward* ]] 2025-12-04T08:44:36.5301427Z + [[ dynamic_cpu_inductor_huggingface == *libtorch_agnostic_targetting* ]] 2025-12-04T08:44:36.5301757Z + [[ dynamic_cpu_inductor_huggingface == *xla* ]] 2025-12-04T08:44:36.5302020Z + [[ dynamic_cpu_inductor_huggingface == *vllm* ]] 2025-12-04T08:44:36.5302321Z + [[ dynamic_cpu_inductor_huggingface == *executorch* ]] 2025-12-04T08:44:36.5302935Z + [[ dynamic_cpu_inductor_huggingface == \j\i\t\_\l\e\g\a\c\y ]] 2025-12-04T08:44:36.5303267Z + [[ dynamic_cpu_inductor_huggingface == \q\u\a\n\t\i\z\a\t\i\o\n ]] 2025-12-04T08:44:36.5303583Z + [[ linux-jammy-py3.10-gcc11-build == *libtorch* ]] 2025-12-04T08:44:36.5303869Z + [[ dynamic_cpu_inductor_huggingface == distributed ]] 2025-12-04T08:44:36.5304172Z + [[ dynamic_cpu_inductor_huggingface == *operator_benchmark* ]] 2025-12-04T08:44:36.5304515Z + [[ dynamic_cpu_inductor_huggingface == *operator_microbenchmark* ]] 2025-12-04T08:44:36.5304871Z + [[ dynamic_cpu_inductor_huggingface == *attention_microbenchmark* ]] 2025-12-04T08:44:36.5305214Z + [[ dynamic_cpu_inductor_huggingface == *inductor_distributed* ]] 2025-12-04T08:44:36.5305532Z + [[ dynamic_cpu_inductor_huggingface == *inductor-halide* ]] 2025-12-04T08:44:36.5305837Z + [[ dynamic_cpu_inductor_huggingface == *inductor-pallas* ]] 2025-12-04T08:44:36.5306264Z + [[ dynamic_cpu_inductor_huggingface == *inductor-triton-cpu* ]] 2025-12-04T08:44:36.5306630Z + [[ dynamic_cpu_inductor_huggingface == *inductor-micro-benchmark* ]] 2025-12-04T08:44:36.5307027Z + [[ dynamic_cpu_inductor_huggingface == *aoti_cross_compile_for_windows* ]] 2025-12-04T08:44:36.5307365Z + [[ dynamic_cpu_inductor_huggingface == *huggingface* ]] 2025-12-04T08:44:36.5307617Z + install_torchvision 2025-12-04T08:44:36.5307932Z + local orig_preload 2025-12-04T08:44:36.5308117Z + local commit 2025-12-04T08:44:36.5308309Z ++ get_pinned_commit vision 2025-12-04T08:44:36.5308533Z ++ cat .github/ci_commit_pins/vision.txt 2025-12-04T08:44:36.5333322Z + commit=617079d944b0e72632311c30ae2bbdf1168b901e 2025-12-04T08:44:36.5333624Z + orig_preload= 2025-12-04T08:44:36.5333832Z + '[' -n '' ']' 2025-12-04T08:44:36.5334044Z + [[ linux-jammy-py3.10-gcc11-build == *cuda* ]] 2025-12-04T08:44:36.5334511Z + pip_build_and_install git+https://github.com/pytorch/vision.git@617079d944b0e72632311c30ae2bbdf1168b901e dist/vision 2025-12-04T08:44:36.5335112Z + local build_target=git+https://github.com/pytorch/vision.git@617079d944b0e72632311c30ae2bbdf1168b901e 2025-12-04T08:44:36.5335490Z + local wheel_dir=dist/vision 2025-12-04T08:44:36.5335689Z + local found_whl=0 2025-12-04T08:44:36.5335882Z + for file in "${wheel_dir}"/*.whl 2025-12-04T08:44:36.5336210Z + [[ -f dist/vision/torchvision-0.25.0a0+617079d-cp310-cp310-linux_x86_64.whl ]] 2025-12-04T08:44:36.5336524Z + found_whl=1 2025-12-04T08:44:36.5336705Z + break 2025-12-04T08:44:36.5336861Z + '[' 1 == 0 ']' 2025-12-04T08:44:36.5337051Z + for file in "${wheel_dir}"/*.whl 2025-12-04T08:44:36.5337394Z + pip_install_whl dist/vision/torchvision-0.25.0a0+617079d-cp310-cp310-linux_x86_64.whl 2025-12-04T08:44:36.5337845Z + args=('dist/vision/torchvision-0.25.0a0+617079d-cp310-cp310-linux_x86_64.whl') 2025-12-04T08:44:36.5338163Z + local args 2025-12-04T08:44:36.5338446Z + [[ dist/vision/torchvision-0.25.0a0+617079d-cp310-cp310-linux_x86_64.whl == *\ * ]] 2025-12-04T08:44:36.5338800Z + for path in "${args[@]}" 2025-12-04T08:44:36.5339132Z + echo 'Installing dist/vision/torchvision-0.25.0a0+617079d-cp310-cp310-linux_x86_64.whl' 2025-12-04T08:44:36.5339579Z Installing dist/vision/torchvision-0.25.0a0+617079d-cp310-cp310-linux_x86_64.whl 2025-12-04T08:44:36.5340085Z + python3 -mpip install --no-index --no-deps dist/vision/torchvision-0.25.0a0+617079d-cp310-cp310-linux_x86_64.whl 2025-12-04T08:44:36.8262843Z Processing ./dist/vision/torchvision-0.25.0a0+617079d-cp310-cp310-linux_x86_64.whl 2025-12-04T08:44:36.8335841Z Installing collected packages: torchvision 2025-12-04T08:44:37.2388169Z Successfully installed torchvision-0.25.0a0+617079d 2025-12-04T08:44:37.2846425Z + '[' -n '' ']' 2025-12-04T08:44:37.2846683Z + id=0 2025-12-04T08:44:37.2846884Z + test_dynamo_benchmark huggingface 0 2025-12-04T08:44:37.2847127Z ++ pwd 2025-12-04T08:44:37.2847364Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports 2025-12-04T08:44:37.2847708Z + local suite=huggingface 2025-12-04T08:44:37.2847909Z + shift 2025-12-04T08:44:37.2848066Z + local shard_id=0 2025-12-04T08:44:37.2848687Z + shift 2025-12-04T08:44:37.2848861Z + extra_args=() 2025-12-04T08:44:37.2849037Z + local extra_args 2025-12-04T08:44:37.2849245Z + [[ linux-jammy-py3.10-gcc11-build == *cuda13* ]] 2025-12-04T08:44:37.2849557Z + [[ dynamic_cpu_inductor_huggingface == *perf_compare* ]] 2025-12-04T08:44:37.2849850Z + [[ dynamic_cpu_inductor_huggingface == *perf* ]] 2025-12-04T08:44:37.2850110Z + [[ dynamic_cpu_inductor_huggingface == *cpu* ]] 2025-12-04T08:44:37.2850349Z + local dt=float32 2025-12-04T08:44:37.2850551Z + [[ dynamic_cpu_inductor_huggingface == *amp* ]] 2025-12-04T08:44:37.2850819Z + [[ dynamic_cpu_inductor_huggingface == *freezing* ]] 2025-12-04T08:44:37.2851171Z + test_single_dynamo_benchmark inference huggingface 0 --inference --float32 2025-12-04T08:44:37.2851558Z ++ pwd 2025-12-04T08:44:37.2853212Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports 2025-12-04T08:44:37.2853772Z + mkdir -p /var/lib/jenkins/workspace/test/test-reports 2025-12-04T08:44:37.2881006Z + local name=inference 2025-12-04T08:44:37.2881389Z + shift 2025-12-04T08:44:37.2881616Z + local suite=huggingface 2025-12-04T08:44:37.2881908Z + shift 2025-12-04T08:44:37.2882130Z + local shard_id=0 2025-12-04T08:44:37.2882391Z + shift 2025-12-04T08:44:37.2882584Z + partition_flags=() 2025-12-04T08:44:37.2882784Z + local partition_flags 2025-12-04T08:44:37.2882980Z + [[ -n 1 ]] 2025-12-04T08:44:37.2883137Z + [[ -n 0 ]] 2025-12-04T08:44:37.2883437Z + partition_flags=(--total-partitions "$NUM_TEST_SHARDS" --partition-id "$shard_id") 2025-12-04T08:44:37.2883834Z + [[ dynamic_cpu_inductor_huggingface == *perf_compare* ]] 2025-12-04T08:44:37.2884130Z + [[ dynamic_cpu_inductor_huggingface == *perf* ]] 2025-12-04T08:44:37.2884397Z + [[ dynamic_cpu_inductor_huggingface == *_avx2* ]] 2025-12-04T08:44:37.2884674Z + [[ dynamic_cpu_inductor_huggingface == *_avx512* ]] 2025-12-04T08:44:37.2885630Z + python benchmarks/dynamo/huggingface.py --ci --accuracy --timing --explain --print-compilation-time --inductor --dynamic-shapes --dynamic-batch-only --device cpu --inference --float32 --total-partitions 1 --partition-id 0 --output /var/lib/jenkins/workspace/test/test-reports/inference_huggingface.csv 2025-12-04T08:44:38.0247988Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/__init__.py:65: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-12-04T08:44:38.0248900Z import pynvml # type: ignore[import] 2025-12-04T08:44:41.5609174Z 2025-12-04T08:44:41.5609898Z config.json: 0% 0.00/694 [00:00bcxy", (query, key)) # multiply 2025-12-04T08:46:58.8503750Z 2025-12-04T08:46:58.8503870Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.8504431Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.8504965Z layer_outputs = layer_module( 2025-12-04T08:46:58.8505346Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.8505737Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.8506199Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.8506657Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.8507118Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.8507567Z self_outputs = self.self( 2025-12-04T08:46:58.8508009Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 523, in forward 2025-12-04T08:46:58.8508499Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-12-04T08:46:58.8509045Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 795, in _sliding_chunks_query_key_matmul 2025-12-04T08:46:58.8509679Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-12-04T08:46:58.8509952Z 2025-12-04T08:46:58.8510066Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.8510635Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.8511166Z layer_outputs = layer_module( 2025-12-04T08:46:58.8511547Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.8511977Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.8512437Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.8512896Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.8513344Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.8513797Z self_outputs = self.self( 2025-12-04T08:46:58.8514234Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 523, in forward 2025-12-04T08:46:58.8514714Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-12-04T08:46:58.8515257Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 795, in _sliding_chunks_query_key_matmul 2025-12-04T08:46:58.8515940Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-12-04T08:46:58.8516207Z 2025-12-04T08:46:58.8516320Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.8516845Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.8517348Z layer_outputs = layer_module( 2025-12-04T08:46:58.8517708Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.8518154Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.8518590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.8519067Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.8519547Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.8520012Z self_outputs = self.self( 2025-12-04T08:46:58.8520456Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 523, in forward 2025-12-04T08:46:58.8521085Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-12-04T08:46:58.8521602Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 795, in _sliding_chunks_query_key_matmul 2025-12-04T08:46:58.8522191Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-12-04T08:46:58.8522450Z 2025-12-04T08:46:58.8522537Z cudagraph partition due to non gpu ops 2025-12-04T08:46:58.8522765Z cudagraph partition due to non gpu ops 2025-12-04T08:46:58.8522989Z cudagraph partition due to non gpu ops 2025-12-04T08:46:58.8523198Z cudagraph partition due to non gpu ops 2025-12-04T08:46:58.8523447Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.8523998Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.8524520Z layer_outputs = layer_module( 2025-12-04T08:46:58.8524900Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.8525298Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.8525756Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.8526203Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.8526656Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.8527106Z self_outputs = self.self( 2025-12-04T08:46:58.8527633Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 535, in forward 2025-12-04T08:46:58.8528123Z diagonal_mask = self._sliding_chunks_query_key_matmul( 2025-12-04T08:46:58.8528688Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 833, in _sliding_chunks_query_key_matmul 2025-12-04T08:46:58.8529294Z self._mask_invalid_locations(diagonal_attention_scores, window_overlap) 2025-12-04T08:46:58.8529877Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 763, in _mask_invalid_locations 2025-12-04T08:46:58.8530411Z ).where(beginning_mask.bool(), beginning_input) 2025-12-04T08:46:58.8530583Z 2025-12-04T08:46:58.8530670Z cudagraph partition due to non gpu ops 2025-12-04T08:46:58.8531000Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.8531563Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.8532102Z layer_outputs = layer_module( 2025-12-04T08:46:58.8532488Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.8532890Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.8533347Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.8533813Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.8534279Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.8534741Z self_outputs = self.self( 2025-12-04T08:46:58.8535178Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 540, in forward 2025-12-04T08:46:58.8535636Z attn_scores += diagonal_mask 2025-12-04T08:46:58.8535769Z 2025-12-04T08:46:58.8535892Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.8536451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.8536984Z layer_outputs = layer_module( 2025-12-04T08:46:58.8537364Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.8537764Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.8538216Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.8538677Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.8539135Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.8539587Z self_outputs = self.self( 2025-12-04T08:46:58.8540013Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 578, in forward 2025-12-04T08:46:58.8540472Z attn_probs = nn.functional.softmax( 2025-12-04T08:46:58.8540619Z 2025-12-04T08:46:58.8540741Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.8541293Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.8541821Z layer_outputs = layer_module( 2025-12-04T08:46:58.8542199Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.8542600Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.8543091Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.8543553Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.8544009Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.8544462Z self_outputs = self.self( 2025-12-04T08:46:58.8544891Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 612, in forward 2025-12-04T08:46:58.8545430Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-12-04T08:46:58.8545979Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 862, in _sliding_chunks_matmul_attn_probs_value 2025-12-04T08:46:58.8546624Z padded_value = nn.functional.pad(value, (0, 0, window_overlap, window_overlap), value=-1) 2025-12-04T08:46:58.8547073Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/functional.py", line 5418, in pad 2025-12-04T08:46:58.8547443Z return torch._C._nn.pad(input, pad, mode, value) 2025-12-04T08:46:58.8547602Z 2025-12-04T08:46:58.8547718Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.8548247Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.8548772Z layer_outputs = layer_module( 2025-12-04T08:46:58.8549156Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.8549557Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.8550012Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.8550477Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.8550937Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.8551390Z self_outputs = self.self( 2025-12-04T08:46:58.8551818Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 612, in forward 2025-12-04T08:46:58.8552324Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-12-04T08:46:58.8552903Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 875, in _sliding_chunks_matmul_attn_probs_value 2025-12-04T08:46:58.8553502Z chunked_attn_probs = self._pad_and_diagonalize(chunked_attn_probs) 2025-12-04T08:46:58.8554052Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 698, in _pad_and_diagonalize 2025-12-04T08:46:58.8554571Z chunked_hidden_states = nn.functional.pad( 2025-12-04T08:46:58.8554953Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/functional.py", line 5418, in pad 2025-12-04T08:46:58.8555345Z return torch._C._nn.pad(input, pad, mode, value) 2025-12-04T08:46:58.8555520Z 2025-12-04T08:46:58.8555798Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.8556377Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.8556914Z layer_outputs = layer_module( 2025-12-04T08:46:58.8557292Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.8557694Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.8558227Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.8558754Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.8559216Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.8559690Z self_outputs = self.self( 2025-12-04T08:46:58.8560104Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 612, in forward 2025-12-04T08:46:58.8560569Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-12-04T08:46:58.8561122Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 877, in _sliding_chunks_matmul_attn_probs_value 2025-12-04T08:46:58.8561738Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-12-04T08:46:58.8561971Z 2025-12-04T08:46:58.8562093Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.8562682Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.8563180Z layer_outputs = layer_module( 2025-12-04T08:46:58.8563542Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.8563914Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.8564335Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.8564764Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.8565188Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.8565605Z self_outputs = self.self( 2025-12-04T08:46:58.8566006Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 612, in forward 2025-12-04T08:46:58.8566476Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-12-04T08:46:58.8567038Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 877, in _sliding_chunks_matmul_attn_probs_value 2025-12-04T08:46:58.8567636Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-12-04T08:46:58.8567848Z 2025-12-04T08:46:58.8568482Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.8569043Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.8569571Z layer_outputs = layer_module( 2025-12-04T08:46:58.8569948Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.8570344Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.8570805Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.8571262Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.8571705Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.8572155Z self_outputs = self.self( 2025-12-04T08:46:58.8572593Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 617, in forward 2025-12-04T08:46:58.8573163Z attn_output = attn_output.transpose(0, 1).reshape(seq_len, batch_size, embed_dim).contiguous() 2025-12-04T08:46:58.8573422Z 2025-12-04T08:46:58.8573536Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.8574138Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.8574673Z layer_outputs = layer_module( 2025-12-04T08:46:58.8575053Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.8575464Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.8575923Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.8576384Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.8576839Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1143, in forward 2025-12-04T08:46:58.8577326Z attn_output = self.output(self_outputs[0], hidden_states) 2025-12-04T08:46:58.8577816Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1093, in forward 2025-12-04T08:46:58.8578331Z hidden_states = self.dense(hidden_states) 2025-12-04T08:46:58.8578487Z 2025-12-04T08:46:58.8578600Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.8579166Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.8579694Z layer_outputs = layer_module( 2025-12-04T08:46:58.8580073Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.8580461Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.8580920Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1210, in forward 2025-12-04T08:46:58.8581385Z layer_output = apply_chunking_to_forward( 2025-12-04T08:46:58.8581836Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:46:58.8582268Z return forward_fn(*input_tensors) 2025-12-04T08:46:58.8582724Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1217, in ff_chunk 2025-12-04T08:46:58.8583196Z intermediate_output = self.intermediate(attn_output) 2025-12-04T08:46:58.8583657Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1159, in forward 2025-12-04T08:46:58.8584087Z hidden_states = self.dense(hidden_states) 2025-12-04T08:46:58.8584234Z 2025-12-04T08:46:58.8584341Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.8584869Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.8585365Z layer_outputs = layer_module( 2025-12-04T08:46:58.8585724Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.8586094Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.8586525Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1210, in forward 2025-12-04T08:46:58.8586960Z layer_output = apply_chunking_to_forward( 2025-12-04T08:46:58.8587380Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:46:58.8587803Z return forward_fn(*input_tensors) 2025-12-04T08:46:58.8588277Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1217, in ff_chunk 2025-12-04T08:46:58.8588766Z intermediate_output = self.intermediate(attn_output) 2025-12-04T08:46:58.8589256Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1160, in forward 2025-12-04T08:46:58.8589810Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:46:58.8590238Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T08:46:58.8590627Z return self.act(input) 2025-12-04T08:46:58.8590762Z 2025-12-04T08:46:58.8590879Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.8591450Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.8591985Z layer_outputs = layer_module( 2025-12-04T08:46:58.8592376Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.8592784Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.8593291Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1210, in forward 2025-12-04T08:46:58.8593748Z layer_output = apply_chunking_to_forward( 2025-12-04T08:46:58.8594199Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:46:58.8594644Z return forward_fn(*input_tensors) 2025-12-04T08:46:58.8595102Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-12-04T08:46:58.8595631Z layer_output = self.output(intermediate_output, attn_output) 2025-12-04T08:46:58.8596146Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1173, in forward 2025-12-04T08:46:58.8596623Z hidden_states = self.dense(hidden_states) 2025-12-04T08:46:58.8596774Z 2025-12-04T08:46:58.8596894Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.8597471Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.8598011Z layer_outputs = layer_module( 2025-12-04T08:46:58.8598489Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.8598894Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.8599384Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.8599865Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.8600347Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.8600814Z self_outputs = self.self( 2025-12-04T08:46:58.8601287Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 508, in forward 2025-12-04T08:46:58.8601781Z query_vectors = self.query(hidden_states) 2025-12-04T08:46:58.8601936Z 2025-12-04T08:46:58.8602053Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.8602646Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.8603202Z layer_outputs = layer_module( 2025-12-04T08:46:58.8603598Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.8604008Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.8604490Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.8604971Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.8605507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.8605966Z self_outputs = self.self( 2025-12-04T08:46:58.8606413Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 523, in forward 2025-12-04T08:46:58.8606912Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-12-04T08:46:58.8607470Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 795, in _sliding_chunks_query_key_matmul 2025-12-04T08:46:58.8608134Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-12-04T08:46:58.8608412Z 2025-12-04T08:46:58.8608527Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.8609108Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.8609697Z layer_outputs = layer_module( 2025-12-04T08:46:58.8610080Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.8610485Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.8610955Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.8611417Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.8611889Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.8612352Z self_outputs = self.self( 2025-12-04T08:46:58.8612799Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 509, in forward 2025-12-04T08:46:58.8613274Z key_vectors = self.key(hidden_states) 2025-12-04T08:46:58.8613432Z 2025-12-04T08:46:58.8613552Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.8614134Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.8614686Z layer_outputs = layer_module( 2025-12-04T08:46:58.8615058Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.8615458Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.8615915Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.8616369Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.8616823Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.8617275Z self_outputs = self.self( 2025-12-04T08:46:58.8617713Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 523, in forward 2025-12-04T08:46:58.8618191Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-12-04T08:46:58.8618733Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 795, in _sliding_chunks_query_key_matmul 2025-12-04T08:46:58.8619366Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-12-04T08:46:58.8619630Z 2025-12-04T08:46:58.8619750Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.8620300Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.8621011Z layer_outputs = layer_module( 2025-12-04T08:46:58.8621498Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.8621911Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.8622370Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.8622832Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.8623297Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.8623742Z self_outputs = self.self( 2025-12-04T08:46:58.8624182Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 523, in forward 2025-12-04T08:46:58.8624672Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-12-04T08:46:58.8625218Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 795, in _sliding_chunks_query_key_matmul 2025-12-04T08:46:58.8625918Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-12-04T08:46:58.8626189Z 2025-12-04T08:46:58.8626303Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.8626865Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.8627392Z layer_outputs = layer_module( 2025-12-04T08:46:58.8627760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.8628153Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.8628607Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.8629068Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.8629515Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.8629966Z self_outputs = self.self( 2025-12-04T08:46:58.8630397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 523, in forward 2025-12-04T08:46:58.8630875Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-12-04T08:46:58.8631388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 795, in _sliding_chunks_query_key_matmul 2025-12-04T08:46:58.8631997Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-12-04T08:46:58.8632242Z 2025-12-04T08:46:58.8632334Z cudagraph partition due to non gpu ops 2025-12-04T08:46:58.8632550Z cudagraph partition due to non gpu ops 2025-12-04T08:46:58.8632770Z cudagraph partition due to non gpu ops 2025-12-04T08:46:58.8632989Z cudagraph partition due to non gpu ops 2025-12-04T08:46:58.8633224Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.8633750Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.8634252Z layer_outputs = layer_module( 2025-12-04T08:46:58.8634619Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.8635007Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.8635466Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.8635924Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.8636379Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.8636864Z self_outputs = self.self( 2025-12-04T08:46:58.8637304Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 535, in forward 2025-12-04T08:46:58.8637797Z diagonal_mask = self._sliding_chunks_query_key_matmul( 2025-12-04T08:46:58.8638408Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 833, in _sliding_chunks_query_key_matmul 2025-12-04T08:46:58.8639036Z self._mask_invalid_locations(diagonal_attention_scores, window_overlap) 2025-12-04T08:46:58.8639670Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 763, in _mask_invalid_locations 2025-12-04T08:46:58.8640214Z ).where(beginning_mask.bool(), beginning_input) 2025-12-04T08:46:58.8640385Z 2025-12-04T08:46:58.8640541Z cudagraph partition due to non gpu ops 2025-12-04T08:46:58.8640793Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.8641333Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.8641851Z layer_outputs = layer_module( 2025-12-04T08:46:58.8642226Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.8642633Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.8643095Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.8643553Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.8644000Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.8644457Z self_outputs = self.self( 2025-12-04T08:46:58.8644899Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 540, in forward 2025-12-04T08:46:58.8645344Z attn_scores += diagonal_mask 2025-12-04T08:46:58.8645486Z 2025-12-04T08:46:58.8645600Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.8646163Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.8646692Z layer_outputs = layer_module( 2025-12-04T08:46:58.8647058Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.8647441Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.8647897Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.8648350Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.8648804Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.8649254Z self_outputs = self.self( 2025-12-04T08:46:58.8649688Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 578, in forward 2025-12-04T08:46:58.8650138Z attn_probs = nn.functional.softmax( 2025-12-04T08:46:58.8650292Z 2025-12-04T08:46:58.8650405Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.8650966Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.8651494Z layer_outputs = layer_module( 2025-12-04T08:46:58.8651864Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.8652264Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.8652779Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.8653237Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.8653682Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.8654135Z self_outputs = self.self( 2025-12-04T08:46:58.8654571Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 510, in forward 2025-12-04T08:46:58.8655038Z value_vectors = self.value(hidden_states) 2025-12-04T08:46:58.8655190Z 2025-12-04T08:46:58.8655303Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.8655863Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.8656435Z layer_outputs = layer_module( 2025-12-04T08:46:58.8656806Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.8657203Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.8657661Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.8658115Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.8658556Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.8659004Z self_outputs = self.self( 2025-12-04T08:46:58.8659434Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 612, in forward 2025-12-04T08:46:58.8659941Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-12-04T08:46:58.8660518Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 862, in _sliding_chunks_matmul_attn_probs_value 2025-12-04T08:46:58.8661167Z padded_value = nn.functional.pad(value, (0, 0, window_overlap, window_overlap), value=-1) 2025-12-04T08:46:58.8661633Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/functional.py", line 5418, in pad 2025-12-04T08:46:58.8662022Z return torch._C._nn.pad(input, pad, mode, value) 2025-12-04T08:46:58.8662186Z 2025-12-04T08:46:58.8662298Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.8662858Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.8663383Z layer_outputs = layer_module( 2025-12-04T08:46:58.8663756Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.8664158Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.8664613Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.8665067Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.8665514Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.8665964Z self_outputs = self.self( 2025-12-04T08:46:58.8666399Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 612, in forward 2025-12-04T08:46:58.8666901Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-12-04T08:46:58.8667467Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 875, in _sliding_chunks_matmul_attn_probs_value 2025-12-04T08:46:58.8668108Z chunked_attn_probs = self._pad_and_diagonalize(chunked_attn_probs) 2025-12-04T08:46:58.8668668Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 698, in _pad_and_diagonalize 2025-12-04T08:46:58.8669182Z chunked_hidden_states = nn.functional.pad( 2025-12-04T08:46:58.8669550Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/functional.py", line 5418, in pad 2025-12-04T08:46:58.8669936Z return torch._C._nn.pad(input, pad, mode, value) 2025-12-04T08:46:58.8670102Z 2025-12-04T08:46:58.8670222Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.8670781Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.8671317Z layer_outputs = layer_module( 2025-12-04T08:46:58.8671690Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.8672088Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.8672537Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.8672995Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.8673451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.8673888Z self_outputs = self.self( 2025-12-04T08:46:58.8674311Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 612, in forward 2025-12-04T08:46:58.8674811Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-12-04T08:46:58.8675394Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 877, in _sliding_chunks_matmul_attn_probs_value 2025-12-04T08:46:58.8676017Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-12-04T08:46:58.8676244Z 2025-12-04T08:46:58.8676357Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.8676919Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.8677450Z layer_outputs = layer_module( 2025-12-04T08:46:58.8677827Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.8678307Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.8678778Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.8679267Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.8679728Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.8680189Z self_outputs = self.self( 2025-12-04T08:46:58.8680630Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 612, in forward 2025-12-04T08:46:58.8681130Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-12-04T08:46:58.8681697Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 877, in _sliding_chunks_matmul_attn_probs_value 2025-12-04T08:46:58.8682341Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-12-04T08:46:58.8682579Z 2025-12-04T08:46:58.8682711Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.8683337Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.8683879Z layer_outputs = layer_module( 2025-12-04T08:46:58.8684269Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.8684682Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.8685145Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.8685621Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.8686091Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.8686556Z self_outputs = self.self( 2025-12-04T08:46:58.8687000Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 617, in forward 2025-12-04T08:46:58.8687627Z attn_output = attn_output.transpose(0, 1).reshape(seq_len, batch_size, embed_dim).contiguous() 2025-12-04T08:46:58.8687908Z 2025-12-04T08:46:58.8688025Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.8688610Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.8689145Z layer_outputs = layer_module( 2025-12-04T08:46:58.8689536Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.8689944Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.8690414Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.8690876Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.8691349Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1143, in forward 2025-12-04T08:46:58.8691860Z attn_output = self.output(self_outputs[0], hidden_states) 2025-12-04T08:46:58.8692358Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1093, in forward 2025-12-04T08:46:58.8692835Z hidden_states = self.dense(hidden_states) 2025-12-04T08:46:58.8692997Z 2025-12-04T08:46:58.8693115Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.8693689Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.8694230Z layer_outputs = layer_module( 2025-12-04T08:46:58.8694619Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.8695035Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.8695507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1210, in forward 2025-12-04T08:46:58.8695986Z layer_output = apply_chunking_to_forward( 2025-12-04T08:46:58.8696444Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:46:58.8696891Z return forward_fn(*input_tensors) 2025-12-04T08:46:58.8697363Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1217, in ff_chunk 2025-12-04T08:46:58.8697840Z intermediate_output = self.intermediate(attn_output) 2025-12-04T08:46:58.8698301Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1159, in forward 2025-12-04T08:46:58.8698760Z hidden_states = self.dense(hidden_states) 2025-12-04T08:46:58.8698914Z 2025-12-04T08:46:58.8699026Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.8699627Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.8700162Z layer_outputs = layer_module( 2025-12-04T08:46:58.8700544Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.8700939Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.8701397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1210, in forward 2025-12-04T08:46:58.8701870Z layer_output = apply_chunking_to_forward( 2025-12-04T08:46:58.8702312Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:46:58.8702796Z return forward_fn(*input_tensors) 2025-12-04T08:46:58.8703256Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1217, in ff_chunk 2025-12-04T08:46:58.8703757Z intermediate_output = self.intermediate(attn_output) 2025-12-04T08:46:58.8704256Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1160, in forward 2025-12-04T08:46:58.8704775Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:46:58.8705211Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T08:46:58.8705603Z return self.act(input) 2025-12-04T08:46:58.8705730Z 2025-12-04T08:46:58.8705855Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.8706421Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.8706958Z layer_outputs = layer_module( 2025-12-04T08:46:58.8707343Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.8707737Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.8708195Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1210, in forward 2025-12-04T08:46:58.8708667Z layer_output = apply_chunking_to_forward( 2025-12-04T08:46:58.8709104Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:46:58.8709536Z return forward_fn(*input_tensors) 2025-12-04T08:46:58.8709994Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-12-04T08:46:58.8710515Z layer_output = self.output(intermediate_output, attn_output) 2025-12-04T08:46:58.8711024Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1173, in forward 2025-12-04T08:46:58.8711490Z hidden_states = self.dense(hidden_states) 2025-12-04T08:46:58.8711647Z 2025-12-04T08:46:58.8711761Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.8712326Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.8712852Z layer_outputs = layer_module( 2025-12-04T08:46:58.8713232Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.8713634Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.8714129Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.8714585Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.8715107Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.8715557Z self_outputs = self.self( 2025-12-04T08:46:58.8715988Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 508, in forward 2025-12-04T08:46:58.8716452Z query_vectors = self.query(hidden_states) 2025-12-04T08:46:58.8716605Z 2025-12-04T08:46:58.8716718Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.8717276Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.8717798Z layer_outputs = layer_module( 2025-12-04T08:46:58.8718262Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.8718721Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.8719198Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.8719663Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.8720137Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.8720594Z self_outputs = self.self( 2025-12-04T08:46:58.8721218Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 523, in forward 2025-12-04T08:46:58.8721709Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-12-04T08:46:58.8722258Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 795, in _sliding_chunks_query_key_matmul 2025-12-04T08:46:58.8722905Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-12-04T08:46:58.8723175Z 2025-12-04T08:46:58.8723288Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.8723851Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.8724378Z layer_outputs = layer_module( 2025-12-04T08:46:58.8724758Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.8725147Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.8725605Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.8726065Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.8726519Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.8726969Z self_outputs = self.self( 2025-12-04T08:46:58.8727404Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 509, in forward 2025-12-04T08:46:58.8727858Z key_vectors = self.key(hidden_states) 2025-12-04T08:46:58.8728003Z 2025-12-04T08:46:58.8728123Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.8728674Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.8729196Z layer_outputs = layer_module( 2025-12-04T08:46:58.8729571Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.8729960Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.8730419Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.8730956Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.8731413Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.8731844Z self_outputs = self.self( 2025-12-04T08:46:58.8732254Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 523, in forward 2025-12-04T08:46:58.8732717Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-12-04T08:46:58.8733236Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 795, in _sliding_chunks_query_key_matmul 2025-12-04T08:46:58.8733833Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-12-04T08:46:58.8734156Z 2025-12-04T08:46:58.8734262Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.8734794Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.8735291Z layer_outputs = layer_module( 2025-12-04T08:46:58.8735639Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.8736010Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.8736438Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.8736862Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.8737286Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.8737725Z self_outputs = self.self( 2025-12-04T08:46:58.8738164Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 523, in forward 2025-12-04T08:46:58.8738638Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-12-04T08:46:58.8739185Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 795, in _sliding_chunks_query_key_matmul 2025-12-04T08:46:58.8739811Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-12-04T08:46:58.8740057Z 2025-12-04T08:46:58.8740171Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.8740695Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.8741197Z layer_outputs = layer_module( 2025-12-04T08:46:58.8741555Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.8741933Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.8742358Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.8742788Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.8743215Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.8743633Z self_outputs = self.self( 2025-12-04T08:46:58.8744050Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 523, in forward 2025-12-04T08:46:58.8744509Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-12-04T08:46:58.8745022Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 795, in _sliding_chunks_query_key_matmul 2025-12-04T08:46:58.8745648Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-12-04T08:46:58.8745908Z 2025-12-04T08:46:58.8745994Z cudagraph partition due to non gpu ops 2025-12-04T08:46:58.8746227Z cudagraph partition due to non gpu ops 2025-12-04T08:46:58.8746456Z cudagraph partition due to non gpu ops 2025-12-04T08:46:58.8746685Z cudagraph partition due to non gpu ops 2025-12-04T08:46:58.8746936Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.8747493Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.8748014Z layer_outputs = layer_module( 2025-12-04T08:46:58.8748391Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.8748788Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.8749284Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.8749739Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.8750199Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.8750653Z self_outputs = self.self( 2025-12-04T08:46:58.8751087Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 535, in forward 2025-12-04T08:46:58.8751583Z diagonal_mask = self._sliding_chunks_query_key_matmul( 2025-12-04T08:46:58.8752147Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 833, in _sliding_chunks_query_key_matmul 2025-12-04T08:46:58.8752748Z self._mask_invalid_locations(diagonal_attention_scores, window_overlap) 2025-12-04T08:46:58.8753329Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 763, in _mask_invalid_locations 2025-12-04T08:46:58.8753856Z ).where(beginning_mask.bool(), beginning_input) 2025-12-04T08:46:58.8754026Z 2025-12-04T08:46:58.8754114Z cudagraph partition due to non gpu ops 2025-12-04T08:46:58.8754377Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.8754938Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.8755471Z layer_outputs = layer_module( 2025-12-04T08:46:58.8755851Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.8756252Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.8756702Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.8757163Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.8757623Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.8758170Z self_outputs = self.self( 2025-12-04T08:46:58.8758783Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 540, in forward 2025-12-04T08:46:58.8759587Z attn_scores += diagonal_mask 2025-12-04T08:46:58.8759747Z 2025-12-04T08:46:58.8759884Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.8760529Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.8761176Z layer_outputs = layer_module( 2025-12-04T08:46:58.8761630Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.8762127Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.8762688Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.8763219Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.8763751Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.8764249Z self_outputs = self.self( 2025-12-04T08:46:58.8764760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 578, in forward 2025-12-04T08:46:58.8765273Z attn_probs = nn.functional.softmax( 2025-12-04T08:46:58.8765444Z 2025-12-04T08:46:58.8765605Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.8766214Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.8766787Z layer_outputs = layer_module( 2025-12-04T08:46:58.8767201Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.8787330Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.8787846Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.8788342Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.8788812Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.8789289Z self_outputs = self.self( 2025-12-04T08:46:58.8789744Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 510, in forward 2025-12-04T08:46:58.8790238Z value_vectors = self.value(hidden_states) 2025-12-04T08:46:58.8790415Z 2025-12-04T08:46:58.8790541Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.8791123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.8791666Z layer_outputs = layer_module( 2025-12-04T08:46:58.8792055Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.8792471Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.8792940Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.8793408Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.8793862Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.8794331Z self_outputs = self.self( 2025-12-04T08:46:58.8794776Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 612, in forward 2025-12-04T08:46:58.8795288Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-12-04T08:46:58.8795869Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 862, in _sliding_chunks_matmul_attn_probs_value 2025-12-04T08:46:58.8796523Z padded_value = nn.functional.pad(value, (0, 0, window_overlap, window_overlap), value=-1) 2025-12-04T08:46:58.8796999Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/functional.py", line 5418, in pad 2025-12-04T08:46:58.8797410Z return torch._C._nn.pad(input, pad, mode, value) 2025-12-04T08:46:58.8797586Z 2025-12-04T08:46:58.8797706Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.8798507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.8799077Z layer_outputs = layer_module( 2025-12-04T08:46:58.8799476Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.8799907Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.8800376Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.8800844Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.8801299Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.8801759Z self_outputs = self.self( 2025-12-04T08:46:58.8802252Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 612, in forward 2025-12-04T08:46:58.8802762Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-12-04T08:46:58.8803332Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 875, in _sliding_chunks_matmul_attn_probs_value 2025-12-04T08:46:58.8803958Z chunked_attn_probs = self._pad_and_diagonalize(chunked_attn_probs) 2025-12-04T08:46:58.8804530Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 698, in _pad_and_diagonalize 2025-12-04T08:46:58.8805048Z chunked_hidden_states = nn.functional.pad( 2025-12-04T08:46:58.8805418Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/functional.py", line 5418, in pad 2025-12-04T08:46:58.8805804Z return torch._C._nn.pad(input, pad, mode, value) 2025-12-04T08:46:58.8805975Z 2025-12-04T08:46:58.8806101Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.8806665Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.8807226Z layer_outputs = layer_module( 2025-12-04T08:46:58.8807619Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.8808018Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.8808467Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.8808928Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.8809381Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.8809833Z self_outputs = self.self( 2025-12-04T08:46:58.8810273Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 612, in forward 2025-12-04T08:46:58.8810773Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-12-04T08:46:58.8811350Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 877, in _sliding_chunks_matmul_attn_probs_value 2025-12-04T08:46:58.8811970Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-12-04T08:46:58.8812212Z 2025-12-04T08:46:58.8812343Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.8812909Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.8813444Z layer_outputs = layer_module( 2025-12-04T08:46:58.8813816Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.8814255Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.8814717Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.8815178Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.8815633Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.8816084Z self_outputs = self.self( 2025-12-04T08:46:58.8816523Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 612, in forward 2025-12-04T08:46:58.8817024Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-12-04T08:46:58.8817592Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 877, in _sliding_chunks_matmul_attn_probs_value 2025-12-04T08:46:58.8818264Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-12-04T08:46:58.8818494Z 2025-12-04T08:46:58.8818622Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.8819194Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.8819731Z layer_outputs = layer_module( 2025-12-04T08:46:58.8820113Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.8820513Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.8821201Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.8821669Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.8822165Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.8822624Z self_outputs = self.self( 2025-12-04T08:46:58.8823055Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 617, in forward 2025-12-04T08:46:58.8823640Z attn_output = attn_output.transpose(0, 1).reshape(seq_len, batch_size, embed_dim).contiguous() 2025-12-04T08:46:58.8823912Z 2025-12-04T08:46:58.8824029Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.8824595Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.8825119Z layer_outputs = layer_module( 2025-12-04T08:46:58.8825507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.8825915Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.8826382Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.8826836Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.8827297Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1143, in forward 2025-12-04T08:46:58.8827792Z attn_output = self.output(self_outputs[0], hidden_states) 2025-12-04T08:46:58.8828280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1093, in forward 2025-12-04T08:46:58.8828751Z hidden_states = self.dense(hidden_states) 2025-12-04T08:46:58.8828915Z 2025-12-04T08:46:58.8829031Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.8829596Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.8830231Z layer_outputs = layer_module( 2025-12-04T08:46:58.8830623Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.8831000Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.8831435Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1210, in forward 2025-12-04T08:46:58.8831871Z layer_output = apply_chunking_to_forward( 2025-12-04T08:46:58.8832304Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:46:58.8832723Z return forward_fn(*input_tensors) 2025-12-04T08:46:58.8833151Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1217, in ff_chunk 2025-12-04T08:46:58.8833684Z intermediate_output = self.intermediate(attn_output) 2025-12-04T08:46:58.8834165Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1159, in forward 2025-12-04T08:46:58.8834616Z hidden_states = self.dense(hidden_states) 2025-12-04T08:46:58.8834759Z 2025-12-04T08:46:58.8834870Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.8835415Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.8835931Z layer_outputs = layer_module( 2025-12-04T08:46:58.8836288Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.8836670Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.8837113Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1210, in forward 2025-12-04T08:46:58.8837601Z layer_output = apply_chunking_to_forward( 2025-12-04T08:46:58.8838048Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:46:58.8838560Z return forward_fn(*input_tensors) 2025-12-04T08:46:58.8839023Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1217, in ff_chunk 2025-12-04T08:46:58.8839528Z intermediate_output = self.intermediate(attn_output) 2025-12-04T08:46:58.8840015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1160, in forward 2025-12-04T08:46:58.8840518Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:46:58.8840945Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T08:46:58.8841330Z return self.act(input) 2025-12-04T08:46:58.8841463Z 2025-12-04T08:46:58.8841582Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.8842161Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.8842709Z layer_outputs = layer_module( 2025-12-04T08:46:58.8843067Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.8843484Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.8843946Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1210, in forward 2025-12-04T08:46:58.8844415Z layer_output = apply_chunking_to_forward( 2025-12-04T08:46:58.8844852Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:46:58.8845283Z return forward_fn(*input_tensors) 2025-12-04T08:46:58.8845751Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-12-04T08:46:58.8846226Z layer_output = self.output(intermediate_output, attn_output) 2025-12-04T08:46:58.8846705Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1173, in forward 2025-12-04T08:46:58.8847162Z hidden_states = self.dense(hidden_states) 2025-12-04T08:46:58.8847311Z 2025-12-04T08:46:58.8847432Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.8847987Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.8848514Z layer_outputs = layer_module( 2025-12-04T08:46:58.8848890Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.8849325Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.8849781Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.8850246Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.8850711Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.8851165Z self_outputs = self.self( 2025-12-04T08:46:58.8851603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 508, in forward 2025-12-04T08:46:58.8852071Z query_vectors = self.query(hidden_states) 2025-12-04T08:46:58.8852225Z 2025-12-04T08:46:58.8852352Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.8852921Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.8853462Z layer_outputs = layer_module( 2025-12-04T08:46:58.8853851Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.8854254Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.8854714Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.8855179Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.8855639Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.8856096Z self_outputs = self.self( 2025-12-04T08:46:58.8856532Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 523, in forward 2025-12-04T08:46:58.8857028Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-12-04T08:46:58.8857591Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 795, in _sliding_chunks_query_key_matmul 2025-12-04T08:46:58.8858242Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-12-04T08:46:58.8858513Z 2025-12-04T08:46:58.8858630Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.8859201Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.8859737Z layer_outputs = layer_module( 2025-12-04T08:46:58.8860116Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.8860520Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.8861021Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.8861480Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.8861932Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.8862382Z self_outputs = self.self( 2025-12-04T08:46:58.8862824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 509, in forward 2025-12-04T08:46:58.8863290Z key_vectors = self.key(hidden_states) 2025-12-04T08:46:58.8863436Z 2025-12-04T08:46:58.8863548Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.8864103Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.8864641Z layer_outputs = layer_module( 2025-12-04T08:46:58.8864999Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.8865378Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.8865807Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.8866236Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.8866661Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.8867085Z self_outputs = self.self( 2025-12-04T08:46:58.8867497Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 523, in forward 2025-12-04T08:46:58.8867951Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-12-04T08:46:58.8868469Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 795, in _sliding_chunks_query_key_matmul 2025-12-04T08:46:58.8869077Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-12-04T08:46:58.8869327Z 2025-12-04T08:46:58.8869441Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.8869975Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.8870466Z layer_outputs = layer_module( 2025-12-04T08:46:58.8870824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.8871201Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.8871627Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.8872064Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.8872497Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.8872925Z self_outputs = self.self( 2025-12-04T08:46:58.8873332Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 523, in forward 2025-12-04T08:46:58.8873789Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-12-04T08:46:58.8874313Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 795, in _sliding_chunks_query_key_matmul 2025-12-04T08:46:58.8874949Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-12-04T08:46:58.8875213Z 2025-12-04T08:46:58.8875326Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.8875930Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.8876460Z layer_outputs = layer_module( 2025-12-04T08:46:58.8876839Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.8877230Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.8877685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.8878220Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.8878700Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.8879165Z self_outputs = self.self( 2025-12-04T08:46:58.8879616Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 523, in forward 2025-12-04T08:46:58.8880180Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-12-04T08:46:58.8880727Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 795, in _sliding_chunks_query_key_matmul 2025-12-04T08:46:58.8881376Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-12-04T08:46:58.8881650Z 2025-12-04T08:46:58.8881740Z cudagraph partition due to non gpu ops 2025-12-04T08:46:58.8881978Z cudagraph partition due to non gpu ops 2025-12-04T08:46:58.8882203Z cudagraph partition due to non gpu ops 2025-12-04T08:46:58.8882429Z cudagraph partition due to non gpu ops 2025-12-04T08:46:58.8882684Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.8883239Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.8883782Z layer_outputs = layer_module( 2025-12-04T08:46:58.8884169Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.8884572Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.8885025Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.8885488Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.8885947Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.8886398Z self_outputs = self.self( 2025-12-04T08:46:58.8886830Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 535, in forward 2025-12-04T08:46:58.8887325Z diagonal_mask = self._sliding_chunks_query_key_matmul( 2025-12-04T08:46:58.8887895Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 833, in _sliding_chunks_query_key_matmul 2025-12-04T08:46:58.8888492Z self._mask_invalid_locations(diagonal_attention_scores, window_overlap) 2025-12-04T08:46:58.8889068Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 763, in _mask_invalid_locations 2025-12-04T08:46:58.8889589Z ).where(beginning_mask.bool(), beginning_input) 2025-12-04T08:46:58.8889753Z 2025-12-04T08:46:58.8889848Z cudagraph partition due to non gpu ops 2025-12-04T08:46:58.8890097Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.8890665Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.8891192Z layer_outputs = layer_module( 2025-12-04T08:46:58.8891613Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.8892006Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.8892470Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.8892926Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.8893244Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.8893320Z self_outputs = self.self( 2025-12-04T08:46:58.8893625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 540, in forward 2025-12-04T08:46:58.8893712Z attn_scores += diagonal_mask 2025-12-04T08:46:58.8893716Z 2025-12-04T08:46:58.8893826Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.8894266Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.8894349Z layer_outputs = layer_module( 2025-12-04T08:46:58.8894592Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.8894685Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.8895003Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.8895085Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.8895395Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.8895469Z self_outputs = self.self( 2025-12-04T08:46:58.8895780Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 578, in forward 2025-12-04T08:46:58.8895874Z attn_probs = nn.functional.softmax( 2025-12-04T08:46:58.8895878Z 2025-12-04T08:46:58.8895990Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.8896392Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.8896469Z layer_outputs = layer_module( 2025-12-04T08:46:58.8896712Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.8896806Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.8897118Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.8897203Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.8897494Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.8897566Z self_outputs = self.self( 2025-12-04T08:46:58.8897869Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 510, in forward 2025-12-04T08:46:58.8897957Z value_vectors = self.value(hidden_states) 2025-12-04T08:46:58.8897961Z 2025-12-04T08:46:58.8898072Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.8898450Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.8898524Z layer_outputs = layer_module( 2025-12-04T08:46:58.8898759Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.8898840Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.8899178Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.8899257Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.8899552Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.8899631Z self_outputs = self.self( 2025-12-04T08:46:58.8899918Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 612, in forward 2025-12-04T08:46:58.8900044Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-12-04T08:46:58.8900421Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 862, in _sliding_chunks_matmul_attn_probs_value 2025-12-04T08:46:58.8900602Z padded_value = nn.functional.pad(value, (0, 0, window_overlap, window_overlap), value=-1) 2025-12-04T08:46:58.8900845Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/functional.py", line 5418, in pad 2025-12-04T08:46:58.8900948Z return torch._C._nn.pad(input, pad, mode, value) 2025-12-04T08:46:58.8900951Z 2025-12-04T08:46:58.8901057Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.8901429Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.8901503Z layer_outputs = layer_module( 2025-12-04T08:46:58.8901741Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.8901820Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.8902115Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.8902203Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.8902493Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.8902571Z self_outputs = self.self( 2025-12-04T08:46:58.8902862Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 612, in forward 2025-12-04T08:46:58.8902984Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-12-04T08:46:58.8903360Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 875, in _sliding_chunks_matmul_attn_probs_value 2025-12-04T08:46:58.8903499Z chunked_attn_probs = self._pad_and_diagonalize(chunked_attn_probs) 2025-12-04T08:46:58.8903834Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 698, in _pad_and_diagonalize 2025-12-04T08:46:58.8903934Z chunked_hidden_states = nn.functional.pad( 2025-12-04T08:46:58.8904136Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/functional.py", line 5418, in pad 2025-12-04T08:46:58.8904244Z return torch._C._nn.pad(input, pad, mode, value) 2025-12-04T08:46:58.8904247Z 2025-12-04T08:46:58.8904351Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.8904716Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.8904797Z layer_outputs = layer_module( 2025-12-04T08:46:58.8905024Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.8905111Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.8905403Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.8905486Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.8905828Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.8905907Z self_outputs = self.self( 2025-12-04T08:46:58.8906226Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 612, in forward 2025-12-04T08:46:58.8906345Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-12-04T08:46:58.8906717Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 877, in _sliding_chunks_matmul_attn_probs_value 2025-12-04T08:46:58.8906889Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-12-04T08:46:58.8906893Z 2025-12-04T08:46:58.8906997Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.8907408Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.8907485Z layer_outputs = layer_module( 2025-12-04T08:46:58.8907723Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.8907813Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.8908116Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.8908196Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.8908508Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.8908582Z self_outputs = self.self( 2025-12-04T08:46:58.8908890Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 612, in forward 2025-12-04T08:46:58.8909017Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-12-04T08:46:58.8909408Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 877, in _sliding_chunks_matmul_attn_probs_value 2025-12-04T08:46:58.8909582Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-12-04T08:46:58.8909586Z 2025-12-04T08:46:58.8909696Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.8910096Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.8910172Z layer_outputs = layer_module( 2025-12-04T08:46:58.8910419Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.8910514Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.8910829Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.8910918Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.8911229Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.8911299Z self_outputs = self.self( 2025-12-04T08:46:58.8911596Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 617, in forward 2025-12-04T08:46:58.8911789Z attn_output = attn_output.transpose(0, 1).reshape(seq_len, batch_size, embed_dim).contiguous() 2025-12-04T08:46:58.8911792Z 2025-12-04T08:46:58.8911902Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.8912266Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.8912380Z layer_outputs = layer_module( 2025-12-04T08:46:58.8912618Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.8912698Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.8912989Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.8913073Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.8913362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1143, in forward 2025-12-04T08:46:58.8913482Z attn_output = self.output(self_outputs[0], hidden_states) 2025-12-04T08:46:58.8913771Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1093, in forward 2025-12-04T08:46:58.8913889Z hidden_states = self.dense(hidden_states) 2025-12-04T08:46:58.8913893Z 2025-12-04T08:46:58.8914006Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.8914374Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.8914454Z layer_outputs = layer_module( 2025-12-04T08:46:58.8914684Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.8914762Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.8915065Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1210, in forward 2025-12-04T08:46:58.8915150Z layer_output = apply_chunking_to_forward( 2025-12-04T08:46:58.8915433Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:46:58.8915519Z return forward_fn(*input_tensors) 2025-12-04T08:46:58.8915814Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1217, in ff_chunk 2025-12-04T08:46:58.8915937Z intermediate_output = self.intermediate(attn_output) 2025-12-04T08:46:58.8916234Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1159, in forward 2025-12-04T08:46:58.8916317Z hidden_states = self.dense(hidden_states) 2025-12-04T08:46:58.8916328Z 2025-12-04T08:46:58.8916430Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.8916798Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.8916877Z layer_outputs = layer_module( 2025-12-04T08:46:58.8917113Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.8917193Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.8917497Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1210, in forward 2025-12-04T08:46:58.8917582Z layer_output = apply_chunking_to_forward( 2025-12-04T08:46:58.8917866Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:46:58.8917945Z return forward_fn(*input_tensors) 2025-12-04T08:46:58.8918314Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1217, in ff_chunk 2025-12-04T08:46:58.8918444Z intermediate_output = self.intermediate(attn_output) 2025-12-04T08:46:58.8918757Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1160, in forward 2025-12-04T08:46:58.8918930Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:46:58.8919175Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T08:46:58.8919258Z return self.act(input) 2025-12-04T08:46:58.8919263Z 2025-12-04T08:46:58.8919385Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.8919793Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.8919885Z layer_outputs = layer_module( 2025-12-04T08:46:58.8920136Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.8920223Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.8920547Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1210, in forward 2025-12-04T08:46:58.8921066Z layer_output = apply_chunking_to_forward( 2025-12-04T08:46:58.8921346Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:46:58.8921435Z return forward_fn(*input_tensors) 2025-12-04T08:46:58.8921730Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-12-04T08:46:58.8921864Z layer_output = self.output(intermediate_output, attn_output) 2025-12-04T08:46:58.8922153Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1173, in forward 2025-12-04T08:46:58.8922238Z hidden_states = self.dense(hidden_states) 2025-12-04T08:46:58.8922242Z 2025-12-04T08:46:58.8922357Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.8922739Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.8922827Z layer_outputs = layer_module( 2025-12-04T08:46:58.8923069Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.8923154Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.8923473Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.8923556Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.8923869Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.8923955Z self_outputs = self.self( 2025-12-04T08:46:58.8924258Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 508, in forward 2025-12-04T08:46:58.8924361Z query_vectors = self.query(hidden_states) 2025-12-04T08:46:58.8924365Z 2025-12-04T08:46:58.8924475Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.8924868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.8924955Z layer_outputs = layer_module( 2025-12-04T08:46:58.8925196Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.8925284Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.8925573Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.8925650Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.8926022Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.8926096Z self_outputs = self.self( 2025-12-04T08:46:58.8926405Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 523, in forward 2025-12-04T08:46:58.8926525Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-12-04T08:46:58.8926900Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 795, in _sliding_chunks_query_key_matmul 2025-12-04T08:46:58.8927111Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-12-04T08:46:58.8927115Z 2025-12-04T08:46:58.8927226Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.8927612Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.8927745Z layer_outputs = layer_module( 2025-12-04T08:46:58.8927989Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.8928081Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.8928388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.8928469Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.8928795Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.8928871Z self_outputs = self.self( 2025-12-04T08:46:58.8929186Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 509, in forward 2025-12-04T08:46:58.8929275Z key_vectors = self.key(hidden_states) 2025-12-04T08:46:58.8929279Z 2025-12-04T08:46:58.8929400Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.8929786Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.8929863Z layer_outputs = layer_module( 2025-12-04T08:46:58.8930114Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.8930196Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.8930500Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.8930588Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.8930890Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.8930975Z self_outputs = self.self( 2025-12-04T08:46:58.8931283Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 523, in forward 2025-12-04T08:46:58.8931395Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-12-04T08:46:58.8931779Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 795, in _sliding_chunks_query_key_matmul 2025-12-04T08:46:58.8931981Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-12-04T08:46:58.8931985Z 2025-12-04T08:46:58.8932101Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.8932488Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.8932566Z layer_outputs = layer_module( 2025-12-04T08:46:58.8932820Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.8932959Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.8933273Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.8933354Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.8933657Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.8933739Z self_outputs = self.self( 2025-12-04T08:46:58.8934045Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 523, in forward 2025-12-04T08:46:58.8934146Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-12-04T08:46:58.8934511Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 795, in _sliding_chunks_query_key_matmul 2025-12-04T08:46:58.8934733Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-12-04T08:46:58.8934736Z 2025-12-04T08:46:58.8934846Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.8935200Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.8935271Z layer_outputs = layer_module( 2025-12-04T08:46:58.8935499Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.8935575Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.8935863Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.8935941Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.8936221Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.8936296Z self_outputs = self.self( 2025-12-04T08:46:58.8936574Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 523, in forward 2025-12-04T08:46:58.8936677Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-12-04T08:46:58.8937020Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 795, in _sliding_chunks_query_key_matmul 2025-12-04T08:46:58.8937199Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-12-04T08:46:58.8937202Z 2025-12-04T08:46:58.8937291Z cudagraph partition due to non gpu ops 2025-12-04T08:46:58.8937372Z cudagraph partition due to non gpu ops 2025-12-04T08:46:58.8937467Z cudagraph partition due to non gpu ops 2025-12-04T08:46:58.8937544Z cudagraph partition due to non gpu ops 2025-12-04T08:46:58.8937652Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.8938031Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.8938103Z layer_outputs = layer_module( 2025-12-04T08:46:58.8938331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.8938418Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.8938707Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.8938789Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.8939079Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.8939151Z self_outputs = self.self( 2025-12-04T08:46:58.8939488Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 535, in forward 2025-12-04T08:46:58.8939601Z diagonal_mask = self._sliding_chunks_query_key_matmul( 2025-12-04T08:46:58.8939952Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 833, in _sliding_chunks_query_key_matmul 2025-12-04T08:46:58.8940108Z self._mask_invalid_locations(diagonal_attention_scores, window_overlap) 2025-12-04T08:46:58.8940443Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 763, in _mask_invalid_locations 2025-12-04T08:46:58.8940549Z ).where(beginning_mask.bool(), beginning_input) 2025-12-04T08:46:58.8940553Z 2025-12-04T08:46:58.8940632Z cudagraph partition due to non gpu ops 2025-12-04T08:46:58.8940773Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.8941140Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.8941211Z layer_outputs = layer_module( 2025-12-04T08:46:58.8941443Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.8941521Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.8941809Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.8941893Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.8942179Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.8942254Z self_outputs = self.self( 2025-12-04T08:46:58.8942541Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 540, in forward 2025-12-04T08:46:58.8942616Z attn_scores += diagonal_mask 2025-12-04T08:46:58.8942619Z 2025-12-04T08:46:58.8942728Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.8943091Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.8943168Z layer_outputs = layer_module( 2025-12-04T08:46:58.8943389Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.8943465Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.8943760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.8943839Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.8944123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.8944202Z self_outputs = self.self( 2025-12-04T08:46:58.8944483Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 578, in forward 2025-12-04T08:46:58.8944570Z attn_probs = nn.functional.softmax( 2025-12-04T08:46:58.8944574Z 2025-12-04T08:46:58.8944675Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.8945032Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.8945113Z layer_outputs = layer_module( 2025-12-04T08:46:58.8945335Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.8945422Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.8945736Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.8945812Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.8946099Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.8946167Z self_outputs = self.self( 2025-12-04T08:46:58.8946448Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 510, in forward 2025-12-04T08:46:58.8946542Z value_vectors = self.value(hidden_states) 2025-12-04T08:46:58.8946545Z 2025-12-04T08:46:58.8946647Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.8947018Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.8947120Z layer_outputs = layer_module( 2025-12-04T08:46:58.8947350Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.8947437Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.8947726Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.8947817Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.8948098Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.8948167Z self_outputs = self.self( 2025-12-04T08:46:58.8948454Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 612, in forward 2025-12-04T08:46:58.8948578Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-12-04T08:46:58.8948949Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 862, in _sliding_chunks_matmul_attn_probs_value 2025-12-04T08:46:58.8949126Z padded_value = nn.functional.pad(value, (0, 0, window_overlap, window_overlap), value=-1) 2025-12-04T08:46:58.8949328Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/functional.py", line 5418, in pad 2025-12-04T08:46:58.8949437Z return torch._C._nn.pad(input, pad, mode, value) 2025-12-04T08:46:58.8949440Z 2025-12-04T08:46:58.8949542Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.8949905Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.8949985Z layer_outputs = layer_module( 2025-12-04T08:46:58.8950216Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.8950307Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.8950596Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.8950673Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.8950975Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.8951044Z self_outputs = self.self( 2025-12-04T08:46:58.8951330Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 612, in forward 2025-12-04T08:46:58.8951449Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-12-04T08:46:58.8951812Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 875, in _sliding_chunks_matmul_attn_probs_value 2025-12-04T08:46:58.8951994Z chunked_attn_probs = self._pad_and_diagonalize(chunked_attn_probs) 2025-12-04T08:46:58.8952327Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 698, in _pad_and_diagonalize 2025-12-04T08:46:58.8952428Z chunked_hidden_states = nn.functional.pad( 2025-12-04T08:46:58.8952633Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/functional.py", line 5418, in pad 2025-12-04T08:46:58.8952731Z return torch._C._nn.pad(input, pad, mode, value) 2025-12-04T08:46:58.8952735Z 2025-12-04T08:46:58.8952844Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.8953208Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.8953281Z layer_outputs = layer_module( 2025-12-04T08:46:58.8953568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.8953655Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.8953971Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.8954051Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.8954357Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.8954439Z self_outputs = self.self( 2025-12-04T08:46:58.8954743Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 612, in forward 2025-12-04T08:46:58.8954875Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-12-04T08:46:58.8955263Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 877, in _sliding_chunks_matmul_attn_probs_value 2025-12-04T08:46:58.8955438Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-12-04T08:46:58.8955442Z 2025-12-04T08:46:58.8955560Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.8955944Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.8956029Z layer_outputs = layer_module( 2025-12-04T08:46:58.8956269Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.8956353Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.8956666Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.8956747Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.8957061Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.8957143Z self_outputs = self.self( 2025-12-04T08:46:58.8957445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 612, in forward 2025-12-04T08:46:58.8957576Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-12-04T08:46:58.8958003Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 877, in _sliding_chunks_matmul_attn_probs_value 2025-12-04T08:46:58.8958253Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-12-04T08:46:58.8958268Z 2025-12-04T08:46:58.8958385Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.8958793Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.8958924Z layer_outputs = layer_module( 2025-12-04T08:46:58.8959174Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.8959261Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.8959597Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.8959682Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.8960000Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.8960071Z self_outputs = self.self( 2025-12-04T08:46:58.8960358Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 617, in forward 2025-12-04T08:46:58.8960602Z attn_output = attn_output.transpose(0, 1).reshape(seq_len, batch_size, embed_dim).contiguous() 2025-12-04T08:46:58.8960609Z 2025-12-04T08:46:58.8960713Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.8961084Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.8961158Z layer_outputs = layer_module( 2025-12-04T08:46:58.8961386Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.8961474Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.8961764Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.8961840Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.8962139Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1143, in forward 2025-12-04T08:46:58.8962260Z attn_output = self.output(self_outputs[0], hidden_states) 2025-12-04T08:46:58.8962559Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1093, in forward 2025-12-04T08:46:58.8962644Z hidden_states = self.dense(hidden_states) 2025-12-04T08:46:58.8962647Z 2025-12-04T08:46:58.8962749Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.8963120Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.8963193Z layer_outputs = layer_module( 2025-12-04T08:46:58.8963427Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.8963506Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.8963801Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1210, in forward 2025-12-04T08:46:58.8963894Z layer_output = apply_chunking_to_forward( 2025-12-04T08:46:58.8964164Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:46:58.8964249Z return forward_fn(*input_tensors) 2025-12-04T08:46:58.8964547Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1217, in ff_chunk 2025-12-04T08:46:58.8964657Z intermediate_output = self.intermediate(attn_output) 2025-12-04T08:46:58.8964945Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1159, in forward 2025-12-04T08:46:58.8965028Z hidden_states = self.dense(hidden_states) 2025-12-04T08:46:58.8965032Z 2025-12-04T08:46:58.8965140Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.8965526Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.8965599Z layer_outputs = layer_module( 2025-12-04T08:46:58.8965828Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.8965906Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.8966191Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1210, in forward 2025-12-04T08:46:58.8966282Z layer_output = apply_chunking_to_forward( 2025-12-04T08:46:58.8966545Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:46:58.8966627Z return forward_fn(*input_tensors) 2025-12-04T08:46:58.8966946Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1217, in ff_chunk 2025-12-04T08:46:58.8967056Z intermediate_output = self.intermediate(attn_output) 2025-12-04T08:46:58.8967344Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1160, in forward 2025-12-04T08:46:58.8967456Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:46:58.8967682Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T08:46:58.8967755Z return self.act(input) 2025-12-04T08:46:58.8967758Z 2025-12-04T08:46:58.8967858Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.8968220Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.8968295Z layer_outputs = layer_module( 2025-12-04T08:46:58.8968519Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.8968605Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.8968889Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1210, in forward 2025-12-04T08:46:58.8968978Z layer_output = apply_chunking_to_forward( 2025-12-04T08:46:58.8969239Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:46:58.8969315Z return forward_fn(*input_tensors) 2025-12-04T08:46:58.8969604Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-12-04T08:46:58.8969729Z layer_output = self.output(intermediate_output, attn_output) 2025-12-04T08:46:58.8970035Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1173, in forward 2025-12-04T08:46:58.8970117Z hidden_states = self.dense(hidden_states) 2025-12-04T08:46:58.8970120Z 2025-12-04T08:46:58.8970222Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.8970584Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.8970655Z layer_outputs = layer_module( 2025-12-04T08:46:58.8970882Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.8970959Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.8971240Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.8971326Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.8971637Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.8971708Z self_outputs = self.self( 2025-12-04T08:46:58.8971993Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 508, in forward 2025-12-04T08:46:58.8972075Z query_vectors = self.query(hidden_states) 2025-12-04T08:46:58.8972078Z 2025-12-04T08:46:58.8972184Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.8972531Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.8972600Z layer_outputs = layer_module( 2025-12-04T08:46:58.8972834Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.8972942Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.8973230Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.8973306Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.8973585Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.8973662Z self_outputs = self.self( 2025-12-04T08:46:58.8973938Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 523, in forward 2025-12-04T08:46:58.8974039Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-12-04T08:46:58.8974386Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 795, in _sliding_chunks_query_key_matmul 2025-12-04T08:46:58.8974572Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-12-04T08:46:58.8974576Z 2025-12-04T08:46:58.8974684Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.8975038Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.8975114Z layer_outputs = layer_module( 2025-12-04T08:46:58.8975333Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.8975408Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.8975692Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.8975766Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.8976045Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.8976126Z self_outputs = self.self( 2025-12-04T08:46:58.8976406Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 509, in forward 2025-12-04T08:46:58.8976492Z key_vectors = self.key(hidden_states) 2025-12-04T08:46:58.8976495Z 2025-12-04T08:46:58.8976598Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.8976947Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.8977027Z layer_outputs = layer_module( 2025-12-04T08:46:58.8977246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.8977328Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.8977608Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.8977719Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.8978008Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.8978077Z self_outputs = self.self( 2025-12-04T08:46:58.8978357Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 523, in forward 2025-12-04T08:46:58.8978468Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-12-04T08:46:58.8978811Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 795, in _sliding_chunks_query_key_matmul 2025-12-04T08:46:58.8979000Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-12-04T08:46:58.8979004Z 2025-12-04T08:46:58.8979138Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.8979497Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.8979578Z layer_outputs = layer_module( 2025-12-04T08:46:58.8979800Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.8979888Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.8980174Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.8980251Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.8980542Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.8980612Z self_outputs = self.self( 2025-12-04T08:46:58.8980907Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 523, in forward 2025-12-04T08:46:58.8981012Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-12-04T08:46:58.8981355Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 795, in _sliding_chunks_query_key_matmul 2025-12-04T08:46:58.8981546Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-12-04T08:46:58.8981550Z 2025-12-04T08:46:58.8981653Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.8982015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.8982090Z layer_outputs = layer_module( 2025-12-04T08:46:58.8982314Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.8982403Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.8982688Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.8982767Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.8983056Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.8983127Z self_outputs = self.self( 2025-12-04T08:46:58.8983426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 523, in forward 2025-12-04T08:46:58.8983529Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-12-04T08:46:58.8983872Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 795, in _sliding_chunks_query_key_matmul 2025-12-04T08:46:58.8984065Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-12-04T08:46:58.8984099Z 2025-12-04T08:46:58.8984183Z cudagraph partition due to non gpu ops 2025-12-04T08:46:58.8984270Z cudagraph partition due to non gpu ops 2025-12-04T08:46:58.8984348Z cudagraph partition due to non gpu ops 2025-12-04T08:46:58.8984426Z cudagraph partition due to non gpu ops 2025-12-04T08:46:58.8984534Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.8984891Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.8984962Z layer_outputs = layer_module( 2025-12-04T08:46:58.8985190Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.8985267Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.8985594Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.8985669Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.8985947Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.8986024Z self_outputs = self.self( 2025-12-04T08:46:58.8986302Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 535, in forward 2025-12-04T08:46:58.8986416Z diagonal_mask = self._sliding_chunks_query_key_matmul( 2025-12-04T08:46:58.8986754Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 833, in _sliding_chunks_query_key_matmul 2025-12-04T08:46:58.8986896Z self._mask_invalid_locations(diagonal_attention_scores, window_overlap) 2025-12-04T08:46:58.8987231Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 763, in _mask_invalid_locations 2025-12-04T08:46:58.8987327Z ).where(beginning_mask.bool(), beginning_input) 2025-12-04T08:46:58.8987331Z 2025-12-04T08:46:58.8987414Z cudagraph partition due to non gpu ops 2025-12-04T08:46:58.8987515Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.8987864Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.8987941Z layer_outputs = layer_module( 2025-12-04T08:46:58.8988159Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.8988235Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.8988526Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.8988604Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.8988892Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.8988962Z self_outputs = self.self( 2025-12-04T08:46:58.8989249Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 540, in forward 2025-12-04T08:46:58.8989330Z attn_scores += diagonal_mask 2025-12-04T08:46:58.8989333Z 2025-12-04T08:46:58.8989432Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.8989790Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.8989860Z layer_outputs = layer_module( 2025-12-04T08:46:58.8990081Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.8990170Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.8990491Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.8990567Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.8990857Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.8990928Z self_outputs = self.self( 2025-12-04T08:46:58.8991215Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 578, in forward 2025-12-04T08:46:58.8991294Z attn_probs = nn.functional.softmax( 2025-12-04T08:46:58.8991297Z 2025-12-04T08:46:58.8991397Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.8991760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.8991864Z layer_outputs = layer_module( 2025-12-04T08:46:58.8992099Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.8992177Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.8992461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.8992543Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.8992825Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.8992902Z self_outputs = self.self( 2025-12-04T08:46:58.8993191Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 510, in forward 2025-12-04T08:46:58.8993279Z value_vectors = self.value(hidden_states) 2025-12-04T08:46:58.8993283Z 2025-12-04T08:46:58.8993395Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.8993760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.8993833Z layer_outputs = layer_module( 2025-12-04T08:46:58.8994069Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.8994151Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.8994448Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.8994524Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.8994815Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.8994896Z self_outputs = self.self( 2025-12-04T08:46:58.8995190Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 612, in forward 2025-12-04T08:46:58.8995319Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-12-04T08:46:58.8995685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 862, in _sliding_chunks_matmul_attn_probs_value 2025-12-04T08:46:58.8995862Z padded_value = nn.functional.pad(value, (0, 0, window_overlap, window_overlap), value=-1) 2025-12-04T08:46:58.8996072Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/functional.py", line 5418, in pad 2025-12-04T08:46:58.8996174Z return torch._C._nn.pad(input, pad, mode, value) 2025-12-04T08:46:58.8996177Z 2025-12-04T08:46:58.8996288Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.8996689Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.8996763Z layer_outputs = layer_module( 2025-12-04T08:46:58.8997000Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.8997080Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.8997391Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.8997478Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.8997789Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.8997883Z self_outputs = self.self( 2025-12-04T08:46:58.8998452Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 612, in forward 2025-12-04T08:46:58.8998674Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-12-04T08:46:58.8999081Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 875, in _sliding_chunks_matmul_attn_probs_value 2025-12-04T08:46:58.8999235Z chunked_attn_probs = self._pad_and_diagonalize(chunked_attn_probs) 2025-12-04T08:46:58.8999599Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 698, in _pad_and_diagonalize 2025-12-04T08:46:58.8999694Z chunked_hidden_states = nn.functional.pad( 2025-12-04T08:46:58.8999888Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/functional.py", line 5418, in pad 2025-12-04T08:46:58.8999996Z return torch._C._nn.pad(input, pad, mode, value) 2025-12-04T08:46:58.9000000Z 2025-12-04T08:46:58.9000105Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.9000467Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.9000541Z layer_outputs = layer_module( 2025-12-04T08:46:58.9000766Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.9000852Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.9001144Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.9001220Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.9001517Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.9001589Z self_outputs = self.self( 2025-12-04T08:46:58.9001895Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 612, in forward 2025-12-04T08:46:58.9002015Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-12-04T08:46:58.9002367Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 877, in _sliding_chunks_matmul_attn_probs_value 2025-12-04T08:46:58.9002540Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-12-04T08:46:58.9002543Z 2025-12-04T08:46:58.9002641Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.9002994Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.9003064Z layer_outputs = layer_module( 2025-12-04T08:46:58.9003281Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.9003369Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.9003680Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.9003764Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.9004041Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.9004109Z self_outputs = self.self( 2025-12-04T08:46:58.9004393Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 612, in forward 2025-12-04T08:46:58.9004508Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-12-04T08:46:58.9004868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 877, in _sliding_chunks_matmul_attn_probs_value 2025-12-04T08:46:58.9005047Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-12-04T08:46:58.9005054Z 2025-12-04T08:46:58.9005155Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.9005514Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.9005586Z layer_outputs = layer_module( 2025-12-04T08:46:58.9005821Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.9005900Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.9006182Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.9006264Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.9006544Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.9006618Z self_outputs = self.self( 2025-12-04T08:46:58.9006905Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 617, in forward 2025-12-04T08:46:58.9007091Z attn_output = attn_output.transpose(0, 1).reshape(seq_len, batch_size, embed_dim).contiguous() 2025-12-04T08:46:58.9007094Z 2025-12-04T08:46:58.9007204Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.9007567Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.9007640Z layer_outputs = layer_module( 2025-12-04T08:46:58.9007872Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.9007951Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.9008252Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.9008328Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.9008614Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1143, in forward 2025-12-04T08:46:58.9008736Z attn_output = self.output(self_outputs[0], hidden_states) 2025-12-04T08:46:58.9009040Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1093, in forward 2025-12-04T08:46:58.9009136Z hidden_states = self.dense(hidden_states) 2025-12-04T08:46:58.9009140Z 2025-12-04T08:46:58.9009249Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.9009642Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.9009731Z layer_outputs = layer_module( 2025-12-04T08:46:58.9010015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.9010101Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.9010426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1210, in forward 2025-12-04T08:46:58.9010516Z layer_output = apply_chunking_to_forward( 2025-12-04T08:46:58.9010811Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:46:58.9010894Z return forward_fn(*input_tensors) 2025-12-04T08:46:58.9011203Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1217, in ff_chunk 2025-12-04T08:46:58.9011328Z intermediate_output = self.intermediate(attn_output) 2025-12-04T08:46:58.9011675Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1159, in forward 2025-12-04T08:46:58.9011771Z hidden_states = self.dense(hidden_states) 2025-12-04T08:46:58.9011775Z 2025-12-04T08:46:58.9011883Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.9012277Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.9012361Z layer_outputs = layer_module( 2025-12-04T08:46:58.9012609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.9012699Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.9013011Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1210, in forward 2025-12-04T08:46:58.9013105Z layer_output = apply_chunking_to_forward( 2025-12-04T08:46:58.9013400Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:46:58.9013483Z return forward_fn(*input_tensors) 2025-12-04T08:46:58.9013788Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1217, in ff_chunk 2025-12-04T08:46:58.9013913Z intermediate_output = self.intermediate(attn_output) 2025-12-04T08:46:58.9014221Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1160, in forward 2025-12-04T08:46:58.9014344Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:46:58.9014573Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T08:46:58.9014647Z return self.act(input) 2025-12-04T08:46:58.9014654Z 2025-12-04T08:46:58.9014762Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.9015113Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.9015192Z layer_outputs = layer_module( 2025-12-04T08:46:58.9015415Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.9015490Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.9015829Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1210, in forward 2025-12-04T08:46:58.9015914Z layer_output = apply_chunking_to_forward( 2025-12-04T08:46:58.9016183Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:46:58.9016270Z return forward_fn(*input_tensors) 2025-12-04T08:46:58.9016592Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-12-04T08:46:58.9016725Z layer_output = self.output(intermediate_output, attn_output) 2025-12-04T08:46:58.9017018Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1173, in forward 2025-12-04T08:46:58.9017099Z hidden_states = self.dense(hidden_states) 2025-12-04T08:46:58.9017103Z 2025-12-04T08:46:58.9017214Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.9017577Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.9017655Z layer_outputs = layer_module( 2025-12-04T08:46:58.9017884Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.9017992Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.9018291Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.9018370Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.9018664Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.9018736Z self_outputs = self.self( 2025-12-04T08:46:58.9019021Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 508, in forward 2025-12-04T08:46:58.9019111Z query_vectors = self.query(hidden_states) 2025-12-04T08:46:58.9019114Z 2025-12-04T08:46:58.9019216Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.9019577Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.9019662Z layer_outputs = layer_module( 2025-12-04T08:46:58.9019889Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.9019975Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.9020266Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.9020343Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.9020637Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.9020949Z self_outputs = self.self( 2025-12-04T08:46:58.9021254Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 523, in forward 2025-12-04T08:46:58.9021363Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-12-04T08:46:58.9021719Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 795, in _sliding_chunks_query_key_matmul 2025-12-04T08:46:58.9021917Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-12-04T08:46:58.9021921Z 2025-12-04T08:46:58.9022026Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.9022398Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.9022472Z layer_outputs = layer_module( 2025-12-04T08:46:58.9022710Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.9022805Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.9023191Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.9023273Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.9023590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.9023662Z self_outputs = self.self( 2025-12-04T08:46:58.9023983Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 509, in forward 2025-12-04T08:46:58.9024070Z key_vectors = self.key(hidden_states) 2025-12-04T08:46:58.9024074Z 2025-12-04T08:46:58.9024183Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.9024583Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.9024715Z layer_outputs = layer_module( 2025-12-04T08:46:58.9024966Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.9025049Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.9025359Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.9025444Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.9025733Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.9025809Z self_outputs = self.self( 2025-12-04T08:46:58.9026096Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 523, in forward 2025-12-04T08:46:58.9026201Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-12-04T08:46:58.9026574Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 795, in _sliding_chunks_query_key_matmul 2025-12-04T08:46:58.9026775Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-12-04T08:46:58.9026779Z 2025-12-04T08:46:58.9026895Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.9027281Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.9027357Z layer_outputs = layer_module( 2025-12-04T08:46:58.9027606Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.9027690Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.9027996Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.9028087Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.9028401Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.9028479Z self_outputs = self.self( 2025-12-04T08:46:58.9028778Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 523, in forward 2025-12-04T08:46:58.9028880Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-12-04T08:46:58.9029242Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 795, in _sliding_chunks_query_key_matmul 2025-12-04T08:46:58.9029429Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-12-04T08:46:58.9029432Z 2025-12-04T08:46:58.9029545Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.9029953Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.9030027Z layer_outputs = layer_module( 2025-12-04T08:46:58.9030262Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.9030340Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.9030643Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.9030719Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.9031010Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.9031088Z self_outputs = self.self( 2025-12-04T08:46:58.9031376Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 523, in forward 2025-12-04T08:46:58.9031515Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-12-04T08:46:58.9031874Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 795, in _sliding_chunks_query_key_matmul 2025-12-04T08:46:58.9032058Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-12-04T08:46:58.9032062Z 2025-12-04T08:46:58.9032152Z cudagraph partition due to non gpu ops 2025-12-04T08:46:58.9032233Z cudagraph partition due to non gpu ops 2025-12-04T08:46:58.9032311Z cudagraph partition due to non gpu ops 2025-12-04T08:46:58.9032398Z cudagraph partition due to non gpu ops 2025-12-04T08:46:58.9032502Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.9032875Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.9032950Z layer_outputs = layer_module( 2025-12-04T08:46:58.9033183Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.9033270Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.9033561Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.9033636Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.9033935Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.9034007Z self_outputs = self.self( 2025-12-04T08:46:58.9034302Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 535, in forward 2025-12-04T08:46:58.9034419Z diagonal_mask = self._sliding_chunks_query_key_matmul( 2025-12-04T08:46:58.9034802Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 833, in _sliding_chunks_query_key_matmul 2025-12-04T08:46:58.9034966Z self._mask_invalid_locations(diagonal_attention_scores, window_overlap) 2025-12-04T08:46:58.9035323Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 763, in _mask_invalid_locations 2025-12-04T08:46:58.9035431Z ).where(beginning_mask.bool(), beginning_input) 2025-12-04T08:46:58.9035435Z 2025-12-04T08:46:58.9035520Z cudagraph partition due to non gpu ops 2025-12-04T08:46:58.9035628Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.9036021Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.9036097Z layer_outputs = layer_module( 2025-12-04T08:46:58.9036350Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.9036487Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.9036799Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.9036888Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.9037195Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.9037271Z self_outputs = self.self( 2025-12-04T08:46:58.9037583Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 540, in forward 2025-12-04T08:46:58.9037663Z attn_scores += diagonal_mask 2025-12-04T08:46:58.9037667Z 2025-12-04T08:46:58.9037781Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.9038275Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.9038361Z layer_outputs = layer_module( 2025-12-04T08:46:58.9038620Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.9038706Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.9039031Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.9039116Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.9039449Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.9039533Z self_outputs = self.self( 2025-12-04T08:46:58.9039835Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 578, in forward 2025-12-04T08:46:58.9039923Z attn_probs = nn.functional.softmax( 2025-12-04T08:46:58.9039937Z 2025-12-04T08:46:58.9040046Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.9040430Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.9040518Z layer_outputs = layer_module( 2025-12-04T08:46:58.9040758Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.9040842Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.9041158Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.9041236Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.9041566Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.9041648Z self_outputs = self.self( 2025-12-04T08:46:58.9041949Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 510, in forward 2025-12-04T08:46:58.9042047Z value_vectors = self.value(hidden_states) 2025-12-04T08:46:58.9042051Z 2025-12-04T08:46:58.9042159Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.9042559Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.9042635Z layer_outputs = layer_module( 2025-12-04T08:46:58.9042872Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.9042963Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.9043316Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.9043398Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.9043709Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.9043784Z self_outputs = self.self( 2025-12-04T08:46:58.9044101Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 612, in forward 2025-12-04T08:46:58.9044228Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-12-04T08:46:58.9044614Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 862, in _sliding_chunks_matmul_attn_probs_value 2025-12-04T08:46:58.9044809Z padded_value = nn.functional.pad(value, (0, 0, window_overlap, window_overlap), value=-1) 2025-12-04T08:46:58.9045058Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/functional.py", line 5418, in pad 2025-12-04T08:46:58.9045172Z return torch._C._nn.pad(input, pad, mode, value) 2025-12-04T08:46:58.9045176Z 2025-12-04T08:46:58.9045285Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.9045683Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.9045768Z layer_outputs = layer_module( 2025-12-04T08:46:58.9046009Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.9046098Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.9046431Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.9046514Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.9046832Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.9046907Z self_outputs = self.self( 2025-12-04T08:46:58.9047210Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 612, in forward 2025-12-04T08:46:58.9047353Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-12-04T08:46:58.9047710Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 875, in _sliding_chunks_matmul_attn_probs_value 2025-12-04T08:46:58.9047855Z chunked_attn_probs = self._pad_and_diagonalize(chunked_attn_probs) 2025-12-04T08:46:58.9048182Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 698, in _pad_and_diagonalize 2025-12-04T08:46:58.9048279Z chunked_hidden_states = nn.functional.pad( 2025-12-04T08:46:58.9048491Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/functional.py", line 5418, in pad 2025-12-04T08:46:58.9048590Z return torch._C._nn.pad(input, pad, mode, value) 2025-12-04T08:46:58.9048594Z 2025-12-04T08:46:58.9048704Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.9049069Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.9049142Z layer_outputs = layer_module( 2025-12-04T08:46:58.9049382Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.9049464Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.9049825Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.9049903Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.9050215Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.9050295Z self_outputs = self.self( 2025-12-04T08:46:58.9050575Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 612, in forward 2025-12-04T08:46:58.9050691Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-12-04T08:46:58.9051052Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 877, in _sliding_chunks_matmul_attn_probs_value 2025-12-04T08:46:58.9051202Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-12-04T08:46:58.9051206Z 2025-12-04T08:46:58.9051315Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.9051712Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.9051785Z layer_outputs = layer_module( 2025-12-04T08:46:58.9052019Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.9052100Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.9052402Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.9052479Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.9052766Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.9052844Z self_outputs = self.self( 2025-12-04T08:46:58.9053133Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 612, in forward 2025-12-04T08:46:58.9053266Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-12-04T08:46:58.9053643Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 877, in _sliding_chunks_matmul_attn_probs_value 2025-12-04T08:46:58.9053795Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-12-04T08:46:58.9053798Z 2025-12-04T08:46:58.9053907Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.9054267Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.9054343Z layer_outputs = layer_module( 2025-12-04T08:46:58.9054565Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.9054643Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.9054949Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.9055025Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.9055317Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.9055393Z self_outputs = self.self( 2025-12-04T08:46:58.9055682Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 617, in forward 2025-12-04T08:46:58.9055879Z attn_output = attn_output.transpose(0, 1).reshape(seq_len, batch_size, embed_dim).contiguous() 2025-12-04T08:46:58.9055883Z 2025-12-04T08:46:58.9055985Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.9056354Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.9056439Z layer_outputs = layer_module( 2025-12-04T08:46:58.9056701Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.9056789Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.9057077Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.9057152Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.9057448Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1143, in forward 2025-12-04T08:46:58.9057562Z attn_output = self.output(self_outputs[0], hidden_states) 2025-12-04T08:46:58.9057856Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1093, in forward 2025-12-04T08:46:58.9057986Z hidden_states = self.dense(hidden_states) 2025-12-04T08:46:58.9057990Z 2025-12-04T08:46:58.9058096Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.9058465Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.9058537Z layer_outputs = layer_module( 2025-12-04T08:46:58.9058770Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.9058848Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.9059134Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1210, in forward 2025-12-04T08:46:58.9059227Z layer_output = apply_chunking_to_forward( 2025-12-04T08:46:58.9059497Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:46:58.9059578Z return forward_fn(*input_tensors) 2025-12-04T08:46:58.9059881Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1217, in ff_chunk 2025-12-04T08:46:58.9059992Z intermediate_output = self.intermediate(attn_output) 2025-12-04T08:46:58.9060284Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1159, in forward 2025-12-04T08:46:58.9060368Z hidden_states = self.dense(hidden_states) 2025-12-04T08:46:58.9060371Z 2025-12-04T08:46:58.9060471Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.9060837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.9060908Z layer_outputs = layer_module( 2025-12-04T08:46:58.9061144Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.9061224Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.9061528Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1210, in forward 2025-12-04T08:46:58.9061620Z layer_output = apply_chunking_to_forward( 2025-12-04T08:46:58.9061888Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:46:58.9061973Z return forward_fn(*input_tensors) 2025-12-04T08:46:58.9062268Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1217, in ff_chunk 2025-12-04T08:46:58.9062375Z intermediate_output = self.intermediate(attn_output) 2025-12-04T08:46:58.9062665Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1160, in forward 2025-12-04T08:46:58.9062779Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:46:58.9063026Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T08:46:58.9063109Z return self.act(input) 2025-12-04T08:46:58.9063113Z 2025-12-04T08:46:58.9063215Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.9063587Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.9063661Z layer_outputs = layer_module( 2025-12-04T08:46:58.9063885Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.9063971Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.9064258Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1210, in forward 2025-12-04T08:46:58.9064383Z layer_output = apply_chunking_to_forward( 2025-12-04T08:46:58.9064652Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:46:58.9064730Z return forward_fn(*input_tensors) 2025-12-04T08:46:58.9065027Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-12-04T08:46:58.9065154Z layer_output = self.output(intermediate_output, attn_output) 2025-12-04T08:46:58.9065443Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1173, in forward 2025-12-04T08:46:58.9065537Z hidden_states = self.dense(hidden_states) 2025-12-04T08:46:58.9065541Z 2025-12-04T08:46:58.9065643Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.9066016Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.9066089Z layer_outputs = layer_module( 2025-12-04T08:46:58.9066316Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.9066402Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.9066690Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.9066775Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.9067064Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.9067137Z self_outputs = self.self( 2025-12-04T08:46:58.9067434Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 508, in forward 2025-12-04T08:46:58.9067520Z query_vectors = self.query(hidden_states) 2025-12-04T08:46:58.9067526Z 2025-12-04T08:46:58.9067638Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.9068003Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.9068076Z layer_outputs = layer_module( 2025-12-04T08:46:58.9068309Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.9068387Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.9068676Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.9068760Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.9069049Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.9069161Z self_outputs = self.self( 2025-12-04T08:46:58.9069450Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 523, in forward 2025-12-04T08:46:58.9069554Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-12-04T08:46:58.9069927Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 795, in _sliding_chunks_query_key_matmul 2025-12-04T08:46:58.9070116Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-12-04T08:46:58.9070120Z 2025-12-04T08:46:58.9070229Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.9070591Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.9070692Z layer_outputs = layer_module( 2025-12-04T08:46:58.9070929Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.9071007Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.9071304Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.9071381Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.9071676Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.9071753Z self_outputs = self.self( 2025-12-04T08:46:58.9072040Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 509, in forward 2025-12-04T08:46:58.9072120Z key_vectors = self.key(hidden_states) 2025-12-04T08:46:58.9072134Z 2025-12-04T08:46:58.9072237Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.9072604Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.9072684Z layer_outputs = layer_module( 2025-12-04T08:46:58.9072911Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.9072988Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.9073288Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.9073364Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.9073710Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.9073788Z self_outputs = self.self( 2025-12-04T08:46:58.9074093Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 523, in forward 2025-12-04T08:46:58.9074211Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-12-04T08:46:58.9074588Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 795, in _sliding_chunks_query_key_matmul 2025-12-04T08:46:58.9074793Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-12-04T08:46:58.9074797Z 2025-12-04T08:46:58.9074906Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.9075288Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.9075374Z layer_outputs = layer_module( 2025-12-04T08:46:58.9075612Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.9075734Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.9076049Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.9076130Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.9076441Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.9076515Z self_outputs = self.self( 2025-12-04T08:46:58.9076818Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 523, in forward 2025-12-04T08:46:58.9076938Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-12-04T08:46:58.9077321Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 795, in _sliding_chunks_query_key_matmul 2025-12-04T08:46:58.9077570Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-12-04T08:46:58.9077574Z 2025-12-04T08:46:58.9077688Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.9078154Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.9078250Z layer_outputs = layer_module( 2025-12-04T08:46:58.9078505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.9078597Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.9078911Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.9078994Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.9079332Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.9079410Z self_outputs = self.self( 2025-12-04T08:46:58.9079731Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 523, in forward 2025-12-04T08:46:58.9079844Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-12-04T08:46:58.9080224Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 795, in _sliding_chunks_query_key_matmul 2025-12-04T08:46:58.9080433Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-12-04T08:46:58.9080437Z 2025-12-04T08:46:58.9080527Z cudagraph partition due to non gpu ops 2025-12-04T08:46:58.9080623Z cudagraph partition due to non gpu ops 2025-12-04T08:46:58.9080710Z cudagraph partition due to non gpu ops 2025-12-04T08:46:58.9080799Z cudagraph partition due to non gpu ops 2025-12-04T08:46:58.9080922Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.9081320Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.9081400Z layer_outputs = layer_module( 2025-12-04T08:46:58.9081656Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.9081743Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.9082082Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.9082166Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.9082491Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.9082578Z self_outputs = self.self( 2025-12-04T08:46:58.9082934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 535, in forward 2025-12-04T08:46:58.9083057Z diagonal_mask = self._sliding_chunks_query_key_matmul( 2025-12-04T08:46:58.9083445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 833, in _sliding_chunks_query_key_matmul 2025-12-04T08:46:58.9083606Z self._mask_invalid_locations(diagonal_attention_scores, window_overlap) 2025-12-04T08:46:58.9083980Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 763, in _mask_invalid_locations 2025-12-04T08:46:58.9084087Z ).where(beginning_mask.bool(), beginning_input) 2025-12-04T08:46:58.9084091Z 2025-12-04T08:46:58.9084179Z cudagraph partition due to non gpu ops 2025-12-04T08:46:58.9084346Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.9084756Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.9084844Z layer_outputs = layer_module( 2025-12-04T08:46:58.9085094Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.9085182Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.9085523Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.9085607Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.9085933Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.9086010Z self_outputs = self.self( 2025-12-04T08:46:58.9086327Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 540, in forward 2025-12-04T08:46:58.9086417Z attn_scores += diagonal_mask 2025-12-04T08:46:58.9086421Z 2025-12-04T08:46:58.9086533Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.9086939Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.9087025Z layer_outputs = layer_module( 2025-12-04T08:46:58.9087275Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.9087368Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.9087689Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.9087771Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.9088097Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.9088174Z self_outputs = self.self( 2025-12-04T08:46:58.9088495Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 578, in forward 2025-12-04T08:46:58.9088583Z attn_probs = nn.functional.softmax( 2025-12-04T08:46:58.9088587Z 2025-12-04T08:46:58.9088697Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.9089110Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.9089189Z layer_outputs = layer_module( 2025-12-04T08:46:58.9089447Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.9089536Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.9089882Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.9089972Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.9090283Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.9090359Z self_outputs = self.self( 2025-12-04T08:46:58.9090680Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 510, in forward 2025-12-04T08:46:58.9090772Z value_vectors = self.value(hidden_states) 2025-12-04T08:46:58.9090776Z 2025-12-04T08:46:58.9090894Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.9091298Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.9091411Z layer_outputs = layer_module( 2025-12-04T08:46:58.9091669Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.9091754Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.9092071Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.9092153Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.9092464Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.9092547Z self_outputs = self.self( 2025-12-04T08:46:58.9092858Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 612, in forward 2025-12-04T08:46:58.9092989Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-12-04T08:46:58.9093399Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 862, in _sliding_chunks_matmul_attn_probs_value 2025-12-04T08:46:58.9093590Z padded_value = nn.functional.pad(value, (0, 0, window_overlap, window_overlap), value=-1) 2025-12-04T08:46:58.9093799Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/functional.py", line 5418, in pad 2025-12-04T08:46:58.9093900Z return torch._C._nn.pad(input, pad, mode, value) 2025-12-04T08:46:58.9093904Z 2025-12-04T08:46:58.9094008Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.9094379Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.9094451Z layer_outputs = layer_module( 2025-12-04T08:46:58.9094682Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.9094764Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.9095054Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.9095136Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.9095423Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.9095500Z self_outputs = self.self( 2025-12-04T08:46:58.9095800Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 612, in forward 2025-12-04T08:46:58.9095921Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-12-04T08:46:58.9096314Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 875, in _sliding_chunks_matmul_attn_probs_value 2025-12-04T08:46:58.9096467Z chunked_attn_probs = self._pad_and_diagonalize(chunked_attn_probs) 2025-12-04T08:46:58.9096852Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 698, in _pad_and_diagonalize 2025-12-04T08:46:58.9096953Z chunked_hidden_states = nn.functional.pad( 2025-12-04T08:46:58.9097164Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/functional.py", line 5418, in pad 2025-12-04T08:46:58.9097275Z return torch._C._nn.pad(input, pad, mode, value) 2025-12-04T08:46:58.9097279Z 2025-12-04T08:46:58.9097388Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.9097771Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.9097854Z layer_outputs = layer_module( 2025-12-04T08:46:58.9098095Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.9098218Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.9098524Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.9098604Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.9098911Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.9098982Z self_outputs = self.self( 2025-12-04T08:46:58.9099273Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 612, in forward 2025-12-04T08:46:58.9099390Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-12-04T08:46:58.9099754Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 877, in _sliding_chunks_matmul_attn_probs_value 2025-12-04T08:46:58.9099927Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-12-04T08:46:58.9099931Z 2025-12-04T08:46:58.9100041Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.9100428Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.9100505Z layer_outputs = layer_module( 2025-12-04T08:46:58.9100744Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.9100834Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.9101136Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.9101216Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.9101532Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.9101605Z self_outputs = self.self( 2025-12-04T08:46:58.9101915Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 612, in forward 2025-12-04T08:46:58.9102038Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-12-04T08:46:58.9102420Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 877, in _sliding_chunks_matmul_attn_probs_value 2025-12-04T08:46:58.9102589Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-12-04T08:46:58.9102593Z 2025-12-04T08:46:58.9102702Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.9103101Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.9103182Z layer_outputs = layer_module( 2025-12-04T08:46:58.9103466Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.9103560Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.9103868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.9103957Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.9104263Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.9104337Z self_outputs = self.self( 2025-12-04T08:46:58.9104647Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 617, in forward 2025-12-04T08:46:58.9104886Z attn_output = attn_output.transpose(0, 1).reshape(seq_len, batch_size, embed_dim).contiguous() 2025-12-04T08:46:58.9104890Z 2025-12-04T08:46:58.9105008Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.9105393Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.9105469Z layer_outputs = layer_module( 2025-12-04T08:46:58.9105718Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.9105801Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.9106106Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.9106194Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.9106503Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1143, in forward 2025-12-04T08:46:58.9106636Z attn_output = self.output(self_outputs[0], hidden_states) 2025-12-04T08:46:58.9106943Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1093, in forward 2025-12-04T08:46:58.9107033Z hidden_states = self.dense(hidden_states) 2025-12-04T08:46:58.9107037Z 2025-12-04T08:46:58.9107153Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.9107536Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.9107620Z layer_outputs = layer_module( 2025-12-04T08:46:58.9107859Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.9107942Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.9108260Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1210, in forward 2025-12-04T08:46:58.9108352Z layer_output = apply_chunking_to_forward( 2025-12-04T08:46:58.9108650Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:46:58.9108733Z return forward_fn(*input_tensors) 2025-12-04T08:46:58.9109047Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1217, in ff_chunk 2025-12-04T08:46:58.9109171Z intermediate_output = self.intermediate(attn_output) 2025-12-04T08:46:58.9109477Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1159, in forward 2025-12-04T08:46:58.9109566Z hidden_states = self.dense(hidden_states) 2025-12-04T08:46:58.9109576Z 2025-12-04T08:46:58.9109685Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.9110114Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.9110199Z layer_outputs = layer_module( 2025-12-04T08:46:58.9110439Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.9110523Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.9110835Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1210, in forward 2025-12-04T08:46:58.9110923Z layer_output = apply_chunking_to_forward( 2025-12-04T08:46:58.9111212Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:46:58.9111296Z return forward_fn(*input_tensors) 2025-12-04T08:46:58.9111604Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1217, in ff_chunk 2025-12-04T08:46:58.9111765Z intermediate_output = self.intermediate(attn_output) 2025-12-04T08:46:58.9112069Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1160, in forward 2025-12-04T08:46:58.9112197Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:46:58.9112434Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T08:46:58.9112511Z return self.act(input) 2025-12-04T08:46:58.9112515Z 2025-12-04T08:46:58.9112631Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.9113011Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.9113087Z layer_outputs = layer_module( 2025-12-04T08:46:58.9113343Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.9113426Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.9113740Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1210, in forward 2025-12-04T08:46:58.9113829Z layer_output = apply_chunking_to_forward( 2025-12-04T08:46:58.9114113Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:46:58.9114204Z return forward_fn(*input_tensors) 2025-12-04T08:46:58.9114511Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-12-04T08:46:58.9114650Z layer_output = self.output(intermediate_output, attn_output) 2025-12-04T08:46:58.9114953Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1173, in forward 2025-12-04T08:46:58.9115045Z hidden_states = self.dense(hidden_states) 2025-12-04T08:46:58.9115049Z 2025-12-04T08:46:58.9115164Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.9115549Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.9115634Z layer_outputs = layer_module( 2025-12-04T08:46:58.9115880Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.9115964Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.9116286Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.9116369Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.9116713Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.9116800Z self_outputs = self.self( 2025-12-04T08:46:58.9117111Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 508, in forward 2025-12-04T08:46:58.9117209Z query_vectors = self.query(hidden_states) 2025-12-04T08:46:58.9117213Z 2025-12-04T08:46:58.9117325Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.9117720Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.9117808Z layer_outputs = layer_module( 2025-12-04T08:46:58.9118057Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.9118275Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.9118593Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.9118678Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.9119018Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.9119096Z self_outputs = self.self( 2025-12-04T08:46:58.9119414Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 523, in forward 2025-12-04T08:46:58.9119546Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-12-04T08:46:58.9119924Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 795, in _sliding_chunks_query_key_matmul 2025-12-04T08:46:58.9120134Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-12-04T08:46:58.9120141Z 2025-12-04T08:46:58.9120255Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.9120641Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.9120891Z layer_outputs = layer_module( 2025-12-04T08:46:58.9121142Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.9121234Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.9121548Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.9121629Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.9121951Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.9122034Z self_outputs = self.self( 2025-12-04T08:46:58.9122348Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 509, in forward 2025-12-04T08:46:58.9122435Z key_vectors = self.key(hidden_states) 2025-12-04T08:46:58.9122439Z 2025-12-04T08:46:58.9122551Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.9122948Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.9123027Z layer_outputs = layer_module( 2025-12-04T08:46:58.9123274Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.9123357Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.9123664Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.9123756Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.9124137Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.9124212Z self_outputs = self.self( 2025-12-04T08:46:58.9124525Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 523, in forward 2025-12-04T08:46:58.9124633Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-12-04T08:46:58.9125016Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 795, in _sliding_chunks_query_key_matmul 2025-12-04T08:46:58.9125220Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-12-04T08:46:58.9125224Z 2025-12-04T08:46:58.9125332Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.9125778Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.9125857Z layer_outputs = layer_module( 2025-12-04T08:46:58.9126107Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.9126192Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.9126509Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.9126600Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.9126974Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.9127059Z self_outputs = self.self( 2025-12-04T08:46:58.9127363Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 523, in forward 2025-12-04T08:46:58.9127477Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-12-04T08:46:58.9127856Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 795, in _sliding_chunks_query_key_matmul 2025-12-04T08:46:58.9128055Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-12-04T08:46:58.9128059Z 2025-12-04T08:46:58.9128175Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.9128558Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.9128635Z layer_outputs = layer_module( 2025-12-04T08:46:58.9128885Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.9128972Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.9129279Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.9129368Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.9129672Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.9129755Z self_outputs = self.self( 2025-12-04T08:46:58.9130060Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 523, in forward 2025-12-04T08:46:58.9130169Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-12-04T08:46:58.9130551Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 795, in _sliding_chunks_query_key_matmul 2025-12-04T08:46:58.9130749Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-12-04T08:46:58.9130753Z 2025-12-04T08:46:58.9130884Z cudagraph partition due to non gpu ops 2025-12-04T08:46:58.9130974Z cudagraph partition due to non gpu ops 2025-12-04T08:46:58.9131059Z cudagraph partition due to non gpu ops 2025-12-04T08:46:58.9131148Z cudagraph partition due to non gpu ops 2025-12-04T08:46:58.9131259Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.9131643Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.9131724Z layer_outputs = layer_module( 2025-12-04T08:46:58.9131962Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.9132051Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.9132357Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.9132474Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.9132788Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.9132863Z self_outputs = self.self( 2025-12-04T08:46:58.9133175Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 535, in forward 2025-12-04T08:46:58.9133291Z diagonal_mask = self._sliding_chunks_query_key_matmul( 2025-12-04T08:46:58.9133667Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 833, in _sliding_chunks_query_key_matmul 2025-12-04T08:46:58.9133831Z self._mask_invalid_locations(diagonal_attention_scores, window_overlap) 2025-12-04T08:46:58.9134187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 763, in _mask_invalid_locations 2025-12-04T08:46:58.9134301Z ).where(beginning_mask.bool(), beginning_input) 2025-12-04T08:46:58.9134311Z 2025-12-04T08:46:58.9134389Z cudagraph partition due to non gpu ops 2025-12-04T08:46:58.9134493Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.9134861Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.9134933Z layer_outputs = layer_module( 2025-12-04T08:46:58.9135162Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.9135247Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.9135538Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.9135626Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.9135917Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.9135988Z self_outputs = self.self( 2025-12-04T08:46:58.9136283Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 540, in forward 2025-12-04T08:46:58.9136358Z attn_scores += diagonal_mask 2025-12-04T08:46:58.9136362Z 2025-12-04T08:46:58.9136470Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.9136835Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.9136907Z layer_outputs = layer_module( 2025-12-04T08:46:58.9137154Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.9137240Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.9137596Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.9137686Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.9137971Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.9138048Z self_outputs = self.self( 2025-12-04T08:46:58.9138334Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 578, in forward 2025-12-04T08:46:58.9138415Z attn_probs = nn.functional.softmax( 2025-12-04T08:46:58.9138419Z 2025-12-04T08:46:58.9138528Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.9138895Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.9139014Z layer_outputs = layer_module( 2025-12-04T08:46:58.9139253Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.9139337Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.9139651Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.9139732Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.9140035Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.9140118Z self_outputs = self.self( 2025-12-04T08:46:58.9140430Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 510, in forward 2025-12-04T08:46:58.9140533Z value_vectors = self.value(hidden_states) 2025-12-04T08:46:58.9140537Z 2025-12-04T08:46:58.9140649Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.9141036Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.9141120Z layer_outputs = layer_module( 2025-12-04T08:46:58.9141367Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.9141457Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.9141762Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.9141842Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.9142153Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.9142230Z self_outputs = self.self( 2025-12-04T08:46:58.9142540Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 612, in forward 2025-12-04T08:46:58.9142667Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-12-04T08:46:58.9143053Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 862, in _sliding_chunks_matmul_attn_probs_value 2025-12-04T08:46:58.9143248Z padded_value = nn.functional.pad(value, (0, 0, window_overlap, window_overlap), value=-1) 2025-12-04T08:46:58.9143466Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/functional.py", line 5418, in pad 2025-12-04T08:46:58.9143582Z return torch._C._nn.pad(input, pad, mode, value) 2025-12-04T08:46:58.9143593Z 2025-12-04T08:46:58.9143702Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.9144174Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.9144257Z layer_outputs = layer_module( 2025-12-04T08:46:58.9144484Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.9144564Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.9144865Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.9144942Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.9145238Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.9145309Z self_outputs = self.self( 2025-12-04T08:46:58.9145593Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 612, in forward 2025-12-04T08:46:58.9145755Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-12-04T08:46:58.9146129Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 875, in _sliding_chunks_matmul_attn_probs_value 2025-12-04T08:46:58.9146274Z chunked_attn_probs = self._pad_and_diagonalize(chunked_attn_probs) 2025-12-04T08:46:58.9146601Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 698, in _pad_and_diagonalize 2025-12-04T08:46:58.9146696Z chunked_hidden_states = nn.functional.pad( 2025-12-04T08:46:58.9146902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/functional.py", line 5418, in pad 2025-12-04T08:46:58.9147002Z return torch._C._nn.pad(input, pad, mode, value) 2025-12-04T08:46:58.9147006Z 2025-12-04T08:46:58.9147117Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.9147482Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.9147555Z layer_outputs = layer_module( 2025-12-04T08:46:58.9147788Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.9147866Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.9148157Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.9148242Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.9148535Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.9148613Z self_outputs = self.self( 2025-12-04T08:46:58.9148900Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 612, in forward 2025-12-04T08:46:58.9149022Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-12-04T08:46:58.9149391Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 877, in _sliding_chunks_matmul_attn_probs_value 2025-12-04T08:46:58.9149545Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-12-04T08:46:58.9149549Z 2025-12-04T08:46:58.9149660Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.9150021Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.9150095Z layer_outputs = layer_module( 2025-12-04T08:46:58.9150336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.9150415Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.9150737Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.9150818Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.9151124Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.9151206Z self_outputs = self.self( 2025-12-04T08:46:58.9151512Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 612, in forward 2025-12-04T08:46:58.9151629Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-12-04T08:46:58.9152001Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 877, in _sliding_chunks_matmul_attn_probs_value 2025-12-04T08:46:58.9152186Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-12-04T08:46:58.9152190Z 2025-12-04T08:46:58.9152303Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.9152668Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.9152740Z layer_outputs = layer_module( 2025-12-04T08:46:58.9152972Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.9153050Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.9153346Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.9153421Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.9153710Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.9153791Z self_outputs = self.self( 2025-12-04T08:46:58.9154097Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 617, in forward 2025-12-04T08:46:58.9154308Z attn_output = attn_output.transpose(0, 1).reshape(seq_len, batch_size, embed_dim).contiguous() 2025-12-04T08:46:58.9154312Z 2025-12-04T08:46:58.9154421Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.9154816Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.9154900Z layer_outputs = layer_module( 2025-12-04T08:46:58.9155140Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.9155230Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.9155541Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.9155622Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.9155930Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1143, in forward 2025-12-04T08:46:58.9156052Z attn_output = self.output(self_outputs[0], hidden_states) 2025-12-04T08:46:58.9156358Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1093, in forward 2025-12-04T08:46:58.9156454Z hidden_states = self.dense(hidden_states) 2025-12-04T08:46:58.9156458Z 2025-12-04T08:46:58.9156565Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.9156958Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.9157037Z layer_outputs = layer_module( 2025-12-04T08:46:58.9157310Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.9157403Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.9157710Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1210, in forward 2025-12-04T08:46:58.9157809Z layer_output = apply_chunking_to_forward( 2025-12-04T08:46:58.9158159Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:46:58.9158250Z return forward_fn(*input_tensors) 2025-12-04T08:46:58.9158572Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1217, in ff_chunk 2025-12-04T08:46:58.9158690Z intermediate_output = self.intermediate(attn_output) 2025-12-04T08:46:58.9159051Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1159, in forward 2025-12-04T08:46:58.9159141Z hidden_states = self.dense(hidden_states) 2025-12-04T08:46:58.9159145Z 2025-12-04T08:46:58.9159254Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.9159647Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.9159724Z layer_outputs = layer_module( 2025-12-04T08:46:58.9159963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.9160057Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.9160364Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1210, in forward 2025-12-04T08:46:58.9160466Z layer_output = apply_chunking_to_forward( 2025-12-04T08:46:58.9160754Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:46:58.9160839Z return forward_fn(*input_tensors) 2025-12-04T08:46:58.9161156Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1217, in ff_chunk 2025-12-04T08:46:58.9161275Z intermediate_output = self.intermediate(attn_output) 2025-12-04T08:46:58.9161586Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1160, in forward 2025-12-04T08:46:58.9161708Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:46:58.9161943Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T08:46:58.9162030Z return self.act(input) 2025-12-04T08:46:58.9162036Z 2025-12-04T08:46:58.9162145Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.9162542Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.9162621Z layer_outputs = layer_module( 2025-12-04T08:46:58.9162862Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.9162955Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.9163262Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1210, in forward 2025-12-04T08:46:58.9163352Z layer_output = apply_chunking_to_forward( 2025-12-04T08:46:58.9163641Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:46:58.9163722Z return forward_fn(*input_tensors) 2025-12-04T08:46:58.9164069Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-12-04T08:46:58.9164206Z layer_output = self.output(intermediate_output, attn_output) 2025-12-04T08:46:58.9164510Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1173, in forward 2025-12-04T08:46:58.9164606Z hidden_states = self.dense(hidden_states) 2025-12-04T08:46:58.9164610Z 2025-12-04T08:46:58.9164720Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.9165107Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.9165185Z layer_outputs = layer_module( 2025-12-04T08:46:58.9165424Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.9165555Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.9165861Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.9165951Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.9166254Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.9166329Z self_outputs = self.self( 2025-12-04T08:46:58.9166638Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 508, in forward 2025-12-04T08:46:58.9166727Z query_vectors = self.query(hidden_states) 2025-12-04T08:46:58.9166731Z 2025-12-04T08:46:58.9166840Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.9167230Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.9167312Z layer_outputs = layer_module( 2025-12-04T08:46:58.9167557Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.9167641Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.9167943Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.9168032Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.9168331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.9168412Z self_outputs = self.self( 2025-12-04T08:46:58.9168713Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 523, in forward 2025-12-04T08:46:58.9168826Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-12-04T08:46:58.9169210Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 795, in _sliding_chunks_query_key_matmul 2025-12-04T08:46:58.9169411Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-12-04T08:46:58.9169414Z 2025-12-04T08:46:58.9169529Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.9169936Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.9170011Z layer_outputs = layer_module( 2025-12-04T08:46:58.9170257Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.9170339Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.9170658Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.9170783Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.9171090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.9171173Z self_outputs = self.self( 2025-12-04T08:46:58.9171473Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 509, in forward 2025-12-04T08:46:58.9171558Z key_vectors = self.key(hidden_states) 2025-12-04T08:46:58.9171562Z 2025-12-04T08:46:58.9171677Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.9172067Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.9172150Z layer_outputs = layer_module( 2025-12-04T08:46:58.9172433Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.9172517Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.9172832Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.9172909Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.9173210Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.9173282Z self_outputs = self.self( 2025-12-04T08:46:58.9173570Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 523, in forward 2025-12-04T08:46:58.9173680Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-12-04T08:46:58.9174054Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 795, in _sliding_chunks_query_key_matmul 2025-12-04T08:46:58.9174261Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-12-04T08:46:58.9174272Z 2025-12-04T08:46:58.9174384Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.9174773Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.9174857Z layer_outputs = layer_module( 2025-12-04T08:46:58.9175099Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.9175183Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.9175523Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.9175607Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.9175927Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.9176003Z self_outputs = self.self( 2025-12-04T08:46:58.9176309Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 523, in forward 2025-12-04T08:46:58.9176427Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-12-04T08:46:58.9176804Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 795, in _sliding_chunks_query_key_matmul 2025-12-04T08:46:58.9177009Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-12-04T08:46:58.9177012Z 2025-12-04T08:46:58.9177120Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.9177509Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.9177631Z layer_outputs = layer_module( 2025-12-04T08:46:58.9177871Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.9177965Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.9178269Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.9178349Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.9178659Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.9178734Z self_outputs = self.self( 2025-12-04T08:46:58.9179035Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 523, in forward 2025-12-04T08:46:58.9179185Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-12-04T08:46:58.9179564Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 795, in _sliding_chunks_query_key_matmul 2025-12-04T08:46:58.9179773Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-12-04T08:46:58.9179777Z 2025-12-04T08:46:58.9179865Z cudagraph partition due to non gpu ops 2025-12-04T08:46:58.9179952Z cudagraph partition due to non gpu ops 2025-12-04T08:46:58.9180045Z cudagraph partition due to non gpu ops 2025-12-04T08:46:58.9180126Z cudagraph partition due to non gpu ops 2025-12-04T08:46:58.9180235Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.9180629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.9180709Z layer_outputs = layer_module( 2025-12-04T08:46:58.9180962Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.9181045Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.9181350Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.9181442Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.9181742Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.9181819Z self_outputs = self.self( 2025-12-04T08:46:58.9182107Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 535, in forward 2025-12-04T08:46:58.9182221Z diagonal_mask = self._sliding_chunks_query_key_matmul( 2025-12-04T08:46:58.9182585Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 833, in _sliding_chunks_query_key_matmul 2025-12-04T08:46:58.9182734Z self._mask_invalid_locations(diagonal_attention_scores, window_overlap) 2025-12-04T08:46:58.9183099Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 763, in _mask_invalid_locations 2025-12-04T08:46:58.9183202Z ).where(beginning_mask.bool(), beginning_input) 2025-12-04T08:46:58.9183206Z 2025-12-04T08:46:58.9183289Z cudagraph partition due to non gpu ops 2025-12-04T08:46:58.9183405Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.9183792Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.9183878Z layer_outputs = layer_module( 2025-12-04T08:46:58.9184117Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.9184232Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.9184547Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.9184628Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.9184934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.9185017Z self_outputs = self.self( 2025-12-04T08:46:58.9185323Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 540, in forward 2025-12-04T08:46:58.9185410Z attn_scores += diagonal_mask 2025-12-04T08:46:58.9185413Z 2025-12-04T08:46:58.9185522Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.9185911Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.9186042Z layer_outputs = layer_module( 2025-12-04T08:46:58.9186281Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.9186369Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.9186681Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.9186763Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.9187078Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.9187151Z self_outputs = self.self( 2025-12-04T08:46:58.9187455Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 578, in forward 2025-12-04T08:46:58.9187550Z attn_probs = nn.functional.softmax( 2025-12-04T08:46:58.9187554Z 2025-12-04T08:46:58.9187664Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.9188057Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.9188134Z layer_outputs = layer_module( 2025-12-04T08:46:58.9188373Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.9188462Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.9188776Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.9188859Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.9189173Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.9189250Z self_outputs = self.self( 2025-12-04T08:46:58.9189563Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 510, in forward 2025-12-04T08:46:58.9189652Z value_vectors = self.value(hidden_states) 2025-12-04T08:46:58.9189656Z 2025-12-04T08:46:58.9189769Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.9190151Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.9190230Z layer_outputs = layer_module( 2025-12-04T08:46:58.9190480Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.9190561Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.9190867Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.9190988Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.9191295Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.9191376Z self_outputs = self.self( 2025-12-04T08:46:58.9191682Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 612, in forward 2025-12-04T08:46:58.9191810Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-12-04T08:46:58.9192206Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 862, in _sliding_chunks_matmul_attn_probs_value 2025-12-04T08:46:58.9192394Z padded_value = nn.functional.pad(value, (0, 0, window_overlap, window_overlap), value=-1) 2025-12-04T08:46:58.9192646Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/functional.py", line 5418, in pad 2025-12-04T08:46:58.9192757Z return torch._C._nn.pad(input, pad, mode, value) 2025-12-04T08:46:58.9192760Z 2025-12-04T08:46:58.9192870Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.9193260Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.9193337Z layer_outputs = layer_module( 2025-12-04T08:46:58.9193585Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.9193671Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.9193986Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.9194072Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.9194387Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.9194461Z self_outputs = self.self( 2025-12-04T08:46:58.9194776Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 612, in forward 2025-12-04T08:46:58.9194902Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-12-04T08:46:58.9195297Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 875, in _sliding_chunks_matmul_attn_probs_value 2025-12-04T08:46:58.9195447Z chunked_attn_probs = self._pad_and_diagonalize(chunked_attn_probs) 2025-12-04T08:46:58.9195798Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 698, in _pad_and_diagonalize 2025-12-04T08:46:58.9195906Z chunked_hidden_states = nn.functional.pad( 2025-12-04T08:46:58.9196125Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/functional.py", line 5418, in pad 2025-12-04T08:46:58.9196241Z return torch._C._nn.pad(input, pad, mode, value) 2025-12-04T08:46:58.9196245Z 2025-12-04T08:46:58.9196353Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.9196749Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.9196838Z layer_outputs = layer_module( 2025-12-04T08:46:58.9197093Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.9197184Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.9197503Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.9197586Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.9197931Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.9198006Z self_outputs = self.self( 2025-12-04T08:46:58.9198394Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 612, in forward 2025-12-04T08:46:58.9198539Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-12-04T08:46:58.9198941Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 877, in _sliding_chunks_matmul_attn_probs_value 2025-12-04T08:46:58.9199124Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-12-04T08:46:58.9199129Z 2025-12-04T08:46:58.9199243Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.9199660Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.9199786Z layer_outputs = layer_module( 2025-12-04T08:46:58.9200025Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.9200120Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.9200429Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.9200510Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.9200832Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.9200908Z self_outputs = self.self( 2025-12-04T08:46:58.9201218Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 612, in forward 2025-12-04T08:46:58.9201347Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-12-04T08:46:58.9201744Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 877, in _sliding_chunks_matmul_attn_probs_value 2025-12-04T08:46:58.9201913Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-12-04T08:46:58.9201917Z 2025-12-04T08:46:58.9202026Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.9202416Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.9202494Z layer_outputs = layer_module( 2025-12-04T08:46:58.9202733Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.9202823Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.9203133Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.9203213Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.9203525Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.9203600Z self_outputs = self.self( 2025-12-04T08:46:58.9203919Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 617, in forward 2025-12-04T08:46:58.9204107Z attn_output = attn_output.transpose(0, 1).reshape(seq_len, batch_size, embed_dim).contiguous() 2025-12-04T08:46:58.9204110Z 2025-12-04T08:46:58.9204212Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.9204596Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.9204673Z layer_outputs = layer_module( 2025-12-04T08:46:58.9204941Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.9205021Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.9205307Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.9205390Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.9205677Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1143, in forward 2025-12-04T08:46:58.9205798Z attn_output = self.output(self_outputs[0], hidden_states) 2025-12-04T08:46:58.9206085Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1093, in forward 2025-12-04T08:46:58.9206171Z hidden_states = self.dense(hidden_states) 2025-12-04T08:46:58.9206205Z 2025-12-04T08:46:58.9206319Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.9206680Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.9206759Z layer_outputs = layer_module( 2025-12-04T08:46:58.9206988Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.9207067Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.9207360Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1210, in forward 2025-12-04T08:46:58.9207445Z layer_output = apply_chunking_to_forward( 2025-12-04T08:46:58.9207717Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:46:58.9207807Z return forward_fn(*input_tensors) 2025-12-04T08:46:58.9208102Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1217, in ff_chunk 2025-12-04T08:46:58.9208220Z intermediate_output = self.intermediate(attn_output) 2025-12-04T08:46:58.9208508Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1159, in forward 2025-12-04T08:46:58.9208590Z hidden_states = self.dense(hidden_states) 2025-12-04T08:46:58.9208594Z 2025-12-04T08:46:58.9208702Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.9209067Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.9209146Z layer_outputs = layer_module( 2025-12-04T08:46:58.9209371Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.9209453Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.9209753Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1210, in forward 2025-12-04T08:46:58.9209837Z layer_output = apply_chunking_to_forward( 2025-12-04T08:46:58.9210117Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:46:58.9210204Z return forward_fn(*input_tensors) 2025-12-04T08:46:58.9210496Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1217, in ff_chunk 2025-12-04T08:46:58.9210615Z intermediate_output = self.intermediate(attn_output) 2025-12-04T08:46:58.9210908Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1160, in forward 2025-12-04T08:46:58.9211026Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:46:58.9211314Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T08:46:58.9211391Z return self.act(input) 2025-12-04T08:46:58.9211395Z 2025-12-04T08:46:58.9211506Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.9211885Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.9211961Z layer_outputs = layer_module( 2025-12-04T08:46:58.9212194Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.9212274Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.9212572Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1210, in forward 2025-12-04T08:46:58.9212686Z layer_output = apply_chunking_to_forward( 2025-12-04T08:46:58.9212959Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:46:58.9213045Z return forward_fn(*input_tensors) 2025-12-04T08:46:58.9213338Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-12-04T08:46:58.9213466Z layer_output = self.output(intermediate_output, attn_output) 2025-12-04T08:46:58.9213767Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1173, in forward 2025-12-04T08:46:58.9213851Z hidden_states = self.dense(hidden_states) 2025-12-04T08:46:58.9213854Z 2025-12-04T08:46:58.9213964Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.9214326Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.9214407Z layer_outputs = layer_module( 2025-12-04T08:46:58.9214642Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.9214719Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.9215018Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.9215095Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.9215383Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.9215461Z self_outputs = self.self( 2025-12-04T08:46:58.9215749Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 508, in forward 2025-12-04T08:46:58.9215835Z query_vectors = self.query(hidden_states) 2025-12-04T08:46:58.9215846Z 2025-12-04T08:46:58.9215952Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.9216325Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.9216407Z layer_outputs = layer_module( 2025-12-04T08:46:58.9216634Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.9216713Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.9217010Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.9217088Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.9217383Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.9217457Z self_outputs = self.self( 2025-12-04T08:46:58.9217772Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 523, in forward 2025-12-04T08:46:58.9217885Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-12-04T08:46:58.9218239Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 795, in _sliding_chunks_query_key_matmul 2025-12-04T08:46:58.9218431Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-12-04T08:46:58.9218435Z 2025-12-04T08:46:58.9218538Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.9218909Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.9219051Z layer_outputs = layer_module( 2025-12-04T08:46:58.9219285Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.9219371Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.9219661Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.9219738Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.9220036Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.9220107Z self_outputs = self.self( 2025-12-04T08:46:58.9220395Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 509, in forward 2025-12-04T08:46:58.9220483Z key_vectors = self.key(hidden_states) 2025-12-04T08:46:58.9220487Z 2025-12-04T08:46:58.9220592Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.9221152Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.9221230Z layer_outputs = layer_module( 2025-12-04T08:46:58.9221463Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.9221553Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.9221853Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.9221940Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.9222252Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.9222327Z self_outputs = self.self( 2025-12-04T08:46:58.9222654Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 523, in forward 2025-12-04T08:46:58.9222775Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-12-04T08:46:58.9223144Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 795, in _sliding_chunks_query_key_matmul 2025-12-04T08:46:58.9223334Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-12-04T08:46:58.9223338Z 2025-12-04T08:46:58.9223442Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.9223835Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.9223908Z layer_outputs = layer_module( 2025-12-04T08:46:58.9224147Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.9224230Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.9224590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.9224676Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.9224967Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.9225038Z self_outputs = self.self( 2025-12-04T08:46:58.9225334Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 523, in forward 2025-12-04T08:46:58.9225434Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-12-04T08:46:58.9225808Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 795, in _sliding_chunks_query_key_matmul 2025-12-04T08:46:58.9226052Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-12-04T08:46:58.9226056Z 2025-12-04T08:46:58.9226160Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.9226540Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.9226613Z layer_outputs = layer_module( 2025-12-04T08:46:58.9226862Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.9226941Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.9227237Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.9227323Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.9227615Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.9227701Z self_outputs = self.self( 2025-12-04T08:46:58.9227994Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 523, in forward 2025-12-04T08:46:58.9228096Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-12-04T08:46:58.9228461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 795, in _sliding_chunks_query_key_matmul 2025-12-04T08:46:58.9228646Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-12-04T08:46:58.9228649Z 2025-12-04T08:46:58.9228738Z cudagraph partition due to non gpu ops 2025-12-04T08:46:58.9228820Z cudagraph partition due to non gpu ops 2025-12-04T08:46:58.9228899Z cudagraph partition due to non gpu ops 2025-12-04T08:46:58.9228988Z cudagraph partition due to non gpu ops 2025-12-04T08:46:58.9229093Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.9229460Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.9229541Z layer_outputs = layer_module( 2025-12-04T08:46:58.9229772Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.9229853Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.9230154Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.9230232Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.9230534Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.9230608Z self_outputs = self.self( 2025-12-04T08:46:58.9230923Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 535, in forward 2025-12-04T08:46:58.9231044Z diagonal_mask = self._sliding_chunks_query_key_matmul( 2025-12-04T08:46:58.9231400Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 833, in _sliding_chunks_query_key_matmul 2025-12-04T08:46:58.9231555Z self._mask_invalid_locations(diagonal_attention_scores, window_overlap) 2025-12-04T08:46:58.9231888Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 763, in _mask_invalid_locations 2025-12-04T08:46:58.9231986Z ).where(beginning_mask.bool(), beginning_input) 2025-12-04T08:46:58.9231989Z 2025-12-04T08:46:58.9232079Z cudagraph partition due to non gpu ops 2025-12-04T08:46:58.9232184Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.9232590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.9232664Z layer_outputs = layer_module( 2025-12-04T08:46:58.9232896Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.9232984Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.9233277Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.9233354Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.9233652Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.9233721Z self_outputs = self.self( 2025-12-04T08:46:58.9234014Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 540, in forward 2025-12-04T08:46:58.9234096Z attn_scores += diagonal_mask 2025-12-04T08:46:58.9234099Z 2025-12-04T08:46:58.9234201Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.9234575Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.9234652Z layer_outputs = layer_module( 2025-12-04T08:46:58.9234901Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.9234984Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.9235301Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.9235389Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.9235701Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.9235783Z self_outputs = self.self( 2025-12-04T08:46:58.9236090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 578, in forward 2025-12-04T08:46:58.9236175Z attn_probs = nn.functional.softmax( 2025-12-04T08:46:58.9236179Z 2025-12-04T08:46:58.9236298Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.9236691Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.9236766Z layer_outputs = layer_module( 2025-12-04T08:46:58.9237014Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.9237096Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.9237456Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.9237538Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.9237846Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.9237928Z self_outputs = self.self( 2025-12-04T08:46:58.9238299Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 510, in forward 2025-12-04T08:46:58.9238400Z value_vectors = self.value(hidden_states) 2025-12-04T08:46:58.9238404Z 2025-12-04T08:46:58.9238514Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.9238901Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.9239022Z layer_outputs = layer_module( 2025-12-04T08:46:58.9239266Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.9239357Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.9239668Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.9239749Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.9240066Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.9240142Z self_outputs = self.self( 2025-12-04T08:46:58.9240449Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 612, in forward 2025-12-04T08:46:58.9240585Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-12-04T08:46:58.9240980Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 862, in _sliding_chunks_matmul_attn_probs_value 2025-12-04T08:46:58.9241175Z padded_value = nn.functional.pad(value, (0, 0, window_overlap, window_overlap), value=-1) 2025-12-04T08:46:58.9241391Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/functional.py", line 5418, in pad 2025-12-04T08:46:58.9241498Z return torch._C._nn.pad(input, pad, mode, value) 2025-12-04T08:46:58.9241502Z 2025-12-04T08:46:58.9241619Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.9242005Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.9242091Z layer_outputs = layer_module( 2025-12-04T08:46:58.9242334Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.9242422Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.9242741Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.9242821Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.9243133Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.9243207Z self_outputs = self.self( 2025-12-04T08:46:58.9243514Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 612, in forward 2025-12-04T08:46:58.9243647Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-12-04T08:46:58.9244035Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 875, in _sliding_chunks_matmul_attn_probs_value 2025-12-04T08:46:58.9244189Z chunked_attn_probs = self._pad_and_diagonalize(chunked_attn_probs) 2025-12-04T08:46:58.9244587Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 698, in _pad_and_diagonalize 2025-12-04T08:46:58.9244686Z chunked_hidden_states = nn.functional.pad( 2025-12-04T08:46:58.9244902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/functional.py", line 5418, in pad 2025-12-04T08:46:58.9245007Z return torch._C._nn.pad(input, pad, mode, value) 2025-12-04T08:46:58.9245010Z 2025-12-04T08:46:58.9245120Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.9245515Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.9245592Z layer_outputs = layer_module( 2025-12-04T08:46:58.9245840Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.9245954Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.9246266Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.9246356Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.9246662Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.9246743Z self_outputs = self.self( 2025-12-04T08:46:58.9247058Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 612, in forward 2025-12-04T08:46:58.9247176Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-12-04T08:46:58.9247549Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 877, in _sliding_chunks_matmul_attn_probs_value 2025-12-04T08:46:58.9247711Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-12-04T08:46:58.9247717Z 2025-12-04T08:46:58.9247828Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.9248194Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.9248266Z layer_outputs = layer_module( 2025-12-04T08:46:58.9248502Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.9248582Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.9248872Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.9248955Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.9249246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.9249330Z self_outputs = self.self( 2025-12-04T08:46:58.9249618Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 612, in forward 2025-12-04T08:46:58.9249736Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-12-04T08:46:58.9250111Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 877, in _sliding_chunks_matmul_attn_probs_value 2025-12-04T08:46:58.9250267Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-12-04T08:46:58.9250271Z 2025-12-04T08:46:58.9250381Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.9250754Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.9250834Z layer_outputs = layer_module( 2025-12-04T08:46:58.9251114Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.9251202Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.9251514Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.9251596Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.9251902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.9251996Z self_outputs = self.self( 2025-12-04T08:46:58.9252283Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 617, in forward 2025-12-04T08:46:58.9252474Z attn_output = attn_output.transpose(0, 1).reshape(seq_len, batch_size, embed_dim).contiguous() 2025-12-04T08:46:58.9252514Z 2025-12-04T08:46:58.9252621Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.9252984Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.9253063Z layer_outputs = layer_module( 2025-12-04T08:46:58.9253289Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.9253367Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.9253666Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.9253743Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.9254047Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1143, in forward 2025-12-04T08:46:58.9254171Z attn_output = self.output(self_outputs[0], hidden_states) 2025-12-04T08:46:58.9254481Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1093, in forward 2025-12-04T08:46:58.9254581Z hidden_states = self.dense(hidden_states) 2025-12-04T08:46:58.9254584Z 2025-12-04T08:46:58.9254693Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.9255088Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.9255165Z layer_outputs = layer_module( 2025-12-04T08:46:58.9255407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.9255499Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.9255821Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1210, in forward 2025-12-04T08:46:58.9255917Z layer_output = apply_chunking_to_forward( 2025-12-04T08:46:58.9256212Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:46:58.9256296Z return forward_fn(*input_tensors) 2025-12-04T08:46:58.9256631Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1217, in ff_chunk 2025-12-04T08:46:58.9256753Z intermediate_output = self.intermediate(attn_output) 2025-12-04T08:46:58.9257068Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1159, in forward 2025-12-04T08:46:58.9257170Z hidden_states = self.dense(hidden_states) 2025-12-04T08:46:58.9257174Z 2025-12-04T08:46:58.9257286Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.9257735Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.9257815Z layer_outputs = layer_module( 2025-12-04T08:46:58.9258055Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.9258147Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.9258453Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1210, in forward 2025-12-04T08:46:58.9258549Z layer_output = apply_chunking_to_forward( 2025-12-04T08:46:58.9258832Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:46:58.9258914Z return forward_fn(*input_tensors) 2025-12-04T08:46:58.9259227Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1217, in ff_chunk 2025-12-04T08:46:58.9259390Z intermediate_output = self.intermediate(attn_output) 2025-12-04T08:46:58.9259696Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1160, in forward 2025-12-04T08:46:58.9259826Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:46:58.9260064Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T08:46:58.9260150Z return self.act(input) 2025-12-04T08:46:58.9260154Z 2025-12-04T08:46:58.9260264Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.9260652Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.9260735Z layer_outputs = layer_module( 2025-12-04T08:46:58.9260980Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.9261072Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.9261378Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1210, in forward 2025-12-04T08:46:58.9261467Z layer_output = apply_chunking_to_forward( 2025-12-04T08:46:58.9261760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:46:58.9261841Z return forward_fn(*input_tensors) 2025-12-04T08:46:58.9262159Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-12-04T08:46:58.9262294Z layer_output = self.output(intermediate_output, attn_output) 2025-12-04T08:46:58.9262601Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1173, in forward 2025-12-04T08:46:58.9262706Z hidden_states = self.dense(hidden_states) 2025-12-04T08:46:58.9262709Z 2025-12-04T08:46:58.9262818Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.9263201Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.9263287Z layer_outputs = layer_module( 2025-12-04T08:46:58.9263527Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.9263618Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.9263925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.9264007Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.9264319Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.9264428Z self_outputs = self.self( 2025-12-04T08:46:58.9264740Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 508, in forward 2025-12-04T08:46:58.9264831Z query_vectors = self.query(hidden_states) 2025-12-04T08:46:58.9264835Z 2025-12-04T08:46:58.9264943Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.9265335Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.9265412Z layer_outputs = layer_module( 2025-12-04T08:46:58.9265660Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.9265743Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.9266083Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.9266173Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.9266494Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.9266570Z self_outputs = self.self( 2025-12-04T08:46:58.9266893Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 523, in forward 2025-12-04T08:46:58.9267002Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-12-04T08:46:58.9267381Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 795, in _sliding_chunks_query_key_matmul 2025-12-04T08:46:58.9267579Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-12-04T08:46:58.9267586Z 2025-12-04T08:46:58.9267697Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.9268090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.9268167Z layer_outputs = layer_module( 2025-12-04T08:46:58.9268415Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.9268496Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.9268810Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.9268900Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.9269209Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.9269293Z self_outputs = self.self( 2025-12-04T08:46:58.9269600Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 509, in forward 2025-12-04T08:46:58.9269686Z key_vectors = self.key(hidden_states) 2025-12-04T08:46:58.9269690Z 2025-12-04T08:46:58.9269807Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.9270188Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.9270264Z layer_outputs = layer_module( 2025-12-04T08:46:58.9270511Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.9270595Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.9270906Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.9270989Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.9271321Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.9271406Z self_outputs = self.self( 2025-12-04T08:46:58.9271713Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 523, in forward 2025-12-04T08:46:58.9271829Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-12-04T08:46:58.9272205Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 795, in _sliding_chunks_query_key_matmul 2025-12-04T08:46:58.9272407Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-12-04T08:46:58.9272410Z 2025-12-04T08:46:58.9272529Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.9272953Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.9273037Z layer_outputs = layer_module( 2025-12-04T08:46:58.9273275Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.9273354Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.9273650Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.9273726Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.9274016Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.9274092Z self_outputs = self.self( 2025-12-04T08:46:58.9274395Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 523, in forward 2025-12-04T08:46:58.9274516Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-12-04T08:46:58.9274891Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 795, in _sliding_chunks_query_key_matmul 2025-12-04T08:46:58.9275087Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-12-04T08:46:58.9275099Z 2025-12-04T08:46:58.9275208Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.9275590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.9275683Z layer_outputs = layer_module( 2025-12-04T08:46:58.9275912Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.9275994Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.9276293Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.9276370Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.9276664Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.9276735Z self_outputs = self.self( 2025-12-04T08:46:58.9277034Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 523, in forward 2025-12-04T08:46:58.9277146Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-12-04T08:46:58.9277515Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 795, in _sliding_chunks_query_key_matmul 2025-12-04T08:46:58.9277717Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-12-04T08:46:58.9277724Z 2025-12-04T08:46:58.9277841Z cudagraph partition due to non gpu ops 2025-12-04T08:46:58.9277929Z cudagraph partition due to non gpu ops 2025-12-04T08:46:58.9278019Z cudagraph partition due to non gpu ops 2025-12-04T08:46:58.9278168Z cudagraph partition due to non gpu ops 2025-12-04T08:46:58.9278291Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.9278701Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.9278783Z layer_outputs = layer_module( 2025-12-04T08:46:58.9279041Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.9279127Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.9279442Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.9279572Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.9279899Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.9279974Z self_outputs = self.self( 2025-12-04T08:46:58.9280290Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 535, in forward 2025-12-04T08:46:58.9280408Z diagonal_mask = self._sliding_chunks_query_key_matmul( 2025-12-04T08:46:58.9280779Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 833, in _sliding_chunks_query_key_matmul 2025-12-04T08:46:58.9280927Z self._mask_invalid_locations(diagonal_attention_scores, window_overlap) 2025-12-04T08:46:58.9281258Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 763, in _mask_invalid_locations 2025-12-04T08:46:58.9281368Z ).where(beginning_mask.bool(), beginning_input) 2025-12-04T08:46:58.9281373Z 2025-12-04T08:46:58.9281454Z cudagraph partition due to non gpu ops 2025-12-04T08:46:58.9281564Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.9281924Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.9281996Z layer_outputs = layer_module( 2025-12-04T08:46:58.9282230Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.9282309Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.9282603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.9282685Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.9282978Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.9283058Z self_outputs = self.self( 2025-12-04T08:46:58.9283357Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 540, in forward 2025-12-04T08:46:58.9283437Z attn_scores += diagonal_mask 2025-12-04T08:46:58.9283449Z 2025-12-04T08:46:58.9283557Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.9283937Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.9284023Z layer_outputs = layer_module( 2025-12-04T08:46:58.9284263Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.9284349Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.9284689Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.9284769Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.9285086Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.9285157Z self_outputs = self.self( 2025-12-04T08:46:58.9285444Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 578, in forward 2025-12-04T08:46:58.9285533Z attn_probs = nn.functional.softmax( 2025-12-04T08:46:58.9285536Z 2025-12-04T08:46:58.9285637Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.9286006Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.9286121Z layer_outputs = layer_module( 2025-12-04T08:46:58.9286365Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.9286457Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.9286762Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.9286841Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.9287151Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.9287226Z self_outputs = self.self( 2025-12-04T08:46:58.9287532Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 510, in forward 2025-12-04T08:46:58.9287623Z value_vectors = self.value(hidden_states) 2025-12-04T08:46:58.9287629Z 2025-12-04T08:46:58.9287738Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.9288130Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.9288208Z layer_outputs = layer_module( 2025-12-04T08:46:58.9288453Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.9288542Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.9288845Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.9288933Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.9289237Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.9289321Z self_outputs = self.self( 2025-12-04T08:46:58.9289625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 612, in forward 2025-12-04T08:46:58.9289751Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-12-04T08:46:58.9290142Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 862, in _sliding_chunks_matmul_attn_probs_value 2025-12-04T08:46:58.9290326Z padded_value = nn.functional.pad(value, (0, 0, window_overlap, window_overlap), value=-1) 2025-12-04T08:46:58.9290536Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/functional.py", line 5418, in pad 2025-12-04T08:46:58.9290649Z return torch._C._nn.pad(input, pad, mode, value) 2025-12-04T08:46:58.9290652Z 2025-12-04T08:46:58.9290762Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.9291153Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.9291264Z layer_outputs = layer_module( 2025-12-04T08:46:58.9291505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.9291595Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.9291904Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.9291992Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.9292298Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.9292374Z self_outputs = self.self( 2025-12-04T08:46:58.9292684Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 612, in forward 2025-12-04T08:46:58.9292843Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-12-04T08:46:58.9293237Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 875, in _sliding_chunks_matmul_attn_probs_value 2025-12-04T08:46:58.9293385Z chunked_attn_probs = self._pad_and_diagonalize(chunked_attn_probs) 2025-12-04T08:46:58.9293732Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 698, in _pad_and_diagonalize 2025-12-04T08:46:58.9293839Z chunked_hidden_states = nn.functional.pad( 2025-12-04T08:46:58.9294051Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/functional.py", line 5418, in pad 2025-12-04T08:46:58.9294156Z return torch._C._nn.pad(input, pad, mode, value) 2025-12-04T08:46:58.9294166Z 2025-12-04T08:46:58.9294276Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.9294665Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.9294748Z layer_outputs = layer_module( 2025-12-04T08:46:58.9294987Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.9295071Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.9295395Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.9295474Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.9295783Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.9295856Z self_outputs = self.self( 2025-12-04T08:46:58.9296157Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 612, in forward 2025-12-04T08:46:58.9296295Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-12-04T08:46:58.9296679Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 877, in _sliding_chunks_matmul_attn_probs_value 2025-12-04T08:46:58.9296849Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-12-04T08:46:58.9296852Z 2025-12-04T08:46:58.9296962Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.9297345Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.9297429Z layer_outputs = layer_module( 2025-12-04T08:46:58.9297669Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.9297761Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.9298099Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.9298179Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.9298487Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.9298562Z self_outputs = self.self( 2025-12-04T08:46:58.9298860Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 612, in forward 2025-12-04T08:46:58.9298991Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-12-04T08:46:58.9299394Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 877, in _sliding_chunks_matmul_attn_probs_value 2025-12-04T08:46:58.9299569Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-12-04T08:46:58.9299607Z 2025-12-04T08:46:58.9299719Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.9300103Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.9300188Z layer_outputs = layer_module( 2025-12-04T08:46:58.9300426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.9300518Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.9300823Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.9300905Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.9301216Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1134, in forward 2025-12-04T08:46:58.9301294Z self_outputs = self.self( 2025-12-04T08:46:58.9301608Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 617, in forward 2025-12-04T08:46:58.9301811Z attn_output = attn_output.transpose(0, 1).reshape(seq_len, batch_size, embed_dim).contiguous() 2025-12-04T08:46:58.9301815Z 2025-12-04T08:46:58.9301923Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.9302313Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.9302389Z layer_outputs = layer_module( 2025-12-04T08:46:58.9302646Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.9302730Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.9303034Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1198, in forward 2025-12-04T08:46:58.9303130Z self_attn_outputs = self.attention( 2025-12-04T08:46:58.9303436Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1143, in forward 2025-12-04T08:46:58.9303557Z attn_output = self.output(self_outputs[0], hidden_states) 2025-12-04T08:46:58.9303872Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1093, in forward 2025-12-04T08:46:58.9303962Z hidden_states = self.dense(hidden_states) 2025-12-04T08:46:58.9303965Z 2025-12-04T08:46:58.9304082Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.9304465Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.9304545Z layer_outputs = layer_module( 2025-12-04T08:46:58.9304829Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.9304913Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.9305227Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1210, in forward 2025-12-04T08:46:58.9305320Z layer_output = apply_chunking_to_forward( 2025-12-04T08:46:58.9305605Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:46:58.9305694Z return forward_fn(*input_tensors) 2025-12-04T08:46:58.9306003Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1217, in ff_chunk 2025-12-04T08:46:58.9306119Z intermediate_output = self.intermediate(attn_output) 2025-12-04T08:46:58.9306430Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1159, in forward 2025-12-04T08:46:58.9306554Z hidden_states = self.dense(hidden_states) 2025-12-04T08:46:58.9306558Z 2025-12-04T08:46:58.9306673Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.9307063Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.9307141Z layer_outputs = layer_module( 2025-12-04T08:46:58.9307391Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.9307474Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.9307824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1210, in forward 2025-12-04T08:46:58.9307907Z layer_output = apply_chunking_to_forward( 2025-12-04T08:46:58.9308189Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:46:58.9308274Z return forward_fn(*input_tensors) 2025-12-04T08:46:58.9308589Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1217, in ff_chunk 2025-12-04T08:46:58.9308712Z intermediate_output = self.intermediate(attn_output) 2025-12-04T08:46:58.9309027Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1160, in forward 2025-12-04T08:46:58.9309150Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:46:58.9309398Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T08:46:58.9309475Z return self.act(input) 2025-12-04T08:46:58.9309480Z 2025-12-04T08:46:58.9309588Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:46:58.9309996Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1258, in torch_dynamo_resume_in_forward_at_1243 2025-12-04T08:46:58.9310074Z layer_outputs = layer_module( 2025-12-04T08:46:58.9310328Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:46:58.9310411Z return super().__call__(*args, **kwargs) 2025-12-04T08:46:58.9310725Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1210, in forward 2025-12-04T08:46:58.9310823Z layer_output = apply_chunking_to_forward( 2025-12-04T08:46:58.9311113Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:46:58.9311203Z return forward_fn(*input_tensors) 2025-12-04T08:46:58.9311521Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-12-04T08:46:58.9311694Z layer_output = self.output(intermediate_output, attn_output) 2025-12-04T08:46:58.9312010Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1173, in forward 2025-12-04T08:46:58.9312099Z hidden_states = self.dense(hidden_states) 2025-12-04T08:46:58.9312103Z 2025-12-04T08:48:10.3110872Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:48:10.3113398Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1715, in torch_dynamo_resume_in_forward_at_1702 2025-12-04T08:48:10.3114241Z prediction_scores = self.lm_head(sequence_output) 2025-12-04T08:48:10.3114783Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1332, in forward 2025-12-04T08:48:10.3115930Z x = self.dense(features) 2025-12-04T08:48:10.3116059Z 2025-12-04T08:48:10.3117713Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:48:10.3118511Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1715, in torch_dynamo_resume_in_forward_at_1702 2025-12-04T08:48:10.3119130Z prediction_scores = self.lm_head(sequence_output) 2025-12-04T08:48:10.3119645Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1337, in forward 2025-12-04T08:48:10.3120120Z x = self.decoder(x) 2025-12-04T08:48:10.3120234Z 2025-12-04T08:48:10.3120344Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:48:10.3121058Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1722, in torch_dynamo_resume_in_forward_at_1702 2025-12-04T08:48:10.3121704Z masked_lm_loss = loss_fct(prediction_scores.view(-1, self.config.vocab_size), labels.view(-1)) 2025-12-04T08:48:10.3121955Z 2025-12-04T08:48:12.4986558Z Compilation time (from dynamo_timed): 105.742740033 2025-12-04T08:48:12.5187783Z pass 2025-12-04T08:48:12.5188243Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T08:48:12.5189160Z TIMING: gc:0.00693 entire_frame_compile:105.74274 _recursive_pre_grad_passes:0.02103 _recursive_joint_graph_passes:0.99427 _recursive_post_grad_passes:1.79398 async_compile.wait:3.46334 code_gen:81.68761 inductor_compile:89.35791 backend_compile:99.81995 total_wall_time:105.74274 2025-12-04T08:48:12.5190370Z STATS: call_* op count: 1787 | FakeTensorMode.__torch_dispatch__:40518 | FakeTensor.__torch_dispatch__:16210 | ProxyTorchDispatchMode.__torch_dispatch__:10021 2025-12-04T08:48:12.5190967Z Dynamo produced 4 graphs covering 1787 ops with 4 graph breaks (1 unique) 2025-12-04T08:48:16.3332195Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/__init__.py:65: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-12-04T08:48:16.3333114Z import pynvml # type: ignore[import] 2025-12-04T08:48:19.8219872Z 2025-12-04T08:48:23.0040972Z loading model: 0it [00:00, ?it/s] 2025-12-04T08:48:23.0041299Z loading model: 0it [00:03, ?it/s] 2025-12-04T08:48:23.0067303Z cpu eval BartForCausalLM 2025-12-04T08:48:27.4995899Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T08:48:29.5255410Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T08:48:31.5710989Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T08:48:38.7997137Z cudagraph partition due to non gpu ops 2025-12-04T08:48:38.7997536Z cudagraph partition due to non gpu ops 2025-12-04T08:48:38.7998392Z cudagraph partition due to non gpu ops 2025-12-04T08:48:38.7998653Z cudagraph partition due to non gpu ops 2025-12-04T08:48:38.7998892Z cudagraph partition due to non gpu ops 2025-12-04T08:48:38.7999122Z cudagraph partition due to non gpu ops 2025-12-04T08:48:38.7999359Z cudagraph partition due to non gpu ops 2025-12-04T08:48:38.7999618Z cudagraph partition due to non gpu ops 2025-12-04T08:48:38.7999849Z cudagraph partition due to non gpu ops 2025-12-04T08:48:38.8000082Z cudagraph partition due to non gpu ops 2025-12-04T08:48:38.8000316Z cudagraph partition due to non gpu ops 2025-12-04T08:48:38.8000543Z cudagraph partition due to non gpu ops 2025-12-04T08:48:38.8000817Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:48:38.8001261Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:48:38.8001653Z res = mod(**inputs) 2025-12-04T08:48:38.8002242Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T08:48:38.8002706Z outputs = self.model.decoder( 2025-12-04T08:48:38.8003165Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T08:48:38.8003611Z layer_outputs = decoder_layer( 2025-12-04T08:48:38.8004027Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:48:38.8004447Z return super().__call__(*args, **kwargs) 2025-12-04T08:48:38.8004900Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8005323Z return func(*args, **kwargs) 2025-12-04T08:48:38.8005754Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T08:48:38.8006253Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:48:38.8006706Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8007159Z return func(*args, **kwargs) 2025-12-04T08:48:38.8007579Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 219, in forward 2025-12-04T08:48:38.8008115Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T08:48:38.8008348Z 2025-12-04T08:48:38.8008481Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:48:38.8008892Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:48:38.8009273Z res = mod(**inputs) 2025-12-04T08:48:38.8009699Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T08:48:38.8010146Z outputs = self.model.decoder( 2025-12-04T08:48:38.8010589Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T08:48:38.8011033Z layer_outputs = decoder_layer( 2025-12-04T08:48:38.8011438Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:48:38.8011844Z return super().__call__(*args, **kwargs) 2025-12-04T08:48:38.8012285Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8012711Z return func(*args, **kwargs) 2025-12-04T08:48:38.8013129Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T08:48:38.8013602Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:48:38.8014051Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8014483Z return func(*args, **kwargs) 2025-12-04T08:48:38.8014950Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 239, in forward 2025-12-04T08:48:38.8015393Z key_states = self.k_proj(current_states) 2025-12-04T08:48:38.8015556Z 2025-12-04T08:48:38.8015677Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:48:38.8016082Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:48:38.8016454Z res = mod(**inputs) 2025-12-04T08:48:38.8016857Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T08:48:38.8017305Z outputs = self.model.decoder( 2025-12-04T08:48:38.8017719Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T08:48:38.8018228Z layer_outputs = decoder_layer( 2025-12-04T08:48:38.8018636Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:48:38.8019051Z return super().__call__(*args, **kwargs) 2025-12-04T08:48:38.8019476Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8019902Z return func(*args, **kwargs) 2025-12-04T08:48:38.8020324Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T08:48:38.8021091Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:48:38.8021548Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8021973Z return func(*args, **kwargs) 2025-12-04T08:48:38.8022395Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 240, in forward 2025-12-04T08:48:38.8022845Z value_states = self.v_proj(current_states) 2025-12-04T08:48:38.8023022Z 2025-12-04T08:48:38.8023116Z cudagraph partition due to non gpu ops 2025-12-04T08:48:38.8023552Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:48:38.8023960Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:48:38.8024319Z res = mod(**inputs) 2025-12-04T08:48:38.8024725Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T08:48:38.8025162Z outputs = self.model.decoder( 2025-12-04T08:48:38.8025588Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T08:48:38.8026018Z layer_outputs = decoder_layer( 2025-12-04T08:48:38.8026415Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:48:38.8026839Z return super().__call__(*args, **kwargs) 2025-12-04T08:48:38.8027258Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8027683Z return func(*args, **kwargs) 2025-12-04T08:48:38.8028095Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T08:48:38.8028557Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:48:38.8029000Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8029418Z return func(*args, **kwargs) 2025-12-04T08:48:38.8029832Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 258, in forward 2025-12-04T08:48:38.8030292Z attn_output, attn_weights = attention_interface( 2025-12-04T08:48:38.8030869Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T08:48:38.8031420Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T08:48:38.8031632Z 2025-12-04T08:48:38.8031757Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:48:38.8032156Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:48:38.8032526Z res = mod(**inputs) 2025-12-04T08:48:38.8032926Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T08:48:38.8033367Z outputs = self.model.decoder( 2025-12-04T08:48:38.8033788Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T08:48:38.8034220Z layer_outputs = decoder_layer( 2025-12-04T08:48:38.8034693Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:48:38.8035968Z return super().__call__(*args, **kwargs) 2025-12-04T08:48:38.8036736Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8037429Z return func(*args, **kwargs) 2025-12-04T08:48:38.8039456Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T08:48:38.8040268Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:48:38.8040942Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8041451Z return func(*args, **kwargs) 2025-12-04T08:48:38.8042015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 272, in forward 2025-12-04T08:48:38.8042763Z attn_output = self.out_proj(attn_output) 2025-12-04T08:48:38.8043012Z 2025-12-04T08:48:38.8043231Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:48:38.8043903Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:48:38.8044396Z res = mod(**inputs) 2025-12-04T08:48:38.8045021Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T08:48:38.8045701Z outputs = self.model.decoder( 2025-12-04T08:48:38.8046185Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T08:48:38.8046745Z layer_outputs = decoder_layer( 2025-12-04T08:48:38.8047172Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:48:38.8047669Z return super().__call__(*args, **kwargs) 2025-12-04T08:48:38.8048116Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8048529Z return func(*args, **kwargs) 2025-12-04T08:48:38.8048959Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 450, in forward 2025-12-04T08:48:38.8049468Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:48:38.8049677Z 2025-12-04T08:48:38.8049806Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:48:38.8050223Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:48:38.8050597Z res = mod(**inputs) 2025-12-04T08:48:38.8051006Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T08:48:38.8051442Z outputs = self.model.decoder( 2025-12-04T08:48:38.8051895Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T08:48:38.8052580Z layer_outputs = decoder_layer( 2025-12-04T08:48:38.8052990Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:48:38.8053406Z return super().__call__(*args, **kwargs) 2025-12-04T08:48:38.8053839Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8054263Z return func(*args, **kwargs) 2025-12-04T08:48:38.8054684Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 450, in forward 2025-12-04T08:48:38.8055175Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:48:38.8055695Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T08:48:38.8056221Z return self.act(input) 2025-12-04T08:48:38.8056356Z 2025-12-04T08:48:38.8056486Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:48:38.8056923Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:48:38.8057301Z res = mod(**inputs) 2025-12-04T08:48:38.8057727Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T08:48:38.8058171Z outputs = self.model.decoder( 2025-12-04T08:48:38.8058610Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T08:48:38.8059049Z layer_outputs = decoder_layer( 2025-12-04T08:48:38.8059438Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:48:38.8059857Z return super().__call__(*args, **kwargs) 2025-12-04T08:48:38.8060293Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8060716Z return func(*args, **kwargs) 2025-12-04T08:48:38.8061135Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 452, in forward 2025-12-04T08:48:38.8061582Z hidden_states = self.fc2(hidden_states) 2025-12-04T08:48:38.8061742Z 2025-12-04T08:48:38.8061870Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:48:38.8062276Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:48:38.8062644Z res = mod(**inputs) 2025-12-04T08:48:38.8063068Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T08:48:38.8063514Z outputs = self.model.decoder( 2025-12-04T08:48:38.8063987Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T08:48:38.8064437Z layer_outputs = decoder_layer( 2025-12-04T08:48:38.8064847Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:48:38.8065251Z return super().__call__(*args, **kwargs) 2025-12-04T08:48:38.8065679Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8066115Z return func(*args, **kwargs) 2025-12-04T08:48:38.8066536Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T08:48:38.8067016Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:48:38.8067471Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8067906Z return func(*args, **kwargs) 2025-12-04T08:48:38.8068377Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 219, in forward 2025-12-04T08:48:38.8068897Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T08:48:38.8069130Z 2025-12-04T08:48:38.8069251Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:48:38.8069650Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:48:38.8070145Z res = mod(**inputs) 2025-12-04T08:48:38.8070571Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T08:48:38.8071027Z outputs = self.model.decoder( 2025-12-04T08:48:38.8071469Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T08:48:38.8071904Z layer_outputs = decoder_layer( 2025-12-04T08:48:38.8072353Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:48:38.8072774Z return super().__call__(*args, **kwargs) 2025-12-04T08:48:38.8073197Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8073621Z return func(*args, **kwargs) 2025-12-04T08:48:38.8074053Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T08:48:38.8074522Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:48:38.8074977Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8075401Z return func(*args, **kwargs) 2025-12-04T08:48:38.8075824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 239, in forward 2025-12-04T08:48:38.8076267Z key_states = self.k_proj(current_states) 2025-12-04T08:48:38.8076428Z 2025-12-04T08:48:38.8076554Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:48:38.8076961Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:48:38.8077329Z res = mod(**inputs) 2025-12-04T08:48:38.8077730Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T08:48:38.8078250Z outputs = self.model.decoder( 2025-12-04T08:48:38.8078753Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T08:48:38.8079202Z layer_outputs = decoder_layer( 2025-12-04T08:48:38.8079590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:48:38.8080016Z return super().__call__(*args, **kwargs) 2025-12-04T08:48:38.8080453Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8080869Z return func(*args, **kwargs) 2025-12-04T08:48:38.8081284Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T08:48:38.8081879Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:48:38.8082364Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8082778Z return func(*args, **kwargs) 2025-12-04T08:48:38.8083203Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 240, in forward 2025-12-04T08:48:38.8083645Z value_states = self.v_proj(current_states) 2025-12-04T08:48:38.8083801Z 2025-12-04T08:48:38.8083901Z cudagraph partition due to non gpu ops 2025-12-04T08:48:38.8084162Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:48:38.8084621Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:48:38.8084981Z res = mod(**inputs) 2025-12-04T08:48:38.8085385Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T08:48:38.8085820Z outputs = self.model.decoder( 2025-12-04T08:48:38.8086257Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T08:48:38.8086701Z layer_outputs = decoder_layer( 2025-12-04T08:48:38.8087088Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:48:38.8087497Z return super().__call__(*args, **kwargs) 2025-12-04T08:48:38.8087918Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8088377Z return func(*args, **kwargs) 2025-12-04T08:48:38.8088794Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T08:48:38.8089260Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:48:38.8089707Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8090119Z return func(*args, **kwargs) 2025-12-04T08:48:38.8090543Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 258, in forward 2025-12-04T08:48:38.8091012Z attn_output, attn_weights = attention_interface( 2025-12-04T08:48:38.8091508Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T08:48:38.8092062Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T08:48:38.8092281Z 2025-12-04T08:48:38.8092399Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:48:38.8092804Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:48:38.8093160Z res = mod(**inputs) 2025-12-04T08:48:38.8093570Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T08:48:38.8094025Z outputs = self.model.decoder( 2025-12-04T08:48:38.8094472Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T08:48:38.8094915Z layer_outputs = decoder_layer( 2025-12-04T08:48:38.8095328Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:48:38.8095733Z return super().__call__(*args, **kwargs) 2025-12-04T08:48:38.8096159Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8096597Z return func(*args, **kwargs) 2025-12-04T08:48:38.8097033Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T08:48:38.8097520Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:48:38.8097972Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8098413Z return func(*args, **kwargs) 2025-12-04T08:48:38.8098838Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 272, in forward 2025-12-04T08:48:38.8099296Z attn_output = self.out_proj(attn_output) 2025-12-04T08:48:38.8099474Z 2025-12-04T08:48:38.8099637Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:48:38.8100110Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:48:38.8100485Z res = mod(**inputs) 2025-12-04T08:48:38.8100941Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T08:48:38.8101388Z outputs = self.model.decoder( 2025-12-04T08:48:38.8101822Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T08:48:38.8102269Z layer_outputs = decoder_layer( 2025-12-04T08:48:38.8102658Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:48:38.8103082Z return super().__call__(*args, **kwargs) 2025-12-04T08:48:38.8103519Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8103943Z return func(*args, **kwargs) 2025-12-04T08:48:38.8104355Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 450, in forward 2025-12-04T08:48:38.8104898Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:48:38.8105098Z 2025-12-04T08:48:38.8105223Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:48:38.8105623Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:48:38.8105988Z res = mod(**inputs) 2025-12-04T08:48:38.8106390Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T08:48:38.8106824Z outputs = self.model.decoder( 2025-12-04T08:48:38.8107270Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T08:48:38.8107715Z layer_outputs = decoder_layer( 2025-12-04T08:48:38.8108112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:48:38.8108520Z return super().__call__(*args, **kwargs) 2025-12-04T08:48:38.8108955Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8109373Z return func(*args, **kwargs) 2025-12-04T08:48:38.8109822Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 450, in forward 2025-12-04T08:48:38.8110317Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:48:38.8110762Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T08:48:38.8111162Z return self.act(input) 2025-12-04T08:48:38.8111298Z 2025-12-04T08:48:38.8111424Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:48:38.8111822Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:48:38.8112195Z res = mod(**inputs) 2025-12-04T08:48:38.8112614Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T08:48:38.8113044Z outputs = self.model.decoder( 2025-12-04T08:48:38.8113488Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T08:48:38.8113932Z layer_outputs = decoder_layer( 2025-12-04T08:48:38.8114317Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:48:38.8114736Z return super().__call__(*args, **kwargs) 2025-12-04T08:48:38.8115161Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8115584Z return func(*args, **kwargs) 2025-12-04T08:48:38.8115991Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 452, in forward 2025-12-04T08:48:38.8116433Z hidden_states = self.fc2(hidden_states) 2025-12-04T08:48:38.8116587Z 2025-12-04T08:48:38.8116789Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:48:38.8117196Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:48:38.8117554Z res = mod(**inputs) 2025-12-04T08:48:38.8117959Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T08:48:38.8118568Z outputs = self.model.decoder( 2025-12-04T08:48:38.8118989Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T08:48:38.8119423Z layer_outputs = decoder_layer( 2025-12-04T08:48:38.8119829Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:48:38.8120311Z return super().__call__(*args, **kwargs) 2025-12-04T08:48:38.8120897Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8121338Z return func(*args, **kwargs) 2025-12-04T08:48:38.8121760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T08:48:38.8122218Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:48:38.8122686Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8123122Z return func(*args, **kwargs) 2025-12-04T08:48:38.8123545Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 219, in forward 2025-12-04T08:48:38.8124074Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T08:48:38.8124318Z 2025-12-04T08:48:38.8124444Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:48:38.8124862Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:48:38.8125231Z res = mod(**inputs) 2025-12-04T08:48:38.8125636Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T08:48:38.8126089Z outputs = self.model.decoder( 2025-12-04T08:48:38.8126515Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T08:48:38.8126948Z layer_outputs = decoder_layer( 2025-12-04T08:48:38.8127349Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:48:38.8127768Z return super().__call__(*args, **kwargs) 2025-12-04T08:48:38.8128207Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8128641Z return func(*args, **kwargs) 2025-12-04T08:48:38.8129072Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T08:48:38.8129545Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:48:38.8129995Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8130426Z return func(*args, **kwargs) 2025-12-04T08:48:38.8130855Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 239, in forward 2025-12-04T08:48:38.8131305Z key_states = self.k_proj(current_states) 2025-12-04T08:48:38.8131462Z 2025-12-04T08:48:38.8131580Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:48:38.8131993Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:48:38.8132370Z res = mod(**inputs) 2025-12-04T08:48:38.8132955Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T08:48:38.8133394Z outputs = self.model.decoder( 2025-12-04T08:48:38.8133822Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T08:48:38.8134260Z layer_outputs = decoder_layer( 2025-12-04T08:48:38.8134654Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:48:38.8135070Z return super().__call__(*args, **kwargs) 2025-12-04T08:48:38.8135501Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8135924Z return func(*args, **kwargs) 2025-12-04T08:48:38.8136334Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T08:48:38.8136866Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:48:38.8137322Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8137740Z return func(*args, **kwargs) 2025-12-04T08:48:38.8138156Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 240, in forward 2025-12-04T08:48:38.8138607Z value_states = self.v_proj(current_states) 2025-12-04T08:48:38.8138768Z 2025-12-04T08:48:38.8138865Z cudagraph partition due to non gpu ops 2025-12-04T08:48:38.8139127Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:48:38.8139536Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:48:38.8139905Z res = mod(**inputs) 2025-12-04T08:48:38.8140303Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T08:48:38.8140744Z outputs = self.model.decoder( 2025-12-04T08:48:38.8141171Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T08:48:38.8141604Z layer_outputs = decoder_layer( 2025-12-04T08:48:38.8141992Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:48:38.8142405Z return super().__call__(*args, **kwargs) 2025-12-04T08:48:38.8142830Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8143246Z return func(*args, **kwargs) 2025-12-04T08:48:38.8143646Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T08:48:38.8144268Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:48:38.8144753Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8145192Z return func(*args, **kwargs) 2025-12-04T08:48:38.8145616Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 258, in forward 2025-12-04T08:48:38.8146096Z attn_output, attn_weights = attention_interface( 2025-12-04T08:48:38.8146619Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T08:48:38.8147171Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T08:48:38.8147390Z 2025-12-04T08:48:38.8147508Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:48:38.8147920Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:48:38.8148307Z res = mod(**inputs) 2025-12-04T08:48:38.8148713Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T08:48:38.8149230Z outputs = self.model.decoder( 2025-12-04T08:48:38.8149667Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T08:48:38.8150103Z layer_outputs = decoder_layer( 2025-12-04T08:48:38.8150503Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:48:38.8150927Z return super().__call__(*args, **kwargs) 2025-12-04T08:48:38.8151355Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8151777Z return func(*args, **kwargs) 2025-12-04T08:48:38.8152188Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T08:48:38.8152703Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:48:38.8153160Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8153586Z return func(*args, **kwargs) 2025-12-04T08:48:38.8154007Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 272, in forward 2025-12-04T08:48:38.8154466Z attn_output = self.out_proj(attn_output) 2025-12-04T08:48:38.8154625Z 2025-12-04T08:48:38.8154745Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:48:38.8155154Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:48:38.8155533Z res = mod(**inputs) 2025-12-04T08:48:38.8155940Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T08:48:38.8156377Z outputs = self.model.decoder( 2025-12-04T08:48:38.8156811Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T08:48:38.8157251Z layer_outputs = decoder_layer( 2025-12-04T08:48:38.8157646Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:48:38.8158059Z return super().__call__(*args, **kwargs) 2025-12-04T08:48:38.8158578Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8159000Z return func(*args, **kwargs) 2025-12-04T08:48:38.8159410Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 450, in forward 2025-12-04T08:48:38.8159893Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:48:38.8160090Z 2025-12-04T08:48:38.8160216Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:48:38.8160627Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:48:38.8160985Z res = mod(**inputs) 2025-12-04T08:48:38.8161386Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T08:48:38.8161825Z outputs = self.model.decoder( 2025-12-04T08:48:38.8162244Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T08:48:38.8162679Z layer_outputs = decoder_layer( 2025-12-04T08:48:38.8163074Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:48:38.8163485Z return super().__call__(*args, **kwargs) 2025-12-04T08:48:38.8163910Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8164327Z return func(*args, **kwargs) 2025-12-04T08:48:38.8164807Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 450, in forward 2025-12-04T08:48:38.8165285Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:48:38.8165725Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T08:48:38.8166114Z return self.act(input) 2025-12-04T08:48:38.8166262Z 2025-12-04T08:48:38.8166386Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:48:38.8166783Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:48:38.8167145Z res = mod(**inputs) 2025-12-04T08:48:38.8167545Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T08:48:38.8167985Z outputs = self.model.decoder( 2025-12-04T08:48:38.8168411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T08:48:38.8168898Z layer_outputs = decoder_layer( 2025-12-04T08:48:38.8169296Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:48:38.8169712Z return super().__call__(*args, **kwargs) 2025-12-04T08:48:38.8170144Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8170579Z return func(*args, **kwargs) 2025-12-04T08:48:38.8171020Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 452, in forward 2025-12-04T08:48:38.8171470Z hidden_states = self.fc2(hidden_states) 2025-12-04T08:48:38.8171634Z 2025-12-04T08:48:38.8171750Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:48:38.8172151Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:48:38.8172511Z res = mod(**inputs) 2025-12-04T08:48:38.8172938Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T08:48:38.8173371Z outputs = self.model.decoder( 2025-12-04T08:48:38.8173809Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T08:48:38.8174241Z layer_outputs = decoder_layer( 2025-12-04T08:48:38.8174632Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:48:38.8175047Z return super().__call__(*args, **kwargs) 2025-12-04T08:48:38.8175468Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8175918Z return func(*args, **kwargs) 2025-12-04T08:48:38.8176321Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T08:48:38.8176779Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:48:38.8177212Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8177619Z return func(*args, **kwargs) 2025-12-04T08:48:38.8178021Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 219, in forward 2025-12-04T08:48:38.8178526Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T08:48:38.8178748Z 2025-12-04T08:48:38.8178861Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:48:38.8179252Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:48:38.8179607Z res = mod(**inputs) 2025-12-04T08:48:38.8179991Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T08:48:38.8180489Z outputs = self.model.decoder( 2025-12-04T08:48:38.8180902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T08:48:38.8181324Z layer_outputs = decoder_layer( 2025-12-04T08:48:38.8181702Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:48:38.8182100Z return super().__call__(*args, **kwargs) 2025-12-04T08:48:38.8182518Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8182918Z return func(*args, **kwargs) 2025-12-04T08:48:38.8183320Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T08:48:38.8183770Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:48:38.8184268Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8184682Z return func(*args, **kwargs) 2025-12-04T08:48:38.8185098Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 239, in forward 2025-12-04T08:48:38.8185543Z key_states = self.k_proj(current_states) 2025-12-04T08:48:38.8185695Z 2025-12-04T08:48:38.8185829Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:48:38.8186213Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:48:38.8186568Z res = mod(**inputs) 2025-12-04T08:48:38.8186958Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T08:48:38.8187383Z outputs = self.model.decoder( 2025-12-04T08:48:38.8187809Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T08:48:38.8188247Z layer_outputs = decoder_layer( 2025-12-04T08:48:38.8188637Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:48:38.8189039Z return super().__call__(*args, **kwargs) 2025-12-04T08:48:38.8189454Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8189912Z return func(*args, **kwargs) 2025-12-04T08:48:38.8190317Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T08:48:38.8190791Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:48:38.8191241Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8191661Z return func(*args, **kwargs) 2025-12-04T08:48:38.8192070Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 240, in forward 2025-12-04T08:48:38.8192531Z value_states = self.v_proj(current_states) 2025-12-04T08:48:38.8192688Z 2025-12-04T08:48:38.8192789Z cudagraph partition due to non gpu ops 2025-12-04T08:48:38.8193059Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:48:38.8193453Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:48:38.8193816Z res = mod(**inputs) 2025-12-04T08:48:38.8194218Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T08:48:38.8194663Z outputs = self.model.decoder( 2025-12-04T08:48:38.8195106Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T08:48:38.8195538Z layer_outputs = decoder_layer( 2025-12-04T08:48:38.8195985Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:48:38.8196392Z return super().__call__(*args, **kwargs) 2025-12-04T08:48:38.8196817Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8197250Z return func(*args, **kwargs) 2025-12-04T08:48:38.8197678Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T08:48:38.8198245Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:48:38.8198757Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8199183Z return func(*args, **kwargs) 2025-12-04T08:48:38.8199592Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 258, in forward 2025-12-04T08:48:38.8200108Z attn_output, attn_weights = attention_interface( 2025-12-04T08:48:38.8200619Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T08:48:38.8201169Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T08:48:38.8201376Z 2025-12-04T08:48:38.8201494Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:48:38.8201902Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:48:38.8202266Z res = mod(**inputs) 2025-12-04T08:48:38.8202660Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T08:48:38.8203093Z outputs = self.model.decoder( 2025-12-04T08:48:38.8203516Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T08:48:38.8203956Z layer_outputs = decoder_layer( 2025-12-04T08:48:38.8204348Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:48:38.8204759Z return super().__call__(*args, **kwargs) 2025-12-04T08:48:38.8205187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8205605Z return func(*args, **kwargs) 2025-12-04T08:48:38.8206017Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T08:48:38.8206478Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:48:38.8206924Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8207334Z return func(*args, **kwargs) 2025-12-04T08:48:38.8207751Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 272, in forward 2025-12-04T08:48:38.8208195Z attn_output = self.out_proj(attn_output) 2025-12-04T08:48:38.8208349Z 2025-12-04T08:48:38.8208472Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:48:38.8208867Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:48:38.8209227Z res = mod(**inputs) 2025-12-04T08:48:38.8209624Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T08:48:38.8210063Z outputs = self.model.decoder( 2025-12-04T08:48:38.8210645Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T08:48:38.8211111Z layer_outputs = decoder_layer( 2025-12-04T08:48:38.8211504Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:48:38.8211916Z return super().__call__(*args, **kwargs) 2025-12-04T08:48:38.8212402Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8212825Z return func(*args, **kwargs) 2025-12-04T08:48:38.8213231Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 450, in forward 2025-12-04T08:48:38.8213724Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:48:38.8213925Z 2025-12-04T08:48:38.8214042Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:48:38.8214445Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:48:38.8214808Z res = mod(**inputs) 2025-12-04T08:48:38.8215231Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T08:48:38.8215716Z outputs = self.model.decoder( 2025-12-04T08:48:38.8216161Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T08:48:38.8216583Z layer_outputs = decoder_layer( 2025-12-04T08:48:38.8216974Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:48:38.8217383Z return super().__call__(*args, **kwargs) 2025-12-04T08:48:38.8217804Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8218236Z return func(*args, **kwargs) 2025-12-04T08:48:38.8218701Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 450, in forward 2025-12-04T08:48:38.8219387Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:48:38.8219859Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T08:48:38.8220270Z return self.act(input) 2025-12-04T08:48:38.8220392Z 2025-12-04T08:48:38.8220515Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:48:38.8221052Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:48:38.8221605Z res = mod(**inputs) 2025-12-04T08:48:38.8222134Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T08:48:38.8222568Z outputs = self.model.decoder( 2025-12-04T08:48:38.8222970Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T08:48:38.8223390Z layer_outputs = decoder_layer( 2025-12-04T08:48:38.8223768Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:48:38.8224168Z return super().__call__(*args, **kwargs) 2025-12-04T08:48:38.8224575Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8224980Z return func(*args, **kwargs) 2025-12-04T08:48:38.8225384Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 452, in forward 2025-12-04T08:48:38.8225816Z hidden_states = self.fc2(hidden_states) 2025-12-04T08:48:38.8225978Z 2025-12-04T08:48:38.8226092Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:48:38.8226489Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:48:38.8226848Z res = mod(**inputs) 2025-12-04T08:48:38.8227239Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T08:48:38.8227672Z outputs = self.model.decoder( 2025-12-04T08:48:38.8228245Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T08:48:38.8228659Z layer_outputs = decoder_layer( 2025-12-04T08:48:38.8229045Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:48:38.8229443Z return super().__call__(*args, **kwargs) 2025-12-04T08:48:38.8229859Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8230272Z return func(*args, **kwargs) 2025-12-04T08:48:38.8230681Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T08:48:38.8231183Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:48:38.8231619Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8232108Z return func(*args, **kwargs) 2025-12-04T08:48:38.8232522Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 219, in forward 2025-12-04T08:48:38.8233030Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T08:48:38.8233253Z 2025-12-04T08:48:38.8233369Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:48:38.8233759Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:48:38.8234115Z res = mod(**inputs) 2025-12-04T08:48:38.8234509Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T08:48:38.8234951Z outputs = self.model.decoder( 2025-12-04T08:48:38.8235395Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T08:48:38.8235843Z layer_outputs = decoder_layer( 2025-12-04T08:48:38.8236237Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:48:38.8236650Z return super().__call__(*args, **kwargs) 2025-12-04T08:48:38.8237083Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8237514Z return func(*args, **kwargs) 2025-12-04T08:48:38.8237923Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T08:48:38.8238480Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:48:38.8238941Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8239372Z return func(*args, **kwargs) 2025-12-04T08:48:38.8239782Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 239, in forward 2025-12-04T08:48:38.8240235Z key_states = self.k_proj(current_states) 2025-12-04T08:48:38.8240394Z 2025-12-04T08:48:38.8240521Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:48:38.8240933Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:48:38.8241312Z res = mod(**inputs) 2025-12-04T08:48:38.8241717Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T08:48:38.8242157Z outputs = self.model.decoder( 2025-12-04T08:48:38.8242573Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T08:48:38.8243000Z layer_outputs = decoder_layer( 2025-12-04T08:48:38.8243397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:48:38.8243800Z return super().__call__(*args, **kwargs) 2025-12-04T08:48:38.8244284Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8244708Z return func(*args, **kwargs) 2025-12-04T08:48:38.8245140Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T08:48:38.8245594Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:48:38.8246045Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8246462Z return func(*args, **kwargs) 2025-12-04T08:48:38.8246877Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 240, in forward 2025-12-04T08:48:38.8247317Z value_states = self.v_proj(current_states) 2025-12-04T08:48:38.8247484Z 2025-12-04T08:48:38.8247622Z cudagraph partition due to non gpu ops 2025-12-04T08:48:38.8247894Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:48:38.8248300Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:48:38.8248670Z res = mod(**inputs) 2025-12-04T08:48:38.8249077Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T08:48:38.8249529Z outputs = self.model.decoder( 2025-12-04T08:48:38.8249946Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T08:48:38.8250380Z layer_outputs = decoder_layer( 2025-12-04T08:48:38.8250776Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:48:38.8251182Z return super().__call__(*args, **kwargs) 2025-12-04T08:48:38.8251607Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8252034Z return func(*args, **kwargs) 2025-12-04T08:48:38.8252449Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T08:48:38.8252912Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:48:38.8253365Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8253793Z return func(*args, **kwargs) 2025-12-04T08:48:38.8254193Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 258, in forward 2025-12-04T08:48:38.8254645Z attn_output, attn_weights = attention_interface( 2025-12-04T08:48:38.8255132Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T08:48:38.8255664Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T08:48:38.8255864Z 2025-12-04T08:48:38.8255979Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:48:38.8256372Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:48:38.8256726Z res = mod(**inputs) 2025-12-04T08:48:38.8257118Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T08:48:38.8257546Z outputs = self.model.decoder( 2025-12-04T08:48:38.8257959Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T08:48:38.8258390Z layer_outputs = decoder_layer( 2025-12-04T08:48:38.8258764Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:48:38.8259171Z return super().__call__(*args, **kwargs) 2025-12-04T08:48:38.8259650Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8260075Z return func(*args, **kwargs) 2025-12-04T08:48:38.8260484Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T08:48:38.8260943Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:48:38.8261386Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8261801Z return func(*args, **kwargs) 2025-12-04T08:48:38.8262208Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 272, in forward 2025-12-04T08:48:38.8262659Z attn_output = self.out_proj(attn_output) 2025-12-04T08:48:38.8262810Z 2025-12-04T08:48:38.8262931Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:48:38.8263368Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:48:38.8263724Z res = mod(**inputs) 2025-12-04T08:48:38.8264111Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T08:48:38.8264538Z outputs = self.model.decoder( 2025-12-04T08:48:38.8264949Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T08:48:38.8265389Z layer_outputs = decoder_layer( 2025-12-04T08:48:38.8265774Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:48:38.8266162Z return super().__call__(*args, **kwargs) 2025-12-04T08:48:38.8266603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8267015Z return func(*args, **kwargs) 2025-12-04T08:48:38.8267425Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 450, in forward 2025-12-04T08:48:38.8267891Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:48:38.8268085Z 2025-12-04T08:48:38.8268198Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:48:38.8268595Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:48:38.8268947Z res = mod(**inputs) 2025-12-04T08:48:38.8269344Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T08:48:38.8269770Z outputs = self.model.decoder( 2025-12-04T08:48:38.8270185Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T08:48:38.8270621Z layer_outputs = decoder_layer( 2025-12-04T08:48:38.8271022Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:48:38.8271434Z return super().__call__(*args, **kwargs) 2025-12-04T08:48:38.8271860Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8272286Z return func(*args, **kwargs) 2025-12-04T08:48:38.8272704Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 450, in forward 2025-12-04T08:48:38.8273179Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:48:38.8273609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T08:48:38.8274006Z return self.act(input) 2025-12-04T08:48:38.8274134Z 2025-12-04T08:48:38.8274250Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:48:38.8274650Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:48:38.8275010Z res = mod(**inputs) 2025-12-04T08:48:38.8275466Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T08:48:38.8275904Z outputs = self.model.decoder( 2025-12-04T08:48:38.8276343Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T08:48:38.8276780Z layer_outputs = decoder_layer( 2025-12-04T08:48:38.8277173Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:48:38.8277582Z return super().__call__(*args, **kwargs) 2025-12-04T08:48:38.8278003Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8278522Z return func(*args, **kwargs) 2025-12-04T08:48:38.8279031Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 452, in forward 2025-12-04T08:48:38.8279474Z hidden_states = self.fc2(hidden_states) 2025-12-04T08:48:38.8279632Z 2025-12-04T08:48:38.8279750Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:48:38.8280155Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:48:38.8280523Z res = mod(**inputs) 2025-12-04T08:48:38.8280916Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T08:48:38.8281351Z outputs = self.model.decoder( 2025-12-04T08:48:38.8281782Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T08:48:38.8282216Z layer_outputs = decoder_layer( 2025-12-04T08:48:38.8282603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:48:38.8283015Z return super().__call__(*args, **kwargs) 2025-12-04T08:48:38.8283443Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8283862Z return func(*args, **kwargs) 2025-12-04T08:48:38.8284269Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T08:48:38.8284728Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:48:38.8285174Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8285582Z return func(*args, **kwargs) 2025-12-04T08:48:38.8285997Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 219, in forward 2025-12-04T08:48:38.8286514Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T08:48:38.8286744Z 2025-12-04T08:48:38.8286871Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:48:38.8287267Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:48:38.8287630Z res = mod(**inputs) 2025-12-04T08:48:38.8288031Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T08:48:38.8288457Z outputs = self.model.decoder( 2025-12-04T08:48:38.8288880Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T08:48:38.8289311Z layer_outputs = decoder_layer( 2025-12-04T08:48:38.8289705Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:48:38.8290104Z return super().__call__(*args, **kwargs) 2025-12-04T08:48:38.8290536Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8291012Z return func(*args, **kwargs) 2025-12-04T08:48:38.8291431Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T08:48:38.8291897Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:48:38.8292349Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8292777Z return func(*args, **kwargs) 2025-12-04T08:48:38.8293184Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 239, in forward 2025-12-04T08:48:38.8293633Z key_states = self.k_proj(current_states) 2025-12-04T08:48:38.8293791Z 2025-12-04T08:48:38.8293908Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:48:38.8294308Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:48:38.8294730Z res = mod(**inputs) 2025-12-04T08:48:38.8295137Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T08:48:38.8295583Z outputs = self.model.decoder( 2025-12-04T08:48:38.8296017Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T08:48:38.8296460Z layer_outputs = decoder_layer( 2025-12-04T08:48:38.8296852Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:48:38.8297264Z return super().__call__(*args, **kwargs) 2025-12-04T08:48:38.8297687Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8298121Z return func(*args, **kwargs) 2025-12-04T08:48:38.8298547Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T08:48:38.8299021Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:48:38.8299470Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8299898Z return func(*args, **kwargs) 2025-12-04T08:48:38.8300331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 240, in forward 2025-12-04T08:48:38.8300782Z value_states = self.v_proj(current_states) 2025-12-04T08:48:38.8300949Z 2025-12-04T08:48:38.8301041Z cudagraph partition due to non gpu ops 2025-12-04T08:48:38.8301313Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:48:38.8301715Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:48:38.8302073Z res = mod(**inputs) 2025-12-04T08:48:38.8302492Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T08:48:38.8302925Z outputs = self.model.decoder( 2025-12-04T08:48:38.8303340Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T08:48:38.8303771Z layer_outputs = decoder_layer( 2025-12-04T08:48:38.8304162Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:48:38.8304580Z return super().__call__(*args, **kwargs) 2025-12-04T08:48:38.8304999Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8305417Z return func(*args, **kwargs) 2025-12-04T08:48:38.8305834Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T08:48:38.8306298Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:48:38.8306786Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8307206Z return func(*args, **kwargs) 2025-12-04T08:48:38.8307623Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 258, in forward 2025-12-04T08:48:38.8308076Z attn_output, attn_weights = attention_interface( 2025-12-04T08:48:38.8308581Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T08:48:38.8309129Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T08:48:38.8309335Z 2025-12-04T08:48:38.8309460Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:48:38.8309855Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:48:38.8310267Z res = mod(**inputs) 2025-12-04T08:48:38.8310675Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T08:48:38.8311108Z outputs = self.model.decoder( 2025-12-04T08:48:38.8311532Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T08:48:38.8311971Z layer_outputs = decoder_layer( 2025-12-04T08:48:38.8312361Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:48:38.8312763Z return super().__call__(*args, **kwargs) 2025-12-04T08:48:38.8313190Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8313609Z return func(*args, **kwargs) 2025-12-04T08:48:38.8314027Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T08:48:38.8314489Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:48:38.8314935Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8315355Z return func(*args, **kwargs) 2025-12-04T08:48:38.8315764Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 272, in forward 2025-12-04T08:48:38.8316207Z attn_output = self.out_proj(attn_output) 2025-12-04T08:48:38.8316368Z 2025-12-04T08:48:38.8316483Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:48:38.8316888Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:48:38.8317244Z res = mod(**inputs) 2025-12-04T08:48:38.8317646Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T08:48:38.8318086Z outputs = self.model.decoder( 2025-12-04T08:48:38.8318594Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T08:48:38.8319036Z layer_outputs = decoder_layer( 2025-12-04T08:48:38.8319434Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:48:38.8319843Z return super().__call__(*args, **kwargs) 2025-12-04T08:48:38.8320263Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8320810Z return func(*args, **kwargs) 2025-12-04T08:48:38.8321239Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 450, in forward 2025-12-04T08:48:38.8321731Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:48:38.8321933Z 2025-12-04T08:48:38.8322051Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:48:38.8322574Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:48:38.8322959Z res = mod(**inputs) 2025-12-04T08:48:38.8323355Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T08:48:38.8323794Z outputs = self.model.decoder( 2025-12-04T08:48:38.8324229Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T08:48:38.8324649Z layer_outputs = decoder_layer( 2025-12-04T08:48:38.8325027Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:48:38.8325425Z return super().__call__(*args, **kwargs) 2025-12-04T08:48:38.8325841Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8326299Z return func(*args, **kwargs) 2025-12-04T08:48:38.8326702Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 450, in forward 2025-12-04T08:48:38.8327172Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:48:38.8327612Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T08:48:38.8327996Z return self.act(input) 2025-12-04T08:48:38.8328126Z 2025-12-04T08:48:38.8328242Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:48:38.8328647Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:48:38.8329010Z res = mod(**inputs) 2025-12-04T08:48:38.8329407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T08:48:38.8329834Z outputs = self.model.decoder( 2025-12-04T08:48:38.8330248Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T08:48:38.8330673Z layer_outputs = decoder_layer( 2025-12-04T08:48:38.8331073Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:48:38.8331471Z return super().__call__(*args, **kwargs) 2025-12-04T08:48:38.8331896Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8332359Z return func(*args, **kwargs) 2025-12-04T08:48:38.8332997Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 452, in forward 2025-12-04T08:48:38.8333455Z hidden_states = self.fc2(hidden_states) 2025-12-04T08:48:38.8333609Z 2025-12-04T08:48:38.8333720Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:48:38.8334121Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:48:38.8334473Z res = mod(**inputs) 2025-12-04T08:48:38.8334866Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T08:48:38.8335291Z outputs = self.model.decoder( 2025-12-04T08:48:38.8335701Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T08:48:38.8336137Z layer_outputs = decoder_layer( 2025-12-04T08:48:38.8336523Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:48:38.8336915Z return super().__call__(*args, **kwargs) 2025-12-04T08:48:38.8337336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8337760Z return func(*args, **kwargs) 2025-12-04T08:48:38.8338258Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T08:48:38.8338721Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:48:38.8339171Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8339600Z return func(*args, **kwargs) 2025-12-04T08:48:38.8340009Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 219, in forward 2025-12-04T08:48:38.8340526Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T08:48:38.8340755Z 2025-12-04T08:48:38.8340880Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:48:38.8341286Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:48:38.8341688Z res = mod(**inputs) 2025-12-04T08:48:38.8342092Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T08:48:38.8342527Z outputs = self.model.decoder( 2025-12-04T08:48:38.8342944Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T08:48:38.8343379Z layer_outputs = decoder_layer( 2025-12-04T08:48:38.8343771Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:48:38.8344181Z return super().__call__(*args, **kwargs) 2025-12-04T08:48:38.8344601Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8345018Z return func(*args, **kwargs) 2025-12-04T08:48:38.8345432Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T08:48:38.8345890Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:48:38.8346346Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8346766Z return func(*args, **kwargs) 2025-12-04T08:48:38.8347182Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 239, in forward 2025-12-04T08:48:38.8347616Z key_states = self.k_proj(current_states) 2025-12-04T08:48:38.8347775Z 2025-12-04T08:48:38.8347890Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:48:38.8348291Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:48:38.8348646Z res = mod(**inputs) 2025-12-04T08:48:38.8349044Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T08:48:38.8349478Z outputs = self.model.decoder( 2025-12-04T08:48:38.8349903Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T08:48:38.8350326Z layer_outputs = decoder_layer( 2025-12-04T08:48:38.8350716Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:48:38.8351123Z return super().__call__(*args, **kwargs) 2025-12-04T08:48:38.8351545Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8351952Z return func(*args, **kwargs) 2025-12-04T08:48:38.8352364Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T08:48:38.8352834Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:48:38.8353272Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8353694Z return func(*args, **kwargs) 2025-12-04T08:48:38.8354182Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 240, in forward 2025-12-04T08:48:38.8354632Z value_states = self.v_proj(current_states) 2025-12-04T08:48:38.8354793Z 2025-12-04T08:48:38.8354889Z cudagraph partition due to non gpu ops 2025-12-04T08:48:38.8355167Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:48:38.8355573Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:48:38.8355940Z res = mod(**inputs) 2025-12-04T08:48:38.8356363Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T08:48:38.8356819Z outputs = self.model.decoder( 2025-12-04T08:48:38.8357264Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T08:48:38.8357730Z layer_outputs = decoder_layer( 2025-12-04T08:48:38.8358131Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:48:38.8358623Z return super().__call__(*args, **kwargs) 2025-12-04T08:48:38.8359048Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8359491Z return func(*args, **kwargs) 2025-12-04T08:48:38.8359912Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T08:48:38.8360390Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:48:38.8360841Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8361278Z return func(*args, **kwargs) 2025-12-04T08:48:38.8361711Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 258, in forward 2025-12-04T08:48:38.8362179Z attn_output, attn_weights = attention_interface( 2025-12-04T08:48:38.8362682Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T08:48:38.8363230Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T08:48:38.8363439Z 2025-12-04T08:48:38.8363757Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:48:38.8364176Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:48:38.8364526Z res = mod(**inputs) 2025-12-04T08:48:38.8364922Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T08:48:38.8365362Z outputs = self.model.decoder( 2025-12-04T08:48:38.8365789Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T08:48:38.8366230Z layer_outputs = decoder_layer( 2025-12-04T08:48:38.8366645Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:48:38.8367048Z return super().__call__(*args, **kwargs) 2025-12-04T08:48:38.8367460Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8367871Z return func(*args, **kwargs) 2025-12-04T08:48:38.8368277Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T08:48:38.8368720Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:48:38.8369182Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8369611Z return func(*args, **kwargs) 2025-12-04T08:48:38.8370077Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 272, in forward 2025-12-04T08:48:38.8370517Z attn_output = self.out_proj(attn_output) 2025-12-04T08:48:38.8370679Z 2025-12-04T08:48:38.8370795Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:48:38.8371200Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:48:38.8371559Z res = mod(**inputs) 2025-12-04T08:48:38.8371953Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T08:48:38.8372378Z outputs = self.model.decoder( 2025-12-04T08:48:38.8372802Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T08:48:38.8373228Z layer_outputs = decoder_layer( 2025-12-04T08:48:38.8373684Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:48:38.8374090Z return super().__call__(*args, **kwargs) 2025-12-04T08:48:38.8374512Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8374920Z return func(*args, **kwargs) 2025-12-04T08:48:38.8375351Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 450, in forward 2025-12-04T08:48:38.8375840Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:48:38.8376032Z 2025-12-04T08:48:38.8376151Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:48:38.8376540Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:48:38.8376895Z res = mod(**inputs) 2025-12-04T08:48:38.8377305Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T08:48:38.8377737Z outputs = self.model.decoder( 2025-12-04T08:48:38.8378160Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T08:48:38.8378603Z layer_outputs = decoder_layer( 2025-12-04T08:48:38.8379013Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:48:38.8379425Z return super().__call__(*args, **kwargs) 2025-12-04T08:48:38.8379861Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8380303Z return func(*args, **kwargs) 2025-12-04T08:48:38.8380739Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 450, in forward 2025-12-04T08:48:38.8381245Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:48:38.8381698Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T08:48:38.8382092Z return self.act(input) 2025-12-04T08:48:38.8382216Z 2025-12-04T08:48:38.8382331Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:48:38.8382744Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:48:38.8383114Z res = mod(**inputs) 2025-12-04T08:48:38.8383529Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T08:48:38.8383982Z outputs = self.model.decoder( 2025-12-04T08:48:38.8384429Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T08:48:38.8384866Z layer_outputs = decoder_layer( 2025-12-04T08:48:38.8385259Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:48:38.8385725Z return super().__call__(*args, **kwargs) 2025-12-04T08:48:38.8386156Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8386575Z return func(*args, **kwargs) 2025-12-04T08:48:38.8386981Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 452, in forward 2025-12-04T08:48:38.8387418Z hidden_states = self.fc2(hidden_states) 2025-12-04T08:48:38.8387573Z 2025-12-04T08:48:38.8387697Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:48:38.8388092Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:48:38.8388456Z res = mod(**inputs) 2025-12-04T08:48:38.8388854Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T08:48:38.8389326Z outputs = self.model.decoder( 2025-12-04T08:48:38.8389742Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T08:48:38.8390169Z layer_outputs = decoder_layer( 2025-12-04T08:48:38.8390560Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:48:38.8390957Z return super().__call__(*args, **kwargs) 2025-12-04T08:48:38.8391379Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8391795Z return func(*args, **kwargs) 2025-12-04T08:48:38.8392207Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T08:48:38.8392658Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:48:38.8393112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8393552Z return func(*args, **kwargs) 2025-12-04T08:48:38.8393964Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 219, in forward 2025-12-04T08:48:38.8394468Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T08:48:38.8394704Z 2025-12-04T08:48:38.8394820Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:48:38.8395219Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:48:38.8395583Z res = mod(**inputs) 2025-12-04T08:48:38.8395988Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T08:48:38.8396419Z outputs = self.model.decoder( 2025-12-04T08:48:38.8396844Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T08:48:38.8397274Z layer_outputs = decoder_layer( 2025-12-04T08:48:38.8397668Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:48:38.8398086Z return super().__call__(*args, **kwargs) 2025-12-04T08:48:38.8398597Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8399030Z return func(*args, **kwargs) 2025-12-04T08:48:38.8399445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T08:48:38.8399923Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:48:38.8400369Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8400800Z return func(*args, **kwargs) 2025-12-04T08:48:38.8401286Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 239, in forward 2025-12-04T08:48:38.8401727Z key_states = self.k_proj(current_states) 2025-12-04T08:48:38.8401882Z 2025-12-04T08:48:38.8402002Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:48:38.8402405Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:48:38.8402769Z res = mod(**inputs) 2025-12-04T08:48:38.8403149Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T08:48:38.8403581Z outputs = self.model.decoder( 2025-12-04T08:48:38.8404021Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T08:48:38.8404465Z layer_outputs = decoder_layer( 2025-12-04T08:48:38.8404847Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:48:38.8405303Z return super().__call__(*args, **kwargs) 2025-12-04T08:48:38.8405720Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8406121Z return func(*args, **kwargs) 2025-12-04T08:48:38.8406526Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T08:48:38.8406977Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:48:38.8407418Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8407816Z return func(*args, **kwargs) 2025-12-04T08:48:38.8408217Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 240, in forward 2025-12-04T08:48:38.8408656Z value_states = self.v_proj(current_states) 2025-12-04T08:48:38.8408809Z 2025-12-04T08:48:38.8408904Z cudagraph partition due to non gpu ops 2025-12-04T08:48:38.8409162Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:48:38.8409551Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:48:38.8409902Z res = mod(**inputs) 2025-12-04T08:48:38.8410284Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T08:48:38.8410705Z outputs = self.model.decoder( 2025-12-04T08:48:38.8411116Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T08:48:38.8411536Z layer_outputs = decoder_layer( 2025-12-04T08:48:38.8411911Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:48:38.8412313Z return super().__call__(*args, **kwargs) 2025-12-04T08:48:38.8412730Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8413129Z return func(*args, **kwargs) 2025-12-04T08:48:38.8413531Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T08:48:38.8413981Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:48:38.8414419Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8414821Z return func(*args, **kwargs) 2025-12-04T08:48:38.8415225Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 258, in forward 2025-12-04T08:48:38.8415676Z attn_output, attn_weights = attention_interface( 2025-12-04T08:48:38.8416169Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T08:48:38.8416751Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T08:48:38.8416961Z 2025-12-04T08:48:38.8417074Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:48:38.8417470Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:48:38.8417817Z res = mod(**inputs) 2025-12-04T08:48:38.8418296Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T08:48:38.8418843Z outputs = self.model.decoder( 2025-12-04T08:48:38.8419643Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T08:48:38.8420136Z layer_outputs = decoder_layer( 2025-12-04T08:48:38.8420620Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:48:38.8421587Z return super().__call__(*args, **kwargs) 2025-12-04T08:48:38.8422091Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8422596Z return func(*args, **kwargs) 2025-12-04T08:48:38.8423131Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T08:48:38.8423689Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:48:38.8424194Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8424728Z return func(*args, **kwargs) 2025-12-04T08:48:38.8425247Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 272, in forward 2025-12-04T08:48:38.8425768Z attn_output = self.out_proj(attn_output) 2025-12-04T08:48:38.8425977Z 2025-12-04T08:48:38.8426136Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:48:38.8426629Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:48:38.8427086Z res = mod(**inputs) 2025-12-04T08:48:38.8427606Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T08:48:38.8428101Z outputs = self.model.decoder( 2025-12-04T08:48:38.8428602Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T08:48:38.8429147Z layer_outputs = decoder_layer( 2025-12-04T08:48:38.8429590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:48:38.8430082Z return super().__call__(*args, **kwargs) 2025-12-04T08:48:38.8430649Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8431160Z return func(*args, **kwargs) 2025-12-04T08:48:38.8431610Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 450, in forward 2025-12-04T08:48:38.8432209Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:48:38.8432471Z 2025-12-04T08:48:38.8432616Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:48:38.8433119Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:48:38.8433546Z res = mod(**inputs) 2025-12-04T08:48:38.8434040Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T08:48:38.8434562Z outputs = self.model.decoder( 2025-12-04T08:48:38.8435085Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T08:48:38.8435571Z layer_outputs = decoder_layer( 2025-12-04T08:48:38.8436164Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:48:38.8453508Z return super().__call__(*args, **kwargs) 2025-12-04T08:48:38.8453867Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8453960Z return func(*args, **kwargs) 2025-12-04T08:48:38.8454262Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 450, in forward 2025-12-04T08:48:38.8454423Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:48:38.8454675Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T08:48:38.8454763Z return self.act(input) 2025-12-04T08:48:38.8454778Z 2025-12-04T08:48:38.8454904Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:48:38.8455349Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:48:38.8455434Z res = mod(**inputs) 2025-12-04T08:48:38.8455722Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T08:48:38.8455813Z outputs = self.model.decoder( 2025-12-04T08:48:38.8456102Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T08:48:38.8456187Z layer_outputs = decoder_layer( 2025-12-04T08:48:38.8456448Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:48:38.8456543Z return super().__call__(*args, **kwargs) 2025-12-04T08:48:38.8456815Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8456911Z return func(*args, **kwargs) 2025-12-04T08:48:38.8457191Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 452, in forward 2025-12-04T08:48:38.8457283Z hidden_states = self.fc2(hidden_states) 2025-12-04T08:48:38.8457288Z 2025-12-04T08:48:38.8457418Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:48:38.8457643Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:48:38.8457726Z res = mod(**inputs) 2025-12-04T08:48:38.8458132Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T08:48:38.8458233Z outputs = self.model.decoder( 2025-12-04T08:48:38.8458686Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T08:48:38.8458794Z layer_outputs = decoder_layer( 2025-12-04T08:48:38.8459050Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:48:38.8459148Z return super().__call__(*args, **kwargs) 2025-12-04T08:48:38.8459425Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8459511Z return func(*args, **kwargs) 2025-12-04T08:48:38.8459782Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T08:48:38.8459896Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:48:38.8460169Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8460245Z return func(*args, **kwargs) 2025-12-04T08:48:38.8460520Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 219, in forward 2025-12-04T08:48:38.8460694Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T08:48:38.8460757Z 2025-12-04T08:48:38.8460874Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:48:38.8461099Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:48:38.8461170Z res = mod(**inputs) 2025-12-04T08:48:38.8461440Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T08:48:38.8461529Z outputs = self.model.decoder( 2025-12-04T08:48:38.8461797Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T08:48:38.8461882Z layer_outputs = decoder_layer( 2025-12-04T08:48:38.8462124Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:48:38.8462250Z return super().__call__(*args, **kwargs) 2025-12-04T08:48:38.8462523Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8462597Z return func(*args, **kwargs) 2025-12-04T08:48:38.8462881Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T08:48:38.8462995Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:48:38.8463262Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8463348Z return func(*args, **kwargs) 2025-12-04T08:48:38.8463622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 239, in forward 2025-12-04T08:48:38.8463712Z key_states = self.k_proj(current_states) 2025-12-04T08:48:38.8463716Z 2025-12-04T08:48:38.8463842Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:48:38.8464068Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:48:38.8464148Z res = mod(**inputs) 2025-12-04T08:48:38.8464423Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T08:48:38.8464505Z outputs = self.model.decoder( 2025-12-04T08:48:38.8464789Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T08:48:38.8464870Z layer_outputs = decoder_layer( 2025-12-04T08:48:38.8465123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:48:38.8465209Z return super().__call__(*args, **kwargs) 2025-12-04T08:48:38.8465479Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8465566Z return func(*args, **kwargs) 2025-12-04T08:48:38.8465846Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T08:48:38.8465967Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:48:38.8466236Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8466311Z return func(*args, **kwargs) 2025-12-04T08:48:38.8466600Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 240, in forward 2025-12-04T08:48:38.8466697Z value_states = self.v_proj(current_states) 2025-12-04T08:48:38.8466702Z 2025-12-04T08:48:38.8466793Z cudagraph partition due to non gpu ops 2025-12-04T08:48:38.8466916Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:48:38.8467136Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:48:38.8467213Z res = mod(**inputs) 2025-12-04T08:48:38.8467536Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T08:48:38.8467620Z outputs = self.model.decoder( 2025-12-04T08:48:38.8467909Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T08:48:38.8467989Z layer_outputs = decoder_layer( 2025-12-04T08:48:38.8468239Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:48:38.8468333Z return super().__call__(*args, **kwargs) 2025-12-04T08:48:38.8468605Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8468688Z return func(*args, **kwargs) 2025-12-04T08:48:38.8469080Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T08:48:38.8469275Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:48:38.8469706Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8469785Z return func(*args, **kwargs) 2025-12-04T08:48:38.8470065Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 258, in forward 2025-12-04T08:48:38.8470186Z attn_output, attn_weights = attention_interface( 2025-12-04T08:48:38.8470517Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T08:48:38.8470679Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T08:48:38.8470684Z 2025-12-04T08:48:38.8470798Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:48:38.8471026Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:48:38.8471110Z res = mod(**inputs) 2025-12-04T08:48:38.8471390Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T08:48:38.8471479Z outputs = self.model.decoder( 2025-12-04T08:48:38.8471758Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T08:48:38.8471839Z layer_outputs = decoder_layer( 2025-12-04T08:48:38.8472098Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:48:38.8472186Z return super().__call__(*args, **kwargs) 2025-12-04T08:48:38.8472458Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8472541Z return func(*args, **kwargs) 2025-12-04T08:48:38.8472826Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T08:48:38.8472943Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:48:38.8473213Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8473289Z return func(*args, **kwargs) 2025-12-04T08:48:38.8473577Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 272, in forward 2025-12-04T08:48:38.8473666Z attn_output = self.out_proj(attn_output) 2025-12-04T08:48:38.8473670Z 2025-12-04T08:48:38.8473781Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:48:38.8474011Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:48:38.8474081Z res = mod(**inputs) 2025-12-04T08:48:38.8474373Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T08:48:38.8474512Z outputs = self.model.decoder( 2025-12-04T08:48:38.8474794Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T08:48:38.8474883Z layer_outputs = decoder_layer( 2025-12-04T08:48:38.8475131Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:48:38.8475226Z return super().__call__(*args, **kwargs) 2025-12-04T08:48:38.8475497Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8475573Z return func(*args, **kwargs) 2025-12-04T08:48:38.8475855Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 450, in forward 2025-12-04T08:48:38.8476032Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:48:38.8476036Z 2025-12-04T08:48:38.8476154Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:48:38.8476382Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:48:38.8476456Z res = mod(**inputs) 2025-12-04T08:48:38.8476741Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T08:48:38.8476821Z outputs = self.model.decoder( 2025-12-04T08:48:38.8477097Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T08:48:38.8477188Z layer_outputs = decoder_layer( 2025-12-04T08:48:38.8477433Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:48:38.8477529Z return super().__call__(*args, **kwargs) 2025-12-04T08:48:38.8477806Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8477886Z return func(*args, **kwargs) 2025-12-04T08:48:38.8478266Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 450, in forward 2025-12-04T08:48:38.8478416Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:48:38.8478661Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T08:48:38.8478749Z return self.act(input) 2025-12-04T08:48:38.8478754Z 2025-12-04T08:48:38.8478868Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:48:38.8479096Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:48:38.8479167Z res = mod(**inputs) 2025-12-04T08:48:38.8479447Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T08:48:38.8479541Z outputs = self.model.decoder( 2025-12-04T08:48:38.8479830Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T08:48:38.8479911Z layer_outputs = decoder_layer( 2025-12-04T08:48:38.8480163Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:48:38.8480250Z return super().__call__(*args, **kwargs) 2025-12-04T08:48:38.8480531Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8480607Z return func(*args, **kwargs) 2025-12-04T08:48:38.8480912Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 452, in forward 2025-12-04T08:48:38.8481014Z hidden_states = self.fc2(hidden_states) 2025-12-04T08:48:38.8481021Z 2025-12-04T08:48:38.8481132Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:48:38.8481402Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:48:38.8481475Z res = mod(**inputs) 2025-12-04T08:48:38.8481754Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T08:48:38.8481843Z outputs = self.model.decoder( 2025-12-04T08:48:38.8482121Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T08:48:38.8482201Z layer_outputs = decoder_layer( 2025-12-04T08:48:38.8482456Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:48:38.8482543Z return super().__call__(*args, **kwargs) 2025-12-04T08:48:38.8482823Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8482943Z return func(*args, **kwargs) 2025-12-04T08:48:38.8483228Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T08:48:38.8483348Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:48:38.8483623Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8483700Z return func(*args, **kwargs) 2025-12-04T08:48:38.8484010Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 219, in forward 2025-12-04T08:48:38.8484182Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T08:48:38.8484187Z 2025-12-04T08:48:38.8484308Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:48:38.8484529Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:48:38.8484604Z res = mod(**inputs) 2025-12-04T08:48:38.8484901Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T08:48:38.8484980Z outputs = self.model.decoder( 2025-12-04T08:48:38.8485269Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T08:48:38.8485348Z layer_outputs = decoder_layer( 2025-12-04T08:48:38.8485599Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:48:38.8485696Z return super().__call__(*args, **kwargs) 2025-12-04T08:48:38.8485971Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8486046Z return func(*args, **kwargs) 2025-12-04T08:48:38.8486335Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T08:48:38.8486447Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:48:38.8486733Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8486806Z return func(*args, **kwargs) 2025-12-04T08:48:38.8487075Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 239, in forward 2025-12-04T08:48:38.8487173Z key_states = self.k_proj(current_states) 2025-12-04T08:48:38.8487176Z 2025-12-04T08:48:38.8487287Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:48:38.8487502Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:48:38.8487589Z res = mod(**inputs) 2025-12-04T08:48:38.8487863Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T08:48:38.8487953Z outputs = self.model.decoder( 2025-12-04T08:48:38.8488260Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T08:48:38.8488340Z layer_outputs = decoder_layer( 2025-12-04T08:48:38.8488588Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:48:38.8488672Z return super().__call__(*args, **kwargs) 2025-12-04T08:48:38.8488935Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8489015Z return func(*args, **kwargs) 2025-12-04T08:48:38.8489283Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T08:48:38.8489395Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:48:38.8489701Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8489773Z return func(*args, **kwargs) 2025-12-04T08:48:38.8490049Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 240, in forward 2025-12-04T08:48:38.8490142Z value_states = self.v_proj(current_states) 2025-12-04T08:48:38.8490146Z 2025-12-04T08:48:38.8490235Z cudagraph partition due to non gpu ops 2025-12-04T08:48:38.8490352Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:48:38.8490565Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:48:38.8490640Z res = mod(**inputs) 2025-12-04T08:48:38.8490908Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T08:48:38.8490986Z outputs = self.model.decoder( 2025-12-04T08:48:38.8491268Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T08:48:38.8491346Z layer_outputs = decoder_layer( 2025-12-04T08:48:38.8491592Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:48:38.8491675Z return super().__call__(*args, **kwargs) 2025-12-04T08:48:38.8491936Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8492015Z return func(*args, **kwargs) 2025-12-04T08:48:38.8492284Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T08:48:38.8492388Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:48:38.8492654Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8492730Z return func(*args, **kwargs) 2025-12-04T08:48:38.8493008Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 258, in forward 2025-12-04T08:48:38.8493116Z attn_output, attn_weights = attention_interface( 2025-12-04T08:48:38.8493439Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T08:48:38.8493597Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T08:48:38.8493601Z 2025-12-04T08:48:38.8493712Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:48:38.8493939Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:48:38.8494010Z res = mod(**inputs) 2025-12-04T08:48:38.8494289Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T08:48:38.8494379Z outputs = self.model.decoder( 2025-12-04T08:48:38.8494716Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T08:48:38.8494808Z layer_outputs = decoder_layer( 2025-12-04T08:48:38.8495058Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:48:38.8495145Z return super().__call__(*args, **kwargs) 2025-12-04T08:48:38.8495416Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8495491Z return func(*args, **kwargs) 2025-12-04T08:48:38.8495760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T08:48:38.8495877Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:48:38.8496145Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8496277Z return func(*args, **kwargs) 2025-12-04T08:48:38.8496565Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 272, in forward 2025-12-04T08:48:38.8496665Z attn_output = self.out_proj(attn_output) 2025-12-04T08:48:38.8496669Z 2025-12-04T08:48:38.8496795Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:48:38.8497011Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:48:38.8497081Z res = mod(**inputs) 2025-12-04T08:48:38.8497359Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T08:48:38.8497439Z outputs = self.model.decoder( 2025-12-04T08:48:38.8497716Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T08:48:38.8497798Z layer_outputs = decoder_layer( 2025-12-04T08:48:38.8498042Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:48:38.8498136Z return super().__call__(*args, **kwargs) 2025-12-04T08:48:38.8498398Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8498473Z return func(*args, **kwargs) 2025-12-04T08:48:38.8498747Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 450, in forward 2025-12-04T08:48:38.8498873Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:48:38.8498877Z 2025-12-04T08:48:38.8498992Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:48:38.8499206Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:48:38.8499280Z res = mod(**inputs) 2025-12-04T08:48:38.8499562Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T08:48:38.8499641Z outputs = self.model.decoder( 2025-12-04T08:48:38.8499919Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T08:48:38.8499998Z layer_outputs = decoder_layer( 2025-12-04T08:48:38.8500235Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:48:38.8500326Z return super().__call__(*args, **kwargs) 2025-12-04T08:48:38.8500590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8500663Z return func(*args, **kwargs) 2025-12-04T08:48:38.8500935Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 450, in forward 2025-12-04T08:48:38.8501063Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:48:38.8501343Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T08:48:38.8501423Z return self.act(input) 2025-12-04T08:48:38.8501427Z 2025-12-04T08:48:38.8501538Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:48:38.8501763Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:48:38.8501834Z res = mod(**inputs) 2025-12-04T08:48:38.8502108Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T08:48:38.8502194Z outputs = self.model.decoder( 2025-12-04T08:48:38.8502463Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T08:48:38.8502585Z layer_outputs = decoder_layer( 2025-12-04T08:48:38.8502827Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:48:38.8502910Z return super().__call__(*args, **kwargs) 2025-12-04T08:48:38.8503178Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8503252Z return func(*args, **kwargs) 2025-12-04T08:48:38.8503524Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 452, in forward 2025-12-04T08:48:38.8503612Z hidden_states = self.fc2(hidden_states) 2025-12-04T08:48:38.8503616Z 2025-12-04T08:48:38.8503724Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:48:38.8503946Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:48:38.8504013Z res = mod(**inputs) 2025-12-04T08:48:38.8504288Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T08:48:38.8504377Z outputs = self.model.decoder( 2025-12-04T08:48:38.8504656Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T08:48:38.8504742Z layer_outputs = decoder_layer( 2025-12-04T08:48:38.8504987Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:48:38.8505073Z return super().__call__(*args, **kwargs) 2025-12-04T08:48:38.8505350Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8505425Z return func(*args, **kwargs) 2025-12-04T08:48:38.8505699Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T08:48:38.8505816Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:48:38.8506088Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8506170Z return func(*args, **kwargs) 2025-12-04T08:48:38.8506445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 219, in forward 2025-12-04T08:48:38.8506616Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T08:48:38.8506620Z 2025-12-04T08:48:38.8506737Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:48:38.8506957Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:48:38.8507033Z res = mod(**inputs) 2025-12-04T08:48:38.8507347Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T08:48:38.8507430Z outputs = self.model.decoder( 2025-12-04T08:48:38.8507756Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T08:48:38.8507840Z layer_outputs = decoder_layer( 2025-12-04T08:48:38.8508091Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:48:38.8508183Z return super().__call__(*args, **kwargs) 2025-12-04T08:48:38.8508451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8508533Z return func(*args, **kwargs) 2025-12-04T08:48:38.8508805Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T08:48:38.8508912Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:48:38.8509186Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8509303Z return func(*args, **kwargs) 2025-12-04T08:48:38.8509587Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 239, in forward 2025-12-04T08:48:38.8509677Z key_states = self.k_proj(current_states) 2025-12-04T08:48:38.8509681Z 2025-12-04T08:48:38.8509795Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:48:38.8510036Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:48:38.8510106Z res = mod(**inputs) 2025-12-04T08:48:38.8510398Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T08:48:38.8510485Z outputs = self.model.decoder( 2025-12-04T08:48:38.8510789Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T08:48:38.8510880Z layer_outputs = decoder_layer( 2025-12-04T08:48:38.8511139Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:48:38.8511225Z return super().__call__(*args, **kwargs) 2025-12-04T08:48:38.8511516Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8511591Z return func(*args, **kwargs) 2025-12-04T08:48:38.8511892Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T08:48:38.8512006Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:48:38.8512299Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8512383Z return func(*args, **kwargs) 2025-12-04T08:48:38.8512681Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 240, in forward 2025-12-04T08:48:38.8512782Z value_states = self.v_proj(current_states) 2025-12-04T08:48:38.8512789Z 2025-12-04T08:48:38.8512889Z cudagraph partition due to non gpu ops 2025-12-04T08:48:38.8513001Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:48:38.8513229Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:48:38.8513300Z res = mod(**inputs) 2025-12-04T08:48:38.8513603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T08:48:38.8513692Z outputs = self.model.decoder( 2025-12-04T08:48:38.8513982Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T08:48:38.8514061Z layer_outputs = decoder_layer( 2025-12-04T08:48:38.8514328Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:48:38.8514417Z return super().__call__(*args, **kwargs) 2025-12-04T08:48:38.8514759Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8514838Z return func(*args, **kwargs) 2025-12-04T08:48:38.8515139Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T08:48:38.8515254Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:48:38.8515551Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8515633Z return func(*args, **kwargs) 2025-12-04T08:48:38.8515937Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 258, in forward 2025-12-04T08:48:38.8516047Z attn_output, attn_weights = attention_interface( 2025-12-04T08:48:38.8516438Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T08:48:38.8516593Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T08:48:38.8516598Z 2025-12-04T08:48:38.8516710Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:48:38.8516951Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:48:38.8517022Z res = mod(**inputs) 2025-12-04T08:48:38.8517335Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T08:48:38.8517416Z outputs = self.model.decoder( 2025-12-04T08:48:38.8517722Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T08:48:38.8517812Z layer_outputs = decoder_layer( 2025-12-04T08:48:38.8518066Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:48:38.8518268Z return super().__call__(*args, **kwargs) 2025-12-04T08:48:38.8518627Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8518705Z return func(*args, **kwargs) 2025-12-04T08:48:38.8519022Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T08:48:38.8519130Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:48:38.8519424Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8519510Z return func(*args, **kwargs) 2025-12-04T08:48:38.8519811Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 272, in forward 2025-12-04T08:48:38.8519916Z attn_output = self.out_proj(attn_output) 2025-12-04T08:48:38.8519920Z 2025-12-04T08:48:38.8520037Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:48:38.8520259Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:48:38.8520339Z res = mod(**inputs) 2025-12-04T08:48:38.8520641Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T08:48:38.8520872Z outputs = self.model.decoder( 2025-12-04T08:48:38.8521208Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T08:48:38.8521289Z layer_outputs = decoder_layer( 2025-12-04T08:48:38.8521546Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:48:38.8521633Z return super().__call__(*args, **kwargs) 2025-12-04T08:48:38.8522007Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8522094Z return func(*args, **kwargs) 2025-12-04T08:48:38.8522375Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 450, in forward 2025-12-04T08:48:38.8522512Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:48:38.8522517Z 2025-12-04T08:48:38.8522630Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:48:38.8522851Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:48:38.8522931Z res = mod(**inputs) 2025-12-04T08:48:38.8523231Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T08:48:38.8523312Z outputs = self.model.decoder( 2025-12-04T08:48:38.8523600Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T08:48:38.8523757Z layer_outputs = decoder_layer( 2025-12-04T08:48:38.8524010Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:48:38.8524096Z return super().__call__(*args, **kwargs) 2025-12-04T08:48:38.8524368Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8524451Z return func(*args, **kwargs) 2025-12-04T08:48:38.8524755Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 450, in forward 2025-12-04T08:48:38.8524892Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:48:38.8525139Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T08:48:38.8525220Z return self.act(input) 2025-12-04T08:48:38.8525224Z 2025-12-04T08:48:38.8525341Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:48:38.8525564Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:48:38.8525636Z res = mod(**inputs) 2025-12-04T08:48:38.8525930Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T08:48:38.8526008Z outputs = self.model.decoder( 2025-12-04T08:48:38.8526291Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T08:48:38.8526371Z layer_outputs = decoder_layer( 2025-12-04T08:48:38.8526616Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:48:38.8526712Z return super().__call__(*args, **kwargs) 2025-12-04T08:48:38.8526982Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8527065Z return func(*args, **kwargs) 2025-12-04T08:48:38.8527355Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 452, in forward 2025-12-04T08:48:38.8527446Z hidden_states = self.fc2(hidden_states) 2025-12-04T08:48:38.8527450Z 2025-12-04T08:48:38.8527573Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:48:38.8527793Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:48:38.8527864Z res = mod(**inputs) 2025-12-04T08:48:38.8528150Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T08:48:38.8528230Z outputs = self.model.decoder( 2025-12-04T08:48:38.8528536Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T08:48:38.8528619Z layer_outputs = decoder_layer( 2025-12-04T08:48:38.8528945Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:48:38.8529040Z return super().__call__(*args, **kwargs) 2025-12-04T08:48:38.8529309Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8529386Z return func(*args, **kwargs) 2025-12-04T08:48:38.8529669Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T08:48:38.8529778Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:48:38.8530052Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8530128Z return func(*args, **kwargs) 2025-12-04T08:48:38.8530402Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 219, in forward 2025-12-04T08:48:38.8530624Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T08:48:38.8530628Z 2025-12-04T08:48:38.8530741Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:48:38.8530968Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:48:38.8531038Z res = mod(**inputs) 2025-12-04T08:48:38.8531314Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T08:48:38.8531402Z outputs = self.model.decoder( 2025-12-04T08:48:38.8531680Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T08:48:38.8531762Z layer_outputs = decoder_layer( 2025-12-04T08:48:38.8532017Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:48:38.8532111Z return super().__call__(*args, **kwargs) 2025-12-04T08:48:38.8532398Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8532472Z return func(*args, **kwargs) 2025-12-04T08:48:38.8532749Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T08:48:38.8532865Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:48:38.8533135Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8533209Z return func(*args, **kwargs) 2025-12-04T08:48:38.8533491Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 239, in forward 2025-12-04T08:48:38.8533579Z key_states = self.k_proj(current_states) 2025-12-04T08:48:38.8533586Z 2025-12-04T08:48:38.8533703Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:48:38.8533925Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:48:38.8533996Z res = mod(**inputs) 2025-12-04T08:48:38.8534281Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T08:48:38.8534361Z outputs = self.model.decoder( 2025-12-04T08:48:38.8534647Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T08:48:38.8534725Z layer_outputs = decoder_layer( 2025-12-04T08:48:38.8534968Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:48:38.8535061Z return super().__call__(*args, **kwargs) 2025-12-04T08:48:38.8535329Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8535442Z return func(*args, **kwargs) 2025-12-04T08:48:38.8535728Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T08:48:38.8535836Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:48:38.8536111Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8536186Z return func(*args, **kwargs) 2025-12-04T08:48:38.8536462Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 240, in forward 2025-12-04T08:48:38.8536564Z value_states = self.v_proj(current_states) 2025-12-04T08:48:38.8536568Z 2025-12-04T08:48:38.8536658Z cudagraph partition due to non gpu ops 2025-12-04T08:48:38.8536777Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:48:38.8537044Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:48:38.8537119Z res = mod(**inputs) 2025-12-04T08:48:38.8537415Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T08:48:38.8537495Z outputs = self.model.decoder( 2025-12-04T08:48:38.8537782Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T08:48:38.8537869Z layer_outputs = decoder_layer( 2025-12-04T08:48:38.8538124Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:48:38.8538218Z return super().__call__(*args, **kwargs) 2025-12-04T08:48:38.8538493Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8538571Z return func(*args, **kwargs) 2025-12-04T08:48:38.8538869Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T08:48:38.8538976Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:48:38.8539254Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8539336Z return func(*args, **kwargs) 2025-12-04T08:48:38.8539631Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 258, in forward 2025-12-04T08:48:38.8539747Z attn_output, attn_weights = attention_interface( 2025-12-04T08:48:38.8540082Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T08:48:38.8540231Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T08:48:38.8540238Z 2025-12-04T08:48:38.8540360Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:48:38.8540585Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:48:38.8540662Z res = mod(**inputs) 2025-12-04T08:48:38.8540949Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T08:48:38.8541029Z outputs = self.model.decoder( 2025-12-04T08:48:38.8541338Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T08:48:38.8541417Z layer_outputs = decoder_layer( 2025-12-04T08:48:38.8541681Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:48:38.8541768Z return super().__call__(*args, **kwargs) 2025-12-04T08:48:38.8542046Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8542133Z return func(*args, **kwargs) 2025-12-04T08:48:38.8542463Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 418, in forward 2025-12-04T08:48:38.8542579Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:48:38.8542848Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8542922Z return func(*args, **kwargs) 2025-12-04T08:48:38.8543207Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 272, in forward 2025-12-04T08:48:38.8543296Z attn_output = self.out_proj(attn_output) 2025-12-04T08:48:38.8543300Z 2025-12-04T08:48:38.8543418Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:48:38.8543639Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:48:38.8543746Z res = mod(**inputs) 2025-12-04T08:48:38.8544038Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T08:48:38.8544118Z outputs = self.model.decoder( 2025-12-04T08:48:38.8544399Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T08:48:38.8544486Z layer_outputs = decoder_layer( 2025-12-04T08:48:38.8544732Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:48:38.8544825Z return super().__call__(*args, **kwargs) 2025-12-04T08:48:38.8545099Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8545174Z return func(*args, **kwargs) 2025-12-04T08:48:38.8545458Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 450, in forward 2025-12-04T08:48:38.8545593Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:48:38.8545597Z 2025-12-04T08:48:38.8545717Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:48:38.8545937Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:48:38.8546007Z res = mod(**inputs) 2025-12-04T08:48:38.8546311Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T08:48:38.8546391Z outputs = self.model.decoder( 2025-12-04T08:48:38.8546667Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T08:48:38.8546756Z layer_outputs = decoder_layer( 2025-12-04T08:48:38.8547008Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:48:38.8547104Z return super().__call__(*args, **kwargs) 2025-12-04T08:48:38.8547378Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8547455Z return func(*args, **kwargs) 2025-12-04T08:48:38.8547738Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 450, in forward 2025-12-04T08:48:38.8547868Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:48:38.8548108Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T08:48:38.8548195Z return self.act(input) 2025-12-04T08:48:38.8548199Z 2025-12-04T08:48:38.8548311Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:48:38.8548539Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:48:38.8548611Z res = mod(**inputs) 2025-12-04T08:48:38.8548948Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1902, in forward 2025-12-04T08:48:38.8549038Z outputs = self.model.decoder( 2025-12-04T08:48:38.8549317Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1122, in forward 2025-12-04T08:48:38.8549403Z layer_outputs = decoder_layer( 2025-12-04T08:48:38.8549648Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:48:38.8549736Z return super().__call__(*args, **kwargs) 2025-12-04T08:48:38.8550013Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:48:38.8550089Z return func(*args, **kwargs) 2025-12-04T08:48:38.8550368Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 452, in forward 2025-12-04T08:48:38.8550514Z hidden_states = self.fc2(hidden_states) 2025-12-04T08:48:38.8550518Z 2025-12-04T08:48:38.8550633Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:48:38.8550858Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:48:38.8550930Z res = mod(**inputs) 2025-12-04T08:48:38.8551205Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1918, in forward 2025-12-04T08:48:38.8551300Z logits = self.lm_head(outputs[0]) 2025-12-04T08:48:38.8551304Z 2025-12-04T08:48:38.8551414Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:48:38.8551637Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:48:38.8551707Z res = mod(**inputs) 2025-12-04T08:48:38.8551985Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1924, in forward 2025-12-04T08:48:38.8552160Z loss = loss_fct(logits.view(-1, self.config.vocab_size), labels.view(-1)) 2025-12-04T08:48:38.8552167Z 2025-12-04T08:48:51.2893876Z Compilation time (from dynamo_timed): 16.943236966 2025-12-04T08:48:51.3094155Z pass 2025-12-04T08:48:51.3094618Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T08:48:51.3095544Z TIMING: _recursive_pre_grad_passes:0.0077 _recursive_joint_graph_passes:0.66922 _recursive_post_grad_passes:0.06728 async_compile.wait:0.87695 code_gen:9.64076 inductor_compile:10.9854 backend_compile:14.14555 gc:0.00016 entire_frame_compile:16.94324 total_wall_time:16.94324 2025-12-04T08:48:51.3096730Z STATS: call_* op count: 336 | FakeTensorMode.__torch_dispatch__:7356 | FakeTensor.__torch_dispatch__:4394 | ProxyTorchDispatchMode.__torch_dispatch__:2034 2025-12-04T08:48:51.3097301Z Dynamo produced 1 graphs covering 336 ops with 0 graph breaks (0 unique) 2025-12-04T08:48:53.7392590Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/__init__.py:65: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-12-04T08:48:53.7393433Z import pynvml # type: ignore[import] 2025-12-04T08:48:57.1510257Z 2025-12-04T08:48:58.4628946Z loading model: 0it [00:00, ?it/s] 2025-12-04T08:48:58.4629270Z loading model: 0it [00:01, ?it/s] 2025-12-04T08:48:58.4645858Z cpu eval BertForMaskedLM 2025-12-04T08:48:59.8428768Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T08:49:00.4939502Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T08:49:01.1153019Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T08:49:08.6669909Z cudagraph partition due to non gpu ops 2025-12-04T08:49:08.6670297Z cudagraph partition due to non gpu ops 2025-12-04T08:49:08.6670893Z cudagraph partition due to non gpu ops 2025-12-04T08:49:08.6671143Z cudagraph partition due to non gpu ops 2025-12-04T08:49:08.6671382Z cudagraph partition due to non gpu ops 2025-12-04T08:49:08.6671614Z cudagraph partition due to non gpu ops 2025-12-04T08:49:08.6671837Z cudagraph partition due to non gpu ops 2025-12-04T08:49:08.6672064Z cudagraph partition due to non gpu ops 2025-12-04T08:49:08.6672291Z cudagraph partition due to non gpu ops 2025-12-04T08:49:08.6672515Z cudagraph partition due to non gpu ops 2025-12-04T08:49:08.6672748Z cudagraph partition due to non gpu ops 2025-12-04T08:49:08.6672987Z cudagraph partition due to non gpu ops 2025-12-04T08:49:08.6673257Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:49:08.6673694Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:49:08.6674116Z res = mod(**inputs) 2025-12-04T08:49:08.6674695Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T08:49:08.6675134Z outputs = self.bert( 2025-12-04T08:49:08.6675556Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T08:49:08.6676009Z encoder_outputs = self.encoder( 2025-12-04T08:49:08.6676458Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T08:49:08.6676915Z layer_outputs = layer_module( 2025-12-04T08:49:08.6677322Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:49:08.6677795Z return super().__call__(*args, **kwargs) 2025-12-04T08:49:08.6678544Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.6679079Z return func(*args, **kwargs) 2025-12-04T08:49:08.6679518Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T08:49:08.6679985Z self_attention_outputs = self.attention( 2025-12-04T08:49:08.6680414Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.6680832Z return func(*args, **kwargs) 2025-12-04T08:49:08.6681243Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 488, in forward 2025-12-04T08:49:08.6681664Z self_outputs = self.self( 2025-12-04T08:49:08.6682070Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.6682486Z return func(*args, **kwargs) 2025-12-04T08:49:08.6682903Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 363, in forward 2025-12-04T08:49:08.6683508Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-12-04T08:49:08.6683817Z 2025-12-04T08:49:08.6683939Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:49:08.6684361Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:49:08.6684731Z res = mod(**inputs) 2025-12-04T08:49:08.6685141Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T08:49:08.6685624Z outputs = self.bert( 2025-12-04T08:49:08.6686035Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T08:49:08.6686465Z encoder_outputs = self.encoder( 2025-12-04T08:49:08.6686909Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T08:49:08.6687355Z layer_outputs = layer_module( 2025-12-04T08:49:08.6687818Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:49:08.6688394Z return super().__call__(*args, **kwargs) 2025-12-04T08:49:08.6688827Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.6689246Z return func(*args, **kwargs) 2025-12-04T08:49:08.6689663Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T08:49:08.6690101Z self_attention_outputs = self.attention( 2025-12-04T08:49:08.6690537Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.6690959Z return func(*args, **kwargs) 2025-12-04T08:49:08.6691406Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 488, in forward 2025-12-04T08:49:08.6691839Z self_outputs = self.self( 2025-12-04T08:49:08.6692250Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.6692662Z return func(*args, **kwargs) 2025-12-04T08:49:08.6693063Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 387, in forward 2025-12-04T08:49:08.6693488Z self.key(current_states) 2025-12-04T08:49:08.6693621Z 2025-12-04T08:49:08.6693749Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:49:08.6694150Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:49:08.6694514Z res = mod(**inputs) 2025-12-04T08:49:08.6694917Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T08:49:08.6695351Z outputs = self.bert( 2025-12-04T08:49:08.6695754Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T08:49:08.6696189Z encoder_outputs = self.encoder( 2025-12-04T08:49:08.6696646Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T08:49:08.6697091Z layer_outputs = layer_module( 2025-12-04T08:49:08.6697479Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:49:08.6697889Z return super().__call__(*args, **kwargs) 2025-12-04T08:49:08.6698323Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.6698747Z return func(*args, **kwargs) 2025-12-04T08:49:08.6699164Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T08:49:08.6699619Z self_attention_outputs = self.attention( 2025-12-04T08:49:08.6700057Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.6700466Z return func(*args, **kwargs) 2025-12-04T08:49:08.6700876Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 488, in forward 2025-12-04T08:49:08.6701307Z self_outputs = self.self( 2025-12-04T08:49:08.6701705Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.6702120Z return func(*args, **kwargs) 2025-12-04T08:49:08.6702529Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 392, in forward 2025-12-04T08:49:08.6702958Z self.value(current_states) 2025-12-04T08:49:08.6703094Z 2025-12-04T08:49:08.6703186Z cudagraph partition due to non gpu ops 2025-12-04T08:49:08.6703503Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:49:08.6703915Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:49:08.6704274Z res = mod(**inputs) 2025-12-04T08:49:08.6704679Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T08:49:08.6705106Z outputs = self.bert( 2025-12-04T08:49:08.6705511Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T08:49:08.6705938Z encoder_outputs = self.encoder( 2025-12-04T08:49:08.6706362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T08:49:08.6706789Z layer_outputs = layer_module( 2025-12-04T08:49:08.6707222Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:49:08.6707634Z return super().__call__(*args, **kwargs) 2025-12-04T08:49:08.6708060Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.6708492Z return func(*args, **kwargs) 2025-12-04T08:49:08.6708904Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T08:49:08.6709362Z self_attention_outputs = self.attention( 2025-12-04T08:49:08.6709798Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.6710217Z return func(*args, **kwargs) 2025-12-04T08:49:08.6710627Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 488, in forward 2025-12-04T08:49:08.6711059Z self_outputs = self.self( 2025-12-04T08:49:08.6711470Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.6711882Z return func(*args, **kwargs) 2025-12-04T08:49:08.6712300Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 413, in forward 2025-12-04T08:49:08.6712810Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T08:49:08.6713018Z 2025-12-04T08:49:08.6713144Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:49:08.6713547Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:49:08.6713910Z res = mod(**inputs) 2025-12-04T08:49:08.6714337Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T08:49:08.6714780Z outputs = self.bert( 2025-12-04T08:49:08.6715186Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T08:49:08.6715628Z encoder_outputs = self.encoder( 2025-12-04T08:49:08.6716056Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T08:49:08.6716478Z layer_outputs = layer_module( 2025-12-04T08:49:08.6716874Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:49:08.6717289Z return super().__call__(*args, **kwargs) 2025-12-04T08:49:08.6717718Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.6718147Z return func(*args, **kwargs) 2025-12-04T08:49:08.6718641Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T08:49:08.6719095Z self_attention_outputs = self.attention( 2025-12-04T08:49:08.6719559Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.6719985Z return func(*args, **kwargs) 2025-12-04T08:49:08.6720400Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 497, in forward 2025-12-04T08:49:08.6721683Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T08:49:08.6722172Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 436, in forward 2025-12-04T08:49:08.6722621Z hidden_states = self.dense(hidden_states) 2025-12-04T08:49:08.6722780Z 2025-12-04T08:49:08.6722906Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:49:08.6723315Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:49:08.6723804Z res = mod(**inputs) 2025-12-04T08:49:08.6724211Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T08:49:08.6724646Z outputs = self.bert( 2025-12-04T08:49:08.6725055Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T08:49:08.6725490Z encoder_outputs = self.encoder( 2025-12-04T08:49:08.6725928Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T08:49:08.6726356Z layer_outputs = layer_module( 2025-12-04T08:49:08.6726741Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:49:08.6727152Z return super().__call__(*args, **kwargs) 2025-12-04T08:49:08.6727581Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.6728006Z return func(*args, **kwargs) 2025-12-04T08:49:08.6728431Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 588, in forward 2025-12-04T08:49:08.6728879Z layer_output = apply_chunking_to_forward( 2025-12-04T08:49:08.6729337Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:49:08.6729778Z return forward_fn(*input_tensors) 2025-12-04T08:49:08.6730242Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 596, in feed_forward_chunk 2025-12-04T08:49:08.6730762Z intermediate_output = self.intermediate(attention_output) 2025-12-04T08:49:08.6731248Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 512, in forward 2025-12-04T08:49:08.6731686Z hidden_states = self.dense(hidden_states) 2025-12-04T08:49:08.6731851Z 2025-12-04T08:49:08.6731968Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:49:08.6732379Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:49:08.6732739Z res = mod(**inputs) 2025-12-04T08:49:08.6733141Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T08:49:08.6733576Z outputs = self.bert( 2025-12-04T08:49:08.6733983Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T08:49:08.6734409Z encoder_outputs = self.encoder( 2025-12-04T08:49:08.6734831Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T08:49:08.6735274Z layer_outputs = layer_module( 2025-12-04T08:49:08.6735660Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:49:08.6736138Z return super().__call__(*args, **kwargs) 2025-12-04T08:49:08.6736593Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.6737012Z return func(*args, **kwargs) 2025-12-04T08:49:08.6737420Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 588, in forward 2025-12-04T08:49:08.6737933Z layer_output = apply_chunking_to_forward( 2025-12-04T08:49:08.6738389Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:49:08.6738840Z return forward_fn(*input_tensors) 2025-12-04T08:49:08.6739296Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 596, in feed_forward_chunk 2025-12-04T08:49:08.6739872Z intermediate_output = self.intermediate(attention_output) 2025-12-04T08:49:08.6740488Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 513, in forward 2025-12-04T08:49:08.6740970Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:49:08.6741408Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T08:49:08.6741802Z return self.act(input) 2025-12-04T08:49:08.6741929Z 2025-12-04T08:49:08.6742055Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:49:08.6742458Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:49:08.6742825Z res = mod(**inputs) 2025-12-04T08:49:08.6743234Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T08:49:08.6743660Z outputs = self.bert( 2025-12-04T08:49:08.6744063Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T08:49:08.6744503Z encoder_outputs = self.encoder( 2025-12-04T08:49:08.6744928Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T08:49:08.6745353Z layer_outputs = layer_module( 2025-12-04T08:49:08.6745748Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:49:08.6746163Z return super().__call__(*args, **kwargs) 2025-12-04T08:49:08.6746594Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.6747008Z return func(*args, **kwargs) 2025-12-04T08:49:08.6747421Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 588, in forward 2025-12-04T08:49:08.6747866Z layer_output = apply_chunking_to_forward( 2025-12-04T08:49:08.6748316Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:49:08.6748761Z return forward_fn(*input_tensors) 2025-12-04T08:49:08.6749218Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 597, in feed_forward_chunk 2025-12-04T08:49:08.6749746Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T08:49:08.6750233Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 525, in forward 2025-12-04T08:49:08.6750674Z hidden_states = self.dense(hidden_states) 2025-12-04T08:49:08.6750828Z 2025-12-04T08:49:08.6750954Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:49:08.6751358Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:49:08.6751718Z res = mod(**inputs) 2025-12-04T08:49:08.6752164Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T08:49:08.6752599Z outputs = self.bert( 2025-12-04T08:49:08.6752997Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T08:49:08.6753445Z encoder_outputs = self.encoder( 2025-12-04T08:49:08.6753895Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T08:49:08.6754338Z layer_outputs = layer_module( 2025-12-04T08:49:08.6754724Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:49:08.6755140Z return super().__call__(*args, **kwargs) 2025-12-04T08:49:08.6755568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.6756024Z return func(*args, **kwargs) 2025-12-04T08:49:08.6756442Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T08:49:08.6756887Z self_attention_outputs = self.attention( 2025-12-04T08:49:08.6757323Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.6757746Z return func(*args, **kwargs) 2025-12-04T08:49:08.6758254Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 488, in forward 2025-12-04T08:49:08.6758705Z self_outputs = self.self( 2025-12-04T08:49:08.6759116Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.6759541Z return func(*args, **kwargs) 2025-12-04T08:49:08.6759957Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 363, in forward 2025-12-04T08:49:08.6760561Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-12-04T08:49:08.6760865Z 2025-12-04T08:49:08.6760983Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:49:08.6761393Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:49:08.6761763Z res = mod(**inputs) 2025-12-04T08:49:08.6762170Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T08:49:08.6762591Z outputs = self.bert( 2025-12-04T08:49:08.6763002Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T08:49:08.6763438Z encoder_outputs = self.encoder( 2025-12-04T08:49:08.6763856Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T08:49:08.6764293Z layer_outputs = layer_module( 2025-12-04T08:49:08.6764688Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:49:08.6765102Z return super().__call__(*args, **kwargs) 2025-12-04T08:49:08.6765527Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.6765948Z return func(*args, **kwargs) 2025-12-04T08:49:08.6766365Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T08:49:08.6766810Z self_attention_outputs = self.attention( 2025-12-04T08:49:08.6767238Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.6767657Z return func(*args, **kwargs) 2025-12-04T08:49:08.6768117Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 488, in forward 2025-12-04T08:49:08.6768539Z self_outputs = self.self( 2025-12-04T08:49:08.6768950Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.6769370Z return func(*args, **kwargs) 2025-12-04T08:49:08.6769785Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 387, in forward 2025-12-04T08:49:08.6770205Z self.key(current_states) 2025-12-04T08:49:08.6770345Z 2025-12-04T08:49:08.6770463Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:49:08.6770874Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:49:08.6771231Z res = mod(**inputs) 2025-12-04T08:49:08.6771638Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T08:49:08.6772109Z outputs = self.bert( 2025-12-04T08:49:08.6772522Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T08:49:08.6772954Z encoder_outputs = self.encoder( 2025-12-04T08:49:08.6773382Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T08:49:08.6773813Z layer_outputs = layer_module( 2025-12-04T08:49:08.6774197Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:49:08.6774608Z return super().__call__(*args, **kwargs) 2025-12-04T08:49:08.6775035Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.6775454Z return func(*args, **kwargs) 2025-12-04T08:49:08.6775866Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T08:49:08.6776314Z self_attention_outputs = self.attention( 2025-12-04T08:49:08.6776749Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.6777177Z return func(*args, **kwargs) 2025-12-04T08:49:08.6777584Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 488, in forward 2025-12-04T08:49:08.6777999Z self_outputs = self.self( 2025-12-04T08:49:08.6778396Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.6778799Z return func(*args, **kwargs) 2025-12-04T08:49:08.6779202Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 392, in forward 2025-12-04T08:49:08.6779619Z self.value(current_states) 2025-12-04T08:49:08.6779748Z 2025-12-04T08:49:08.6779844Z cudagraph partition due to non gpu ops 2025-12-04T08:49:08.6780104Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:49:08.6780501Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:49:08.6780853Z res = mod(**inputs) 2025-12-04T08:49:08.6781237Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T08:49:08.6781650Z outputs = self.bert( 2025-12-04T08:49:08.6782043Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T08:49:08.6782465Z encoder_outputs = self.encoder( 2025-12-04T08:49:08.6782873Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T08:49:08.6783288Z layer_outputs = layer_module( 2025-12-04T08:49:08.6783726Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:49:08.6784120Z return super().__call__(*args, **kwargs) 2025-12-04T08:49:08.6784536Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.6784947Z return func(*args, **kwargs) 2025-12-04T08:49:08.6785351Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T08:49:08.6785773Z self_attention_outputs = self.attention( 2025-12-04T08:49:08.6786190Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.6786600Z return func(*args, **kwargs) 2025-12-04T08:49:08.6787000Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 488, in forward 2025-12-04T08:49:08.6787481Z self_outputs = self.self( 2025-12-04T08:49:08.6787884Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.6788300Z return func(*args, **kwargs) 2025-12-04T08:49:08.6788702Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 413, in forward 2025-12-04T08:49:08.6789196Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T08:49:08.6789409Z 2025-12-04T08:49:08.6789529Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:49:08.6789931Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:49:08.6790284Z res = mod(**inputs) 2025-12-04T08:49:08.6790687Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T08:49:08.6791114Z outputs = self.bert( 2025-12-04T08:49:08.6791507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T08:49:08.6791937Z encoder_outputs = self.encoder( 2025-12-04T08:49:08.6792357Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T08:49:08.6792793Z layer_outputs = layer_module( 2025-12-04T08:49:08.6793173Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:49:08.6793579Z return super().__call__(*args, **kwargs) 2025-12-04T08:49:08.6794006Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.6794420Z return func(*args, **kwargs) 2025-12-04T08:49:08.6794823Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T08:49:08.6795277Z self_attention_outputs = self.attention( 2025-12-04T08:49:08.6795721Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.6796156Z return func(*args, **kwargs) 2025-12-04T08:49:08.6796575Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 497, in forward 2025-12-04T08:49:08.6797081Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T08:49:08.6797576Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 436, in forward 2025-12-04T08:49:08.6798022Z hidden_states = self.dense(hidden_states) 2025-12-04T08:49:08.6798270Z 2025-12-04T08:49:08.6798395Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:49:08.6798803Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:49:08.6799163Z res = mod(**inputs) 2025-12-04T08:49:08.6799611Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T08:49:08.6800046Z outputs = self.bert( 2025-12-04T08:49:08.6800458Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T08:49:08.6800884Z encoder_outputs = self.encoder( 2025-12-04T08:49:08.6801312Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T08:49:08.6801754Z layer_outputs = layer_module( 2025-12-04T08:49:08.6802153Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:49:08.6802578Z return super().__call__(*args, **kwargs) 2025-12-04T08:49:08.6803013Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.6803473Z return func(*args, **kwargs) 2025-12-04T08:49:08.6803896Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 588, in forward 2025-12-04T08:49:08.6804340Z layer_output = apply_chunking_to_forward( 2025-12-04T08:49:08.6804795Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:49:08.6805241Z return forward_fn(*input_tensors) 2025-12-04T08:49:08.6805695Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 596, in feed_forward_chunk 2025-12-04T08:49:08.6806215Z intermediate_output = self.intermediate(attention_output) 2025-12-04T08:49:08.6806703Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 512, in forward 2025-12-04T08:49:08.6807147Z hidden_states = self.dense(hidden_states) 2025-12-04T08:49:08.6807300Z 2025-12-04T08:49:08.6807419Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:49:08.6807823Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:49:08.6808188Z res = mod(**inputs) 2025-12-04T08:49:08.6808581Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T08:49:08.6809005Z outputs = self.bert( 2025-12-04T08:49:08.6809416Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T08:49:08.6809850Z encoder_outputs = self.encoder( 2025-12-04T08:49:08.6810266Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T08:49:08.6810698Z layer_outputs = layer_module( 2025-12-04T08:49:08.6811095Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:49:08.6811500Z return super().__call__(*args, **kwargs) 2025-12-04T08:49:08.6811930Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.6812353Z return func(*args, **kwargs) 2025-12-04T08:49:08.6812768Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 588, in forward 2025-12-04T08:49:08.6813202Z layer_output = apply_chunking_to_forward( 2025-12-04T08:49:08.6813656Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:49:08.6814108Z return forward_fn(*input_tensors) 2025-12-04T08:49:08.6814562Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 596, in feed_forward_chunk 2025-12-04T08:49:08.6815079Z intermediate_output = self.intermediate(attention_output) 2025-12-04T08:49:08.6815608Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 513, in forward 2025-12-04T08:49:08.6816087Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:49:08.6816519Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T08:49:08.6816916Z return self.act(input) 2025-12-04T08:49:08.6817054Z 2025-12-04T08:49:08.6817175Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:49:08.6817588Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:49:08.6817952Z res = mod(**inputs) 2025-12-04T08:49:08.6818363Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T08:49:08.6818837Z outputs = self.bert( 2025-12-04T08:49:08.6819235Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T08:49:08.6819667Z encoder_outputs = self.encoder( 2025-12-04T08:49:08.6820279Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T08:49:08.6820838Z layer_outputs = layer_module( 2025-12-04T08:49:08.6821236Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:49:08.6821647Z return super().__call__(*args, **kwargs) 2025-12-04T08:49:08.6822077Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.6822506Z return func(*args, **kwargs) 2025-12-04T08:49:08.6822928Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 588, in forward 2025-12-04T08:49:08.6823378Z layer_output = apply_chunking_to_forward( 2025-12-04T08:49:08.6823840Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:49:08.6824278Z return forward_fn(*input_tensors) 2025-12-04T08:49:08.6824736Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 597, in feed_forward_chunk 2025-12-04T08:49:08.6825262Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T08:49:08.6825753Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 525, in forward 2025-12-04T08:49:08.6826202Z hidden_states = self.dense(hidden_states) 2025-12-04T08:49:08.6826364Z 2025-12-04T08:49:08.6826480Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:49:08.6826882Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:49:08.6827247Z res = mod(**inputs) 2025-12-04T08:49:08.6827645Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T08:49:08.6828070Z outputs = self.bert( 2025-12-04T08:49:08.6828473Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T08:49:08.6828894Z encoder_outputs = self.encoder( 2025-12-04T08:49:08.6829318Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T08:49:08.6829749Z layer_outputs = layer_module( 2025-12-04T08:49:08.6830141Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:49:08.6830542Z return super().__call__(*args, **kwargs) 2025-12-04T08:49:08.6830967Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.6831477Z return func(*args, **kwargs) 2025-12-04T08:49:08.6831888Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T08:49:08.6832332Z self_attention_outputs = self.attention( 2025-12-04T08:49:08.6832767Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.6833185Z return func(*args, **kwargs) 2025-12-04T08:49:08.6833592Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 488, in forward 2025-12-04T08:49:08.6834027Z self_outputs = self.self( 2025-12-04T08:49:08.6834448Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.6834860Z return func(*args, **kwargs) 2025-12-04T08:49:08.6835338Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 363, in forward 2025-12-04T08:49:08.6835926Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-12-04T08:49:08.6836225Z 2025-12-04T08:49:08.6836348Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:49:08.6836749Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:49:08.6837115Z res = mod(**inputs) 2025-12-04T08:49:08.6837533Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T08:49:08.6837977Z outputs = self.bert( 2025-12-04T08:49:08.6838445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T08:49:08.6838885Z encoder_outputs = self.encoder( 2025-12-04T08:49:08.6839320Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T08:49:08.6839746Z layer_outputs = layer_module( 2025-12-04T08:49:08.6840142Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:49:08.6840566Z return super().__call__(*args, **kwargs) 2025-12-04T08:49:08.6840999Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.6841414Z return func(*args, **kwargs) 2025-12-04T08:49:08.6841833Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T08:49:08.6842279Z self_attention_outputs = self.attention( 2025-12-04T08:49:08.6842710Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.6843144Z return func(*args, **kwargs) 2025-12-04T08:49:08.6843552Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 488, in forward 2025-12-04T08:49:08.6843969Z self_outputs = self.self( 2025-12-04T08:49:08.6844361Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.6844769Z return func(*args, **kwargs) 2025-12-04T08:49:08.6845170Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 387, in forward 2025-12-04T08:49:08.6845586Z self.key(current_states) 2025-12-04T08:49:08.6845712Z 2025-12-04T08:49:08.6845825Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:49:08.6846222Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:49:08.6846577Z res = mod(**inputs) 2025-12-04T08:49:08.6846966Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T08:49:08.6847446Z outputs = self.bert( 2025-12-04T08:49:08.6847843Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T08:49:08.6848261Z encoder_outputs = self.encoder( 2025-12-04T08:49:08.6848667Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T08:49:08.6849086Z layer_outputs = layer_module( 2025-12-04T08:49:08.6849466Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:49:08.6849854Z return super().__call__(*args, **kwargs) 2025-12-04T08:49:08.6850267Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.6850717Z return func(*args, **kwargs) 2025-12-04T08:49:08.6851133Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T08:49:08.6851569Z self_attention_outputs = self.attention( 2025-12-04T08:49:08.6852000Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.6852415Z return func(*args, **kwargs) 2025-12-04T08:49:08.6852818Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 488, in forward 2025-12-04T08:49:08.6853246Z self_outputs = self.self( 2025-12-04T08:49:08.6853649Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.6854065Z return func(*args, **kwargs) 2025-12-04T08:49:08.6854468Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 392, in forward 2025-12-04T08:49:08.6854916Z self.value(current_states) 2025-12-04T08:49:08.6855049Z 2025-12-04T08:49:08.6855153Z cudagraph partition due to non gpu ops 2025-12-04T08:49:08.6855424Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:49:08.6855821Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:49:08.6856184Z res = mod(**inputs) 2025-12-04T08:49:08.6856583Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T08:49:08.6857023Z outputs = self.bert( 2025-12-04T08:49:08.6857429Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T08:49:08.6857886Z encoder_outputs = self.encoder( 2025-12-04T08:49:08.6858310Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T08:49:08.6858734Z layer_outputs = layer_module( 2025-12-04T08:49:08.6859129Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:49:08.6859537Z return super().__call__(*args, **kwargs) 2025-12-04T08:49:08.6859956Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.6860385Z return func(*args, **kwargs) 2025-12-04T08:49:08.6860795Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T08:49:08.6861248Z self_attention_outputs = self.attention( 2025-12-04T08:49:08.6861669Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.6862087Z return func(*args, **kwargs) 2025-12-04T08:49:08.6862498Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 488, in forward 2025-12-04T08:49:08.6862919Z self_outputs = self.self( 2025-12-04T08:49:08.6863360Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.6863786Z return func(*args, **kwargs) 2025-12-04T08:49:08.6864201Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 413, in forward 2025-12-04T08:49:08.6864695Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T08:49:08.6864910Z 2025-12-04T08:49:08.6865037Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:49:08.6865433Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:49:08.6865785Z res = mod(**inputs) 2025-12-04T08:49:08.6866173Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T08:49:08.6866629Z outputs = self.bert( 2025-12-04T08:49:08.6867023Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T08:49:08.6867433Z encoder_outputs = self.encoder( 2025-12-04T08:49:08.6867845Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T08:49:08.6868261Z layer_outputs = layer_module( 2025-12-04T08:49:08.6868640Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:49:08.6869030Z return super().__call__(*args, **kwargs) 2025-12-04T08:49:08.6869448Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.6869853Z return func(*args, **kwargs) 2025-12-04T08:49:08.6870250Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T08:49:08.6870704Z self_attention_outputs = self.attention( 2025-12-04T08:49:08.6871125Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.6871545Z return func(*args, **kwargs) 2025-12-04T08:49:08.6871952Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 497, in forward 2025-12-04T08:49:08.6872441Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T08:49:08.6872925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 436, in forward 2025-12-04T08:49:08.6873364Z hidden_states = self.dense(hidden_states) 2025-12-04T08:49:08.6873519Z 2025-12-04T08:49:08.6873636Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:49:08.6874047Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:49:08.6874413Z res = mod(**inputs) 2025-12-04T08:49:08.6874811Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T08:49:08.6875239Z outputs = self.bert( 2025-12-04T08:49:08.6875643Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T08:49:08.6876075Z encoder_outputs = self.encoder( 2025-12-04T08:49:08.6876491Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T08:49:08.6876917Z layer_outputs = layer_module( 2025-12-04T08:49:08.6877311Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:49:08.6877710Z return super().__call__(*args, **kwargs) 2025-12-04T08:49:08.6878142Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.6878688Z return func(*args, **kwargs) 2025-12-04T08:49:08.6879117Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 588, in forward 2025-12-04T08:49:08.6879562Z layer_output = apply_chunking_to_forward( 2025-12-04T08:49:08.6880025Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:49:08.6880474Z return forward_fn(*input_tensors) 2025-12-04T08:49:08.6880938Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 596, in feed_forward_chunk 2025-12-04T08:49:08.6881459Z intermediate_output = self.intermediate(attention_output) 2025-12-04T08:49:08.6881941Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 512, in forward 2025-12-04T08:49:08.6882426Z hidden_states = self.dense(hidden_states) 2025-12-04T08:49:08.6882580Z 2025-12-04T08:49:08.6882701Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:49:08.6883121Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:49:08.6883477Z res = mod(**inputs) 2025-12-04T08:49:08.6883871Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T08:49:08.6884282Z outputs = self.bert( 2025-12-04T08:49:08.6884677Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T08:49:08.6885105Z encoder_outputs = self.encoder( 2025-12-04T08:49:08.6885521Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T08:49:08.6885935Z layer_outputs = layer_module( 2025-12-04T08:49:08.6886330Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:49:08.6886733Z return super().__call__(*args, **kwargs) 2025-12-04T08:49:08.6887147Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.6887557Z return func(*args, **kwargs) 2025-12-04T08:49:08.6887963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 588, in forward 2025-12-04T08:49:08.6888402Z layer_output = apply_chunking_to_forward( 2025-12-04T08:49:08.6888837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:49:08.6889272Z return forward_fn(*input_tensors) 2025-12-04T08:49:08.6889722Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 596, in feed_forward_chunk 2025-12-04T08:49:08.6890224Z intermediate_output = self.intermediate(attention_output) 2025-12-04T08:49:08.6890699Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 513, in forward 2025-12-04T08:49:08.6891159Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:49:08.6891583Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T08:49:08.6891959Z return self.act(input) 2025-12-04T08:49:08.6892088Z 2025-12-04T08:49:08.6892202Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:49:08.6892606Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:49:08.6892964Z res = mod(**inputs) 2025-12-04T08:49:08.6893347Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T08:49:08.6893766Z outputs = self.bert( 2025-12-04T08:49:08.6894204Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T08:49:08.6894626Z encoder_outputs = self.encoder( 2025-12-04T08:49:08.6895048Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T08:49:08.6895473Z layer_outputs = layer_module( 2025-12-04T08:49:08.6895865Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:49:08.6896261Z return super().__call__(*args, **kwargs) 2025-12-04T08:49:08.6896685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.6897103Z return func(*args, **kwargs) 2025-12-04T08:49:08.6897505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 588, in forward 2025-12-04T08:49:08.6897996Z layer_output = apply_chunking_to_forward( 2025-12-04T08:49:08.6898443Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:49:08.6898876Z return forward_fn(*input_tensors) 2025-12-04T08:49:08.6899318Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 597, in feed_forward_chunk 2025-12-04T08:49:08.6899836Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T08:49:08.6900315Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 525, in forward 2025-12-04T08:49:08.6900784Z hidden_states = self.dense(hidden_states) 2025-12-04T08:49:08.6900933Z 2025-12-04T08:49:08.6901048Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:49:08.6901446Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:49:08.6901807Z res = mod(**inputs) 2025-12-04T08:49:08.6902190Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T08:49:08.6902604Z outputs = self.bert( 2025-12-04T08:49:08.6903006Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T08:49:08.6903483Z encoder_outputs = self.encoder( 2025-12-04T08:49:08.6903887Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T08:49:08.6904301Z layer_outputs = layer_module( 2025-12-04T08:49:08.6904684Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:49:08.6905086Z return super().__call__(*args, **kwargs) 2025-12-04T08:49:08.6905520Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.6905947Z return func(*args, **kwargs) 2025-12-04T08:49:08.6906367Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T08:49:08.6906813Z self_attention_outputs = self.attention( 2025-12-04T08:49:08.6907247Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.6907680Z return func(*args, **kwargs) 2025-12-04T08:49:08.6908095Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 488, in forward 2025-12-04T08:49:08.6908518Z self_outputs = self.self( 2025-12-04T08:49:08.6908921Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.6909350Z return func(*args, **kwargs) 2025-12-04T08:49:08.6909790Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 363, in forward 2025-12-04T08:49:08.6910380Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-12-04T08:49:08.6910684Z 2025-12-04T08:49:08.6910803Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:49:08.6911216Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:49:08.6911574Z res = mod(**inputs) 2025-12-04T08:49:08.6911976Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T08:49:08.6912405Z outputs = self.bert( 2025-12-04T08:49:08.6912804Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T08:49:08.6913287Z encoder_outputs = self.encoder( 2025-12-04T08:49:08.6913713Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T08:49:08.6914139Z layer_outputs = layer_module( 2025-12-04T08:49:08.6914521Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:49:08.6914932Z return super().__call__(*args, **kwargs) 2025-12-04T08:49:08.6915363Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.6915779Z return func(*args, **kwargs) 2025-12-04T08:49:08.6916189Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T08:49:08.6916704Z self_attention_outputs = self.attention( 2025-12-04T08:49:08.6917142Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.6917565Z return func(*args, **kwargs) 2025-12-04T08:49:08.6917987Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 488, in forward 2025-12-04T08:49:08.6918483Z self_outputs = self.self( 2025-12-04T08:49:08.6918894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.6919305Z return func(*args, **kwargs) 2025-12-04T08:49:08.6919722Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 387, in forward 2025-12-04T08:49:08.6920149Z self.key(current_states) 2025-12-04T08:49:08.6920279Z 2025-12-04T08:49:08.6920398Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:49:08.6920923Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:49:08.6921299Z res = mod(**inputs) 2025-12-04T08:49:08.6921721Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T08:49:08.6922142Z outputs = self.bert( 2025-12-04T08:49:08.6922544Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T08:49:08.6922975Z encoder_outputs = self.encoder( 2025-12-04T08:49:08.6923397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T08:49:08.6923812Z layer_outputs = layer_module( 2025-12-04T08:49:08.6924208Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:49:08.6924629Z return super().__call__(*args, **kwargs) 2025-12-04T08:49:08.6925057Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.6925489Z return func(*args, **kwargs) 2025-12-04T08:49:08.6925983Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T08:49:08.6926419Z self_attention_outputs = self.attention( 2025-12-04T08:49:08.6926845Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.6927265Z return func(*args, **kwargs) 2025-12-04T08:49:08.6927678Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 488, in forward 2025-12-04T08:49:08.6928099Z self_outputs = self.self( 2025-12-04T08:49:08.6928503Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.6928928Z return func(*args, **kwargs) 2025-12-04T08:49:08.6929343Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 392, in forward 2025-12-04T08:49:08.6929824Z self.value(current_states) 2025-12-04T08:49:08.6929968Z 2025-12-04T08:49:08.6930062Z cudagraph partition due to non gpu ops 2025-12-04T08:49:08.6930336Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:49:08.6930741Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:49:08.6931105Z res = mod(**inputs) 2025-12-04T08:49:08.6931510Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T08:49:08.6931937Z outputs = self.bert( 2025-12-04T08:49:08.6932335Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T08:49:08.6932768Z encoder_outputs = self.encoder( 2025-12-04T08:49:08.6933192Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T08:49:08.6933625Z layer_outputs = layer_module( 2025-12-04T08:49:08.6934002Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:49:08.6934416Z return super().__call__(*args, **kwargs) 2025-12-04T08:49:08.6934842Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.6935255Z return func(*args, **kwargs) 2025-12-04T08:49:08.6935668Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T08:49:08.6936109Z self_attention_outputs = self.attention( 2025-12-04T08:49:08.6936537Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.6936949Z return func(*args, **kwargs) 2025-12-04T08:49:08.6937367Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 488, in forward 2025-12-04T08:49:08.6937795Z self_outputs = self.self( 2025-12-04T08:49:08.6938193Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.6938611Z return func(*args, **kwargs) 2025-12-04T08:49:08.6939025Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 413, in forward 2025-12-04T08:49:08.6939519Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T08:49:08.6939725Z 2025-12-04T08:49:08.6939843Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:49:08.6940251Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:49:08.6940614Z res = mod(**inputs) 2025-12-04T08:49:08.6941007Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T08:49:08.6941439Z outputs = self.bert( 2025-12-04T08:49:08.6941884Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T08:49:08.6942318Z encoder_outputs = self.encoder( 2025-12-04T08:49:08.6942731Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T08:49:08.6943157Z layer_outputs = layer_module( 2025-12-04T08:49:08.6943545Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:49:08.6943957Z return super().__call__(*args, **kwargs) 2025-12-04T08:49:08.6944379Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.6944795Z return func(*args, **kwargs) 2025-12-04T08:49:08.6945246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T08:49:08.6945683Z self_attention_outputs = self.attention( 2025-12-04T08:49:08.6946117Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.6946535Z return func(*args, **kwargs) 2025-12-04T08:49:08.6946952Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 497, in forward 2025-12-04T08:49:08.6947436Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T08:49:08.6947951Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 436, in forward 2025-12-04T08:49:08.6948395Z hidden_states = self.dense(hidden_states) 2025-12-04T08:49:08.6948552Z 2025-12-04T08:49:08.6948679Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:49:08.6949082Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:49:08.6949448Z res = mod(**inputs) 2025-12-04T08:49:08.6949851Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T08:49:08.6950272Z outputs = self.bert( 2025-12-04T08:49:08.6950677Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T08:49:08.6951112Z encoder_outputs = self.encoder( 2025-12-04T08:49:08.6951535Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T08:49:08.6951956Z layer_outputs = layer_module( 2025-12-04T08:49:08.6952344Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:49:08.6952754Z return super().__call__(*args, **kwargs) 2025-12-04T08:49:08.6953189Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.6953613Z return func(*args, **kwargs) 2025-12-04T08:49:08.6954017Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 588, in forward 2025-12-04T08:49:08.6954448Z layer_output = apply_chunking_to_forward( 2025-12-04T08:49:08.6954887Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:49:08.6955327Z return forward_fn(*input_tensors) 2025-12-04T08:49:08.6955774Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 596, in feed_forward_chunk 2025-12-04T08:49:08.6956296Z intermediate_output = self.intermediate(attention_output) 2025-12-04T08:49:08.6956792Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 512, in forward 2025-12-04T08:49:08.6957247Z hidden_states = self.dense(hidden_states) 2025-12-04T08:49:08.6957444Z 2025-12-04T08:49:08.6957573Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:49:08.6957972Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:49:08.6958428Z res = mod(**inputs) 2025-12-04T08:49:08.6958839Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T08:49:08.6959271Z outputs = self.bert( 2025-12-04T08:49:08.6959668Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T08:49:08.6960102Z encoder_outputs = self.encoder( 2025-12-04T08:49:08.6960550Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T08:49:08.6961035Z layer_outputs = layer_module( 2025-12-04T08:49:08.6961426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:49:08.6961829Z return super().__call__(*args, **kwargs) 2025-12-04T08:49:08.6962254Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.6962687Z return func(*args, **kwargs) 2025-12-04T08:49:08.6963107Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 588, in forward 2025-12-04T08:49:08.6963539Z layer_output = apply_chunking_to_forward( 2025-12-04T08:49:08.6963977Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:49:08.6964411Z return forward_fn(*input_tensors) 2025-12-04T08:49:08.6964871Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 596, in feed_forward_chunk 2025-12-04T08:49:08.6965394Z intermediate_output = self.intermediate(attention_output) 2025-12-04T08:49:08.6965849Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 513, in forward 2025-12-04T08:49:08.6966311Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:49:08.6966729Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T08:49:08.6967105Z return self.act(input) 2025-12-04T08:49:08.6967229Z 2025-12-04T08:49:08.6967343Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:49:08.6967743Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:49:08.6968106Z res = mod(**inputs) 2025-12-04T08:49:08.6968519Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T08:49:08.6968948Z outputs = self.bert( 2025-12-04T08:49:08.6969372Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T08:49:08.6969811Z encoder_outputs = self.encoder( 2025-12-04T08:49:08.6970241Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T08:49:08.6970682Z layer_outputs = layer_module( 2025-12-04T08:49:08.6971087Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:49:08.6971509Z return super().__call__(*args, **kwargs) 2025-12-04T08:49:08.6971944Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.6972373Z return func(*args, **kwargs) 2025-12-04T08:49:08.6972809Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 588, in forward 2025-12-04T08:49:08.6973316Z layer_output = apply_chunking_to_forward( 2025-12-04T08:49:08.6973809Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:49:08.6974290Z return forward_fn(*input_tensors) 2025-12-04T08:49:08.6974772Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 597, in feed_forward_chunk 2025-12-04T08:49:08.6975322Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T08:49:08.6975843Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 525, in forward 2025-12-04T08:49:08.6976309Z hidden_states = self.dense(hidden_states) 2025-12-04T08:49:08.6976477Z 2025-12-04T08:49:08.6976614Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:49:08.6977059Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:49:08.6977425Z res = mod(**inputs) 2025-12-04T08:49:08.6977830Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T08:49:08.6978252Z outputs = self.bert( 2025-12-04T08:49:08.6978654Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T08:49:08.6979087Z encoder_outputs = self.encoder( 2025-12-04T08:49:08.6979508Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T08:49:08.6979931Z layer_outputs = layer_module( 2025-12-04T08:49:08.6980327Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:49:08.6980737Z return super().__call__(*args, **kwargs) 2025-12-04T08:49:08.6981167Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.6981588Z return func(*args, **kwargs) 2025-12-04T08:49:08.6981992Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T08:49:08.6982421Z self_attention_outputs = self.attention( 2025-12-04T08:49:08.6982833Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.6983237Z return func(*args, **kwargs) 2025-12-04T08:49:08.6983642Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 488, in forward 2025-12-04T08:49:08.6984047Z self_outputs = self.self( 2025-12-04T08:49:08.6984442Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.6984851Z return func(*args, **kwargs) 2025-12-04T08:49:08.6985252Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 363, in forward 2025-12-04T08:49:08.6985815Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-12-04T08:49:08.6986111Z 2025-12-04T08:49:08.6986226Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:49:08.6986625Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:49:08.6986980Z res = mod(**inputs) 2025-12-04T08:49:08.6987364Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T08:49:08.6987785Z outputs = self.bert( 2025-12-04T08:49:08.6988178Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T08:49:08.6988596Z encoder_outputs = self.encoder( 2025-12-04T08:49:08.6989056Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T08:49:08.6989484Z layer_outputs = layer_module( 2025-12-04T08:49:08.6989877Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:49:08.6990281Z return super().__call__(*args, **kwargs) 2025-12-04T08:49:08.6990709Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.6991129Z return func(*args, **kwargs) 2025-12-04T08:49:08.6991532Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T08:49:08.6991975Z self_attention_outputs = self.attention( 2025-12-04T08:49:08.6992405Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.6992872Z return func(*args, **kwargs) 2025-12-04T08:49:08.6993285Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 488, in forward 2025-12-04T08:49:08.6993714Z self_outputs = self.self( 2025-12-04T08:49:08.6994121Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.6994539Z return func(*args, **kwargs) 2025-12-04T08:49:08.6994946Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 387, in forward 2025-12-04T08:49:08.6995375Z self.key(current_states) 2025-12-04T08:49:08.6995503Z 2025-12-04T08:49:08.6995628Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:49:08.6996028Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:49:08.6996397Z res = mod(**inputs) 2025-12-04T08:49:08.6996804Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T08:49:08.6997230Z outputs = self.bert( 2025-12-04T08:49:08.6997630Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T08:49:08.6998061Z encoder_outputs = self.encoder( 2025-12-04T08:49:08.6998571Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T08:49:08.6998997Z layer_outputs = layer_module( 2025-12-04T08:49:08.6999394Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:49:08.6999807Z return super().__call__(*args, **kwargs) 2025-12-04T08:49:08.7000233Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.7000651Z return func(*args, **kwargs) 2025-12-04T08:49:08.7001073Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T08:49:08.7001512Z self_attention_outputs = self.attention( 2025-12-04T08:49:08.7001945Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.7002357Z return func(*args, **kwargs) 2025-12-04T08:49:08.7002766Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 488, in forward 2025-12-04T08:49:08.7003191Z self_outputs = self.self( 2025-12-04T08:49:08.7003586Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.7004002Z return func(*args, **kwargs) 2025-12-04T08:49:08.7004411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 392, in forward 2025-12-04T08:49:08.7004888Z self.value(current_states) 2025-12-04T08:49:08.7005021Z 2025-12-04T08:49:08.7005111Z cudagraph partition due to non gpu ops 2025-12-04T08:49:08.7005381Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:49:08.7005787Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:49:08.7006155Z res = mod(**inputs) 2025-12-04T08:49:08.7006559Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T08:49:08.7006983Z outputs = self.bert( 2025-12-04T08:49:08.7007401Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T08:49:08.7007833Z encoder_outputs = self.encoder( 2025-12-04T08:49:08.7008255Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T08:49:08.7008738Z layer_outputs = layer_module( 2025-12-04T08:49:08.7009128Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:49:08.7009541Z return super().__call__(*args, **kwargs) 2025-12-04T08:49:08.7009965Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.7010390Z return func(*args, **kwargs) 2025-12-04T08:49:08.7010795Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T08:49:08.7011532Z self_attention_outputs = self.attention( 2025-12-04T08:49:08.7011959Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.7012380Z return func(*args, **kwargs) 2025-12-04T08:49:08.7012814Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 488, in forward 2025-12-04T08:49:08.7013245Z self_outputs = self.self( 2025-12-04T08:49:08.7013661Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.7014081Z return func(*args, **kwargs) 2025-12-04T08:49:08.7014514Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 413, in forward 2025-12-04T08:49:08.7015028Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T08:49:08.7015256Z 2025-12-04T08:49:08.7015380Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:49:08.7015788Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:49:08.7016149Z res = mod(**inputs) 2025-12-04T08:49:08.7016571Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T08:49:08.7017016Z outputs = self.bert( 2025-12-04T08:49:08.7017418Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T08:49:08.7017864Z encoder_outputs = self.encoder( 2025-12-04T08:49:08.7018297Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T08:49:08.7018722Z layer_outputs = layer_module( 2025-12-04T08:49:08.7019124Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:49:08.7019543Z return super().__call__(*args, **kwargs) 2025-12-04T08:49:08.7019969Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.7020368Z return func(*args, **kwargs) 2025-12-04T08:49:08.7021010Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T08:49:08.7021455Z self_attention_outputs = self.attention( 2025-12-04T08:49:08.7021870Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.7022282Z return func(*args, **kwargs) 2025-12-04T08:49:08.7022695Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 497, in forward 2025-12-04T08:49:08.7023182Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T08:49:08.7023657Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 436, in forward 2025-12-04T08:49:08.7024095Z hidden_states = self.dense(hidden_states) 2025-12-04T08:49:08.7024253Z 2025-12-04T08:49:08.7024380Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:49:08.7024829Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:49:08.7025184Z res = mod(**inputs) 2025-12-04T08:49:08.7025575Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T08:49:08.7025992Z outputs = self.bert( 2025-12-04T08:49:08.7026390Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T08:49:08.7026815Z encoder_outputs = self.encoder( 2025-12-04T08:49:08.7027232Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T08:49:08.7027650Z layer_outputs = layer_module( 2025-12-04T08:49:08.7028023Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:49:08.7028427Z return super().__call__(*args, **kwargs) 2025-12-04T08:49:08.7028847Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.7029253Z return func(*args, **kwargs) 2025-12-04T08:49:08.7029652Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 588, in forward 2025-12-04T08:49:08.7030083Z layer_output = apply_chunking_to_forward( 2025-12-04T08:49:08.7030529Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:49:08.7030971Z return forward_fn(*input_tensors) 2025-12-04T08:49:08.7031417Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 596, in feed_forward_chunk 2025-12-04T08:49:08.7031917Z intermediate_output = self.intermediate(attention_output) 2025-12-04T08:49:08.7032387Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 512, in forward 2025-12-04T08:49:08.7032815Z hidden_states = self.dense(hidden_states) 2025-12-04T08:49:08.7032972Z 2025-12-04T08:49:08.7033085Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:49:08.7033485Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:49:08.7033850Z res = mod(**inputs) 2025-12-04T08:49:08.7034265Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T08:49:08.7034710Z outputs = self.bert( 2025-12-04T08:49:08.7035126Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T08:49:08.7035570Z encoder_outputs = self.encoder( 2025-12-04T08:49:08.7036002Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T08:49:08.7036434Z layer_outputs = layer_module( 2025-12-04T08:49:08.7036869Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:49:08.7037299Z return super().__call__(*args, **kwargs) 2025-12-04T08:49:08.7037737Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.7038219Z return func(*args, **kwargs) 2025-12-04T08:49:08.7038640Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 588, in forward 2025-12-04T08:49:08.7039084Z layer_output = apply_chunking_to_forward( 2025-12-04T08:49:08.7039551Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:49:08.7039982Z return forward_fn(*input_tensors) 2025-12-04T08:49:08.7040466Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 596, in feed_forward_chunk 2025-12-04T08:49:08.7040972Z intermediate_output = self.intermediate(attention_output) 2025-12-04T08:49:08.7041443Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 513, in forward 2025-12-04T08:49:08.7041895Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:49:08.7042318Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T08:49:08.7042701Z return self.act(input) 2025-12-04T08:49:08.7042822Z 2025-12-04T08:49:08.7042943Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:49:08.7043331Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:49:08.7043686Z res = mod(**inputs) 2025-12-04T08:49:08.7044076Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T08:49:08.7044486Z outputs = self.bert( 2025-12-04T08:49:08.7044883Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T08:49:08.7045309Z encoder_outputs = self.encoder( 2025-12-04T08:49:08.7045722Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T08:49:08.7046131Z layer_outputs = layer_module( 2025-12-04T08:49:08.7046515Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:49:08.7046919Z return super().__call__(*args, **kwargs) 2025-12-04T08:49:08.7047339Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.7047743Z return func(*args, **kwargs) 2025-12-04T08:49:08.7048156Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 588, in forward 2025-12-04T08:49:08.7048586Z layer_output = apply_chunking_to_forward( 2025-12-04T08:49:08.7049021Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:49:08.7049604Z return forward_fn(*input_tensors) 2025-12-04T08:49:08.7050060Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 597, in feed_forward_chunk 2025-12-04T08:49:08.7050575Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T08:49:08.7051048Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 525, in forward 2025-12-04T08:49:08.7051495Z hidden_states = self.dense(hidden_states) 2025-12-04T08:49:08.7051646Z 2025-12-04T08:49:08.7051772Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:49:08.7052220Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:49:08.7052571Z res = mod(**inputs) 2025-12-04T08:49:08.7052963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T08:49:08.7053383Z outputs = self.bert( 2025-12-04T08:49:08.7053768Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T08:49:08.7054191Z encoder_outputs = self.encoder( 2025-12-04T08:49:08.7054608Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T08:49:08.7055026Z layer_outputs = layer_module( 2025-12-04T08:49:08.7055402Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:49:08.7055831Z return super().__call__(*args, **kwargs) 2025-12-04T08:49:08.7056250Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.7056654Z return func(*args, **kwargs) 2025-12-04T08:49:08.7057056Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T08:49:08.7057484Z self_attention_outputs = self.attention( 2025-12-04T08:49:08.7057898Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.7058276Z return func(*args, **kwargs) 2025-12-04T08:49:08.7058654Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 488, in forward 2025-12-04T08:49:08.7059058Z self_outputs = self.self( 2025-12-04T08:49:08.7059443Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.7059847Z return func(*args, **kwargs) 2025-12-04T08:49:08.7060248Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 363, in forward 2025-12-04T08:49:08.7060820Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-12-04T08:49:08.7061104Z 2025-12-04T08:49:08.7061218Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:49:08.7061613Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:49:08.7061961Z res = mod(**inputs) 2025-12-04T08:49:08.7062352Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T08:49:08.7062757Z outputs = self.bert( 2025-12-04T08:49:08.7063149Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T08:49:08.7063572Z encoder_outputs = self.encoder( 2025-12-04T08:49:08.7063986Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T08:49:08.7064384Z layer_outputs = layer_module( 2025-12-04T08:49:08.7064759Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:49:08.7065160Z return super().__call__(*args, **kwargs) 2025-12-04T08:49:08.7065572Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.7065979Z return func(*args, **kwargs) 2025-12-04T08:49:08.7066383Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T08:49:08.7066814Z self_attention_outputs = self.attention( 2025-12-04T08:49:08.7067232Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.7067685Z return func(*args, **kwargs) 2025-12-04T08:49:08.7068090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 488, in forward 2025-12-04T08:49:08.7068501Z self_outputs = self.self( 2025-12-04T08:49:08.7068894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.7069297Z return func(*args, **kwargs) 2025-12-04T08:49:08.7069694Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 387, in forward 2025-12-04T08:49:08.7070100Z self.key(current_states) 2025-12-04T08:49:08.7070233Z 2025-12-04T08:49:08.7070347Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:49:08.7070764Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:49:08.7071152Z res = mod(**inputs) 2025-12-04T08:49:08.7071546Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T08:49:08.7071961Z outputs = self.bert( 2025-12-04T08:49:08.7072354Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T08:49:08.7072774Z encoder_outputs = self.encoder( 2025-12-04T08:49:08.7073184Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T08:49:08.7073619Z layer_outputs = layer_module( 2025-12-04T08:49:08.7074002Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:49:08.7074417Z return super().__call__(*args, **kwargs) 2025-12-04T08:49:08.7074846Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.7075276Z return func(*args, **kwargs) 2025-12-04T08:49:08.7075707Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T08:49:08.7076167Z self_attention_outputs = self.attention( 2025-12-04T08:49:08.7076597Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.7077015Z return func(*args, **kwargs) 2025-12-04T08:49:08.7077429Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 488, in forward 2025-12-04T08:49:08.7077846Z self_outputs = self.self( 2025-12-04T08:49:08.7078295Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.7078703Z return func(*args, **kwargs) 2025-12-04T08:49:08.7079135Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 392, in forward 2025-12-04T08:49:08.7079578Z self.value(current_states) 2025-12-04T08:49:08.7079711Z 2025-12-04T08:49:08.7079807Z cudagraph partition due to non gpu ops 2025-12-04T08:49:08.7080079Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:49:08.7080472Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:49:08.7080825Z res = mod(**inputs) 2025-12-04T08:49:08.7081212Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T08:49:08.7081631Z outputs = self.bert( 2025-12-04T08:49:08.7082031Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T08:49:08.7082460Z encoder_outputs = self.encoder( 2025-12-04T08:49:08.7082872Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T08:49:08.7083354Z layer_outputs = layer_module( 2025-12-04T08:49:08.7083744Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:49:08.7084142Z return super().__call__(*args, **kwargs) 2025-12-04T08:49:08.7084564Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.7084978Z return func(*args, **kwargs) 2025-12-04T08:49:08.7085384Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T08:49:08.7085813Z self_attention_outputs = self.attention( 2025-12-04T08:49:08.7086238Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.7086686Z return func(*args, **kwargs) 2025-12-04T08:49:08.7087087Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 488, in forward 2025-12-04T08:49:08.7087506Z self_outputs = self.self( 2025-12-04T08:49:08.7087901Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.7088306Z return func(*args, **kwargs) 2025-12-04T08:49:08.7088699Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 413, in forward 2025-12-04T08:49:08.7089181Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T08:49:08.7089388Z 2025-12-04T08:49:08.7089510Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:49:08.7089872Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:49:08.7090189Z res = mod(**inputs) 2025-12-04T08:49:08.7090553Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T08:49:08.7090933Z outputs = self.bert( 2025-12-04T08:49:08.7091289Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T08:49:08.7091677Z encoder_outputs = self.encoder( 2025-12-04T08:49:08.7092057Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T08:49:08.7092447Z layer_outputs = layer_module( 2025-12-04T08:49:08.7092797Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:49:08.7093163Z return super().__call__(*args, **kwargs) 2025-12-04T08:49:08.7093543Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.7093911Z return func(*args, **kwargs) 2025-12-04T08:49:08.7094280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T08:49:08.7094672Z self_attention_outputs = self.attention( 2025-12-04T08:49:08.7095054Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.7095419Z return func(*args, **kwargs) 2025-12-04T08:49:08.7095786Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 497, in forward 2025-12-04T08:49:08.7096227Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T08:49:08.7096669Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 436, in forward 2025-12-04T08:49:08.7097064Z hidden_states = self.dense(hidden_states) 2025-12-04T08:49:08.7097212Z 2025-12-04T08:49:08.7097323Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:49:08.7097736Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:49:08.7098063Z res = mod(**inputs) 2025-12-04T08:49:08.7098430Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T08:49:08.7098506Z outputs = self.bert( 2025-12-04T08:49:08.7098762Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T08:49:08.7098838Z encoder_outputs = self.encoder( 2025-12-04T08:49:08.7099095Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T08:49:08.7099169Z layer_outputs = layer_module( 2025-12-04T08:49:08.7099397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:49:08.7099525Z return super().__call__(*args, **kwargs) 2025-12-04T08:49:08.7099779Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.7099857Z return func(*args, **kwargs) 2025-12-04T08:49:08.7100116Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 588, in forward 2025-12-04T08:49:08.7100205Z layer_output = apply_chunking_to_forward( 2025-12-04T08:49:08.7100482Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:49:08.7100562Z return forward_fn(*input_tensors) 2025-12-04T08:49:08.7100866Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 596, in feed_forward_chunk 2025-12-04T08:49:08.7100990Z intermediate_output = self.intermediate(attention_output) 2025-12-04T08:49:08.7101251Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 512, in forward 2025-12-04T08:49:08.7101344Z hidden_states = self.dense(hidden_states) 2025-12-04T08:49:08.7101348Z 2025-12-04T08:49:08.7101454Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:49:08.7101659Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:49:08.7101733Z res = mod(**inputs) 2025-12-04T08:49:08.7101990Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T08:49:08.7102067Z outputs = self.bert( 2025-12-04T08:49:08.7102323Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T08:49:08.7102400Z encoder_outputs = self.encoder( 2025-12-04T08:49:08.7102661Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T08:49:08.7102738Z layer_outputs = layer_module( 2025-12-04T08:49:08.7102972Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:49:08.7103060Z return super().__call__(*args, **kwargs) 2025-12-04T08:49:08.7103311Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.7103389Z return func(*args, **kwargs) 2025-12-04T08:49:08.7103643Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 588, in forward 2025-12-04T08:49:08.7103728Z layer_output = apply_chunking_to_forward( 2025-12-04T08:49:08.7104011Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:49:08.7104089Z return forward_fn(*input_tensors) 2025-12-04T08:49:08.7104417Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 596, in feed_forward_chunk 2025-12-04T08:49:08.7104541Z intermediate_output = self.intermediate(attention_output) 2025-12-04T08:49:08.7104794Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 513, in forward 2025-12-04T08:49:08.7104921Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:49:08.7105142Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T08:49:08.7105214Z return self.act(input) 2025-12-04T08:49:08.7105225Z 2025-12-04T08:49:08.7105334Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:49:08.7105540Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:49:08.7105614Z res = mod(**inputs) 2025-12-04T08:49:08.7105911Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T08:49:08.7105982Z outputs = self.bert( 2025-12-04T08:49:08.7106246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T08:49:08.7106323Z encoder_outputs = self.encoder( 2025-12-04T08:49:08.7106580Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T08:49:08.7106653Z layer_outputs = layer_module( 2025-12-04T08:49:08.7106881Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:49:08.7106970Z return super().__call__(*args, **kwargs) 2025-12-04T08:49:08.7107221Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.7107296Z return func(*args, **kwargs) 2025-12-04T08:49:08.7107556Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 588, in forward 2025-12-04T08:49:08.7107642Z layer_output = apply_chunking_to_forward( 2025-12-04T08:49:08.7107913Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:49:08.7107991Z return forward_fn(*input_tensors) 2025-12-04T08:49:08.7108273Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 597, in feed_forward_chunk 2025-12-04T08:49:08.7108417Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T08:49:08.7108670Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 525, in forward 2025-12-04T08:49:08.7108760Z hidden_states = self.dense(hidden_states) 2025-12-04T08:49:08.7108763Z 2025-12-04T08:49:08.7108874Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:49:08.7109082Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:49:08.7109154Z res = mod(**inputs) 2025-12-04T08:49:08.7109409Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T08:49:08.7109482Z outputs = self.bert( 2025-12-04T08:49:08.7109760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T08:49:08.7109838Z encoder_outputs = self.encoder( 2025-12-04T08:49:08.7110113Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T08:49:08.7110191Z layer_outputs = layer_module( 2025-12-04T08:49:08.7110431Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:49:08.7110527Z return super().__call__(*args, **kwargs) 2025-12-04T08:49:08.7110825Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.7110901Z return func(*args, **kwargs) 2025-12-04T08:49:08.7111180Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T08:49:08.7111270Z self_attention_outputs = self.attention( 2025-12-04T08:49:08.7111542Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.7111613Z return func(*args, **kwargs) 2025-12-04T08:49:08.7111871Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 488, in forward 2025-12-04T08:49:08.7111952Z self_outputs = self.self( 2025-12-04T08:49:08.7112206Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.7112329Z return func(*args, **kwargs) 2025-12-04T08:49:08.7112582Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 363, in forward 2025-12-04T08:49:08.7112797Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-12-04T08:49:08.7112800Z 2025-12-04T08:49:08.7112913Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:49:08.7113117Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:49:08.7113181Z res = mod(**inputs) 2025-12-04T08:49:08.7113446Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T08:49:08.7113514Z outputs = self.bert( 2025-12-04T08:49:08.7113776Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T08:49:08.7113856Z encoder_outputs = self.encoder( 2025-12-04T08:49:08.7114111Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T08:49:08.7114190Z layer_outputs = layer_module( 2025-12-04T08:49:08.7114418Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:49:08.7114503Z return super().__call__(*args, **kwargs) 2025-12-04T08:49:08.7114752Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.7114822Z return func(*args, **kwargs) 2025-12-04T08:49:08.7115083Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T08:49:08.7115166Z self_attention_outputs = self.attention( 2025-12-04T08:49:08.7115418Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.7115494Z return func(*args, **kwargs) 2025-12-04T08:49:08.7115746Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 488, in forward 2025-12-04T08:49:08.7115827Z self_outputs = self.self( 2025-12-04T08:49:08.7116075Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.7116147Z return func(*args, **kwargs) 2025-12-04T08:49:08.7116422Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 387, in forward 2025-12-04T08:49:08.7116500Z self.key(current_states) 2025-12-04T08:49:08.7116503Z 2025-12-04T08:49:08.7116615Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:49:08.7116846Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:49:08.7116920Z res = mod(**inputs) 2025-12-04T08:49:08.7117238Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T08:49:08.7117312Z outputs = self.bert( 2025-12-04T08:49:08.7117587Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T08:49:08.7117670Z encoder_outputs = self.encoder( 2025-12-04T08:49:08.7117923Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T08:49:08.7118003Z layer_outputs = layer_module( 2025-12-04T08:49:08.7118303Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:49:08.7118390Z return super().__call__(*args, **kwargs) 2025-12-04T08:49:08.7118696Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.7118774Z return func(*args, **kwargs) 2025-12-04T08:49:08.7119043Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T08:49:08.7119142Z self_attention_outputs = self.attention( 2025-12-04T08:49:08.7119404Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.7119485Z return func(*args, **kwargs) 2025-12-04T08:49:08.7119755Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 488, in forward 2025-12-04T08:49:08.7119831Z self_outputs = self.self( 2025-12-04T08:49:08.7120101Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.7120179Z return func(*args, **kwargs) 2025-12-04T08:49:08.7120452Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 392, in forward 2025-12-04T08:49:08.7120540Z self.value(current_states) 2025-12-04T08:49:08.7120544Z 2025-12-04T08:49:08.7120634Z cudagraph partition due to non gpu ops 2025-12-04T08:49:08.7120908Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:49:08.7121132Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:49:08.7121204Z res = mod(**inputs) 2025-12-04T08:49:08.7121486Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T08:49:08.7121569Z outputs = self.bert( 2025-12-04T08:49:08.7121843Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T08:49:08.7121933Z encoder_outputs = self.encoder( 2025-12-04T08:49:08.7122213Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T08:49:08.7122301Z layer_outputs = layer_module( 2025-12-04T08:49:08.7122545Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:49:08.7122630Z return super().__call__(*args, **kwargs) 2025-12-04T08:49:08.7122903Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.7122978Z return func(*args, **kwargs) 2025-12-04T08:49:08.7123252Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T08:49:08.7123335Z self_attention_outputs = self.attention( 2025-12-04T08:49:08.7123642Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.7123727Z return func(*args, **kwargs) 2025-12-04T08:49:08.7124065Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 488, in forward 2025-12-04T08:49:08.7124142Z self_outputs = self.self( 2025-12-04T08:49:08.7124417Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.7124489Z return func(*args, **kwargs) 2025-12-04T08:49:08.7124772Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 413, in forward 2025-12-04T08:49:08.7124921Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T08:49:08.7124926Z 2025-12-04T08:49:08.7125036Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:49:08.7125262Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:49:08.7125393Z res = mod(**inputs) 2025-12-04T08:49:08.7125679Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T08:49:08.7125750Z outputs = self.bert( 2025-12-04T08:49:08.7126026Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T08:49:08.7126112Z encoder_outputs = self.encoder( 2025-12-04T08:49:08.7126381Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T08:49:08.7126459Z layer_outputs = layer_module( 2025-12-04T08:49:08.7126708Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:49:08.7126791Z return super().__call__(*args, **kwargs) 2025-12-04T08:49:08.7127059Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.7127137Z return func(*args, **kwargs) 2025-12-04T08:49:08.7127409Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T08:49:08.7127504Z self_attention_outputs = self.attention( 2025-12-04T08:49:08.7127769Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.7127843Z return func(*args, **kwargs) 2025-12-04T08:49:08.7128122Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 497, in forward 2025-12-04T08:49:08.7128263Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T08:49:08.7128544Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 436, in forward 2025-12-04T08:49:08.7128635Z hidden_states = self.dense(hidden_states) 2025-12-04T08:49:08.7128641Z 2025-12-04T08:49:08.7128751Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:49:08.7128978Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:49:08.7129046Z res = mod(**inputs) 2025-12-04T08:49:08.7129326Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T08:49:08.7129396Z outputs = self.bert( 2025-12-04T08:49:08.7129666Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T08:49:08.7129749Z encoder_outputs = self.encoder( 2025-12-04T08:49:08.7130019Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T08:49:08.7130093Z layer_outputs = layer_module( 2025-12-04T08:49:08.7130340Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:49:08.7130426Z return super().__call__(*args, **kwargs) 2025-12-04T08:49:08.7130731Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.7130806Z return func(*args, **kwargs) 2025-12-04T08:49:08.7131078Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 588, in forward 2025-12-04T08:49:08.7131177Z layer_output = apply_chunking_to_forward( 2025-12-04T08:49:08.7131468Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:49:08.7131551Z return forward_fn(*input_tensors) 2025-12-04T08:49:08.7131863Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 596, in feed_forward_chunk 2025-12-04T08:49:08.7131993Z intermediate_output = self.intermediate(attention_output) 2025-12-04T08:49:08.7132304Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 512, in forward 2025-12-04T08:49:08.7132394Z hidden_states = self.dense(hidden_states) 2025-12-04T08:49:08.7132398Z 2025-12-04T08:49:08.7132510Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:49:08.7132735Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:49:08.7132805Z res = mod(**inputs) 2025-12-04T08:49:08.7133082Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T08:49:08.7133153Z outputs = self.bert( 2025-12-04T08:49:08.7133436Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T08:49:08.7133521Z encoder_outputs = self.encoder( 2025-12-04T08:49:08.7133794Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T08:49:08.7133873Z layer_outputs = layer_module( 2025-12-04T08:49:08.7134121Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:49:08.7134206Z return super().__call__(*args, **kwargs) 2025-12-04T08:49:08.7134478Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.7134552Z return func(*args, **kwargs) 2025-12-04T08:49:08.7134818Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 588, in forward 2025-12-04T08:49:08.7134916Z layer_output = apply_chunking_to_forward( 2025-12-04T08:49:08.7135200Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:49:08.7135293Z return forward_fn(*input_tensors) 2025-12-04T08:49:08.7135595Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 596, in feed_forward_chunk 2025-12-04T08:49:08.7135724Z intermediate_output = self.intermediate(attention_output) 2025-12-04T08:49:08.7136002Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 513, in forward 2025-12-04T08:49:08.7136124Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:49:08.7136359Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T08:49:08.7136443Z return self.act(input) 2025-12-04T08:49:08.7136447Z 2025-12-04T08:49:08.7136556Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:49:08.7136790Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:49:08.7136859Z res = mod(**inputs) 2025-12-04T08:49:08.7137178Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T08:49:08.7137256Z outputs = self.bert( 2025-12-04T08:49:08.7137515Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T08:49:08.7137598Z encoder_outputs = self.encoder( 2025-12-04T08:49:08.7137851Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T08:49:08.7137923Z layer_outputs = layer_module( 2025-12-04T08:49:08.7138157Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:49:08.7138235Z return super().__call__(*args, **kwargs) 2025-12-04T08:49:08.7138486Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.7138609Z return func(*args, **kwargs) 2025-12-04T08:49:08.7138871Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 588, in forward 2025-12-04T08:49:08.7138965Z layer_output = apply_chunking_to_forward( 2025-12-04T08:49:08.7139247Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:49:08.7139328Z return forward_fn(*input_tensors) 2025-12-04T08:49:08.7139636Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 597, in feed_forward_chunk 2025-12-04T08:49:08.7139782Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T08:49:08.7140052Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 525, in forward 2025-12-04T08:49:08.7140148Z hidden_states = self.dense(hidden_states) 2025-12-04T08:49:08.7140155Z 2025-12-04T08:49:08.7140264Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:49:08.7140489Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:49:08.7140559Z res = mod(**inputs) 2025-12-04T08:49:08.7140832Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T08:49:08.7140914Z outputs = self.bert( 2025-12-04T08:49:08.7141185Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T08:49:08.7141269Z encoder_outputs = self.encoder( 2025-12-04T08:49:08.7141539Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T08:49:08.7141615Z layer_outputs = layer_module( 2025-12-04T08:49:08.7141866Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:49:08.7141954Z return super().__call__(*args, **kwargs) 2025-12-04T08:49:08.7142222Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.7142306Z return func(*args, **kwargs) 2025-12-04T08:49:08.7142576Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T08:49:08.7142673Z self_attention_outputs = self.attention( 2025-12-04T08:49:08.7142935Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.7143010Z return func(*args, **kwargs) 2025-12-04T08:49:08.7143290Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 488, in forward 2025-12-04T08:49:08.7143366Z self_outputs = self.self( 2025-12-04T08:49:08.7143638Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.7143747Z return func(*args, **kwargs) 2025-12-04T08:49:08.7144018Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 363, in forward 2025-12-04T08:49:08.7144256Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-12-04T08:49:08.7144260Z 2025-12-04T08:49:08.7144371Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:49:08.7144593Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:49:08.7144668Z res = mod(**inputs) 2025-12-04T08:49:08.7144939Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T08:49:08.7145017Z outputs = self.bert( 2025-12-04T08:49:08.7145323Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T08:49:08.7145405Z encoder_outputs = self.encoder( 2025-12-04T08:49:08.7145681Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T08:49:08.7145758Z layer_outputs = layer_module( 2025-12-04T08:49:08.7146062Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:49:08.7146154Z return super().__call__(*args, **kwargs) 2025-12-04T08:49:08.7146402Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.7146478Z return func(*args, **kwargs) 2025-12-04T08:49:08.7146731Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T08:49:08.7146817Z self_attention_outputs = self.attention( 2025-12-04T08:49:08.7147073Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.7147142Z return func(*args, **kwargs) 2025-12-04T08:49:08.7147400Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 488, in forward 2025-12-04T08:49:08.7147473Z self_outputs = self.self( 2025-12-04T08:49:08.7147722Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.7147799Z return func(*args, **kwargs) 2025-12-04T08:49:08.7148050Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 387, in forward 2025-12-04T08:49:08.7148123Z self.key(current_states) 2025-12-04T08:49:08.7148126Z 2025-12-04T08:49:08.7148240Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:49:08.7148447Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:49:08.7148524Z res = mod(**inputs) 2025-12-04T08:49:08.7148782Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T08:49:08.7148851Z outputs = self.bert( 2025-12-04T08:49:08.7149116Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T08:49:08.7149191Z encoder_outputs = self.encoder( 2025-12-04T08:49:08.7149452Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T08:49:08.7149525Z layer_outputs = layer_module( 2025-12-04T08:49:08.7149753Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:49:08.7149838Z return super().__call__(*args, **kwargs) 2025-12-04T08:49:08.7150118Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.7150189Z return func(*args, **kwargs) 2025-12-04T08:49:08.7150452Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T08:49:08.7150536Z self_attention_outputs = self.attention( 2025-12-04T08:49:08.7150792Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.7150862Z return func(*args, **kwargs) 2025-12-04T08:49:08.7151118Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 488, in forward 2025-12-04T08:49:08.7151200Z self_outputs = self.self( 2025-12-04T08:49:08.7151449Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.7151556Z return func(*args, **kwargs) 2025-12-04T08:49:08.7151819Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 392, in forward 2025-12-04T08:49:08.7151893Z self.value(current_states) 2025-12-04T08:49:08.7151897Z 2025-12-04T08:49:08.7151988Z cudagraph partition due to non gpu ops 2025-12-04T08:49:08.7152095Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:49:08.7152302Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:49:08.7152374Z res = mod(**inputs) 2025-12-04T08:49:08.7152632Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T08:49:08.7152701Z outputs = self.bert( 2025-12-04T08:49:08.7152967Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T08:49:08.7153044Z encoder_outputs = self.encoder( 2025-12-04T08:49:08.7153304Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T08:49:08.7153376Z layer_outputs = layer_module( 2025-12-04T08:49:08.7153604Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:49:08.7153690Z return super().__call__(*args, **kwargs) 2025-12-04T08:49:08.7153947Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.7154027Z return func(*args, **kwargs) 2025-12-04T08:49:08.7154294Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T08:49:08.7154381Z self_attention_outputs = self.attention( 2025-12-04T08:49:08.7154648Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.7154723Z return func(*args, **kwargs) 2025-12-04T08:49:08.7154994Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 488, in forward 2025-12-04T08:49:08.7155077Z self_outputs = self.self( 2025-12-04T08:49:08.7155337Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.7155417Z return func(*args, **kwargs) 2025-12-04T08:49:08.7155685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 413, in forward 2025-12-04T08:49:08.7155828Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T08:49:08.7155831Z 2025-12-04T08:49:08.7155949Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:49:08.7156162Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:49:08.7156242Z res = mod(**inputs) 2025-12-04T08:49:08.7156547Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T08:49:08.7156620Z outputs = self.bert( 2025-12-04T08:49:08.7156902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T08:49:08.7156981Z encoder_outputs = self.encoder( 2025-12-04T08:49:08.7157249Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T08:49:08.7157333Z layer_outputs = layer_module( 2025-12-04T08:49:08.7157574Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:49:08.7157667Z return super().__call__(*args, **kwargs) 2025-12-04T08:49:08.7157931Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.7158042Z return func(*args, **kwargs) 2025-12-04T08:49:08.7158397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T08:49:08.7158494Z self_attention_outputs = self.attention( 2025-12-04T08:49:08.7158764Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.7158848Z return func(*args, **kwargs) 2025-12-04T08:49:08.7159124Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 497, in forward 2025-12-04T08:49:08.7159279Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T08:49:08.7159559Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 436, in forward 2025-12-04T08:49:08.7159652Z hidden_states = self.dense(hidden_states) 2025-12-04T08:49:08.7159660Z 2025-12-04T08:49:08.7159784Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:49:08.7159998Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:49:08.7160076Z res = mod(**inputs) 2025-12-04T08:49:08.7160350Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T08:49:08.7160422Z outputs = self.bert( 2025-12-04T08:49:08.7160699Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T08:49:08.7160778Z encoder_outputs = self.encoder( 2025-12-04T08:49:08.7161052Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T08:49:08.7161136Z layer_outputs = layer_module( 2025-12-04T08:49:08.7161376Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:49:08.7161473Z return super().__call__(*args, **kwargs) 2025-12-04T08:49:08.7161735Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.7161808Z return func(*args, **kwargs) 2025-12-04T08:49:08.7162084Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 588, in forward 2025-12-04T08:49:08.7162172Z layer_output = apply_chunking_to_forward( 2025-12-04T08:49:08.7162457Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:49:08.7162545Z return forward_fn(*input_tensors) 2025-12-04T08:49:08.7162847Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 596, in feed_forward_chunk 2025-12-04T08:49:08.7162986Z intermediate_output = self.intermediate(attention_output) 2025-12-04T08:49:08.7163300Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 512, in forward 2025-12-04T08:49:08.7163392Z hidden_states = self.dense(hidden_states) 2025-12-04T08:49:08.7163396Z 2025-12-04T08:49:08.7163514Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:49:08.7163736Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:49:08.7163811Z res = mod(**inputs) 2025-12-04T08:49:08.7164080Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T08:49:08.7164150Z outputs = self.bert( 2025-12-04T08:49:08.7164427Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T08:49:08.7164505Z encoder_outputs = self.encoder( 2025-12-04T08:49:08.7164810Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T08:49:08.7164894Z layer_outputs = layer_module( 2025-12-04T08:49:08.7165135Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:49:08.7165226Z return super().__call__(*args, **kwargs) 2025-12-04T08:49:08.7165490Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.7165565Z return func(*args, **kwargs) 2025-12-04T08:49:08.7165840Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 588, in forward 2025-12-04T08:49:08.7165930Z layer_output = apply_chunking_to_forward( 2025-12-04T08:49:08.7166220Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:49:08.7166305Z return forward_fn(*input_tensors) 2025-12-04T08:49:08.7166607Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 596, in feed_forward_chunk 2025-12-04T08:49:08.7166745Z intermediate_output = self.intermediate(attention_output) 2025-12-04T08:49:08.7167015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 513, in forward 2025-12-04T08:49:08.7167137Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:49:08.7167382Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T08:49:08.7167459Z return self.act(input) 2025-12-04T08:49:08.7167463Z 2025-12-04T08:49:08.7167582Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:49:08.7167800Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:49:08.7167873Z res = mod(**inputs) 2025-12-04T08:49:08.7168155Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T08:49:08.7168227Z outputs = self.bert( 2025-12-04T08:49:08.7168504Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T08:49:08.7168583Z encoder_outputs = self.encoder( 2025-12-04T08:49:08.7168849Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T08:49:08.7168932Z layer_outputs = layer_module( 2025-12-04T08:49:08.7169176Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:49:08.7169260Z return super().__call__(*args, **kwargs) 2025-12-04T08:49:08.7169531Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.7169609Z return func(*args, **kwargs) 2025-12-04T08:49:08.7169909Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 588, in forward 2025-12-04T08:49:08.7170001Z layer_output = apply_chunking_to_forward( 2025-12-04T08:49:08.7170288Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:49:08.7170375Z return forward_fn(*input_tensors) 2025-12-04T08:49:08.7170682Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 597, in feed_forward_chunk 2025-12-04T08:49:08.7170825Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T08:49:08.7171106Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 525, in forward 2025-12-04T08:49:08.7171225Z hidden_states = self.dense(hidden_states) 2025-12-04T08:49:08.7171229Z 2025-12-04T08:49:08.7171351Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:49:08.7171567Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:49:08.7171636Z res = mod(**inputs) 2025-12-04T08:49:08.7171919Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T08:49:08.7171989Z outputs = self.bert( 2025-12-04T08:49:08.7172267Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T08:49:08.7172345Z encoder_outputs = self.encoder( 2025-12-04T08:49:08.7172614Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T08:49:08.7172699Z layer_outputs = layer_module( 2025-12-04T08:49:08.7172940Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:49:08.7173030Z return super().__call__(*args, **kwargs) 2025-12-04T08:49:08.7173301Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.7173375Z return func(*args, **kwargs) 2025-12-04T08:49:08.7173653Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T08:49:08.7173743Z self_attention_outputs = self.attention( 2025-12-04T08:49:08.7174006Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.7174090Z return func(*args, **kwargs) 2025-12-04T08:49:08.7174356Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 488, in forward 2025-12-04T08:49:08.7174443Z self_outputs = self.self( 2025-12-04T08:49:08.7174706Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.7174780Z return func(*args, **kwargs) 2025-12-04T08:49:08.7175054Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 363, in forward 2025-12-04T08:49:08.7175282Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-12-04T08:49:08.7175287Z 2025-12-04T08:49:08.7175399Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:49:08.7175625Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:49:08.7175693Z res = mod(**inputs) 2025-12-04T08:49:08.7175970Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T08:49:08.7176041Z outputs = self.bert( 2025-12-04T08:49:08.7176353Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T08:49:08.7176440Z encoder_outputs = self.encoder( 2025-12-04T08:49:08.7176709Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T08:49:08.7176785Z layer_outputs = layer_module( 2025-12-04T08:49:08.7177033Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:49:08.7177117Z return super().__call__(*args, **kwargs) 2025-12-04T08:49:08.7177384Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.7177457Z return func(*args, **kwargs) 2025-12-04T08:49:08.7177724Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T08:49:08.7177851Z self_attention_outputs = self.attention( 2025-12-04T08:49:08.7178116Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.7178196Z return func(*args, **kwargs) 2025-12-04T08:49:08.7178465Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 488, in forward 2025-12-04T08:49:08.7178541Z self_outputs = self.self( 2025-12-04T08:49:08.7178808Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.7178881Z return func(*args, **kwargs) 2025-12-04T08:49:08.7179147Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 387, in forward 2025-12-04T08:49:08.7179231Z self.key(current_states) 2025-12-04T08:49:08.7179234Z 2025-12-04T08:49:08.7179344Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:49:08.7179572Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:49:08.7179642Z res = mod(**inputs) 2025-12-04T08:49:08.7179913Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T08:49:08.7179993Z outputs = self.bert( 2025-12-04T08:49:08.7180262Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T08:49:08.7180340Z encoder_outputs = self.encoder( 2025-12-04T08:49:08.7180615Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T08:49:08.7180690Z layer_outputs = layer_module( 2025-12-04T08:49:08.7180936Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:49:08.7181023Z return super().__call__(*args, **kwargs) 2025-12-04T08:49:08.7181290Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.7181372Z return func(*args, **kwargs) 2025-12-04T08:49:08.7181640Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T08:49:08.7181736Z self_attention_outputs = self.attention( 2025-12-04T08:49:08.7182002Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.7182075Z return func(*args, **kwargs) 2025-12-04T08:49:08.7182355Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 488, in forward 2025-12-04T08:49:08.7182430Z self_outputs = self.self( 2025-12-04T08:49:08.7182693Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.7182777Z return func(*args, **kwargs) 2025-12-04T08:49:08.7183078Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 392, in forward 2025-12-04T08:49:08.7183166Z self.value(current_states) 2025-12-04T08:49:08.7183170Z 2025-12-04T08:49:08.7183260Z cudagraph partition due to non gpu ops 2025-12-04T08:49:08.7183372Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:49:08.7183598Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:49:08.7183666Z res = mod(**inputs) 2025-12-04T08:49:08.7183941Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T08:49:08.7184021Z outputs = self.bert( 2025-12-04T08:49:08.7184294Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T08:49:08.7184413Z encoder_outputs = self.encoder( 2025-12-04T08:49:08.7184686Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T08:49:08.7184762Z layer_outputs = layer_module( 2025-12-04T08:49:08.7185011Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:49:08.7185096Z return super().__call__(*args, **kwargs) 2025-12-04T08:49:08.7185365Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.7185438Z return func(*args, **kwargs) 2025-12-04T08:49:08.7185708Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T08:49:08.7185801Z self_attention_outputs = self.attention( 2025-12-04T08:49:08.7186076Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.7186153Z return func(*args, **kwargs) 2025-12-04T08:49:08.7186430Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 488, in forward 2025-12-04T08:49:08.7186506Z self_outputs = self.self( 2025-12-04T08:49:08.7186780Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.7186854Z return func(*args, **kwargs) 2025-12-04T08:49:08.7187124Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 413, in forward 2025-12-04T08:49:08.7187273Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T08:49:08.7187277Z 2025-12-04T08:49:08.7187388Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:49:08.7187611Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:49:08.7187683Z res = mod(**inputs) 2025-12-04T08:49:08.7187959Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T08:49:08.7188036Z outputs = self.bert( 2025-12-04T08:49:08.7188304Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T08:49:08.7188383Z encoder_outputs = self.encoder( 2025-12-04T08:49:08.7188655Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T08:49:08.7188730Z layer_outputs = layer_module( 2025-12-04T08:49:08.7188979Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:49:08.7189062Z return super().__call__(*args, **kwargs) 2025-12-04T08:49:08.7189326Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.7189451Z return func(*args, **kwargs) 2025-12-04T08:49:08.7189721Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T08:49:08.7189811Z self_attention_outputs = self.attention( 2025-12-04T08:49:08.7190080Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.7190153Z return func(*args, **kwargs) 2025-12-04T08:49:08.7190431Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 497, in forward 2025-12-04T08:49:08.7190572Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T08:49:08.7190842Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 436, in forward 2025-12-04T08:49:08.7190973Z hidden_states = self.dense(hidden_states) 2025-12-04T08:49:08.7190977Z 2025-12-04T08:49:08.7191093Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:49:08.7191317Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:49:08.7191390Z res = mod(**inputs) 2025-12-04T08:49:08.7191666Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T08:49:08.7191747Z outputs = self.bert( 2025-12-04T08:49:08.7192019Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T08:49:08.7192100Z encoder_outputs = self.encoder( 2025-12-04T08:49:08.7192378Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T08:49:08.7192456Z layer_outputs = layer_module( 2025-12-04T08:49:08.7192710Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:49:08.7192799Z return super().__call__(*args, **kwargs) 2025-12-04T08:49:08.7193071Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.7193157Z return func(*args, **kwargs) 2025-12-04T08:49:08.7193437Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 588, in forward 2025-12-04T08:49:08.7193533Z layer_output = apply_chunking_to_forward( 2025-12-04T08:49:08.7193837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:49:08.7193925Z return forward_fn(*input_tensors) 2025-12-04T08:49:08.7194244Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 596, in feed_forward_chunk 2025-12-04T08:49:08.7194382Z intermediate_output = self.intermediate(attention_output) 2025-12-04T08:49:08.7194666Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 512, in forward 2025-12-04T08:49:08.7194770Z hidden_states = self.dense(hidden_states) 2025-12-04T08:49:08.7194774Z 2025-12-04T08:49:08.7194890Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:49:08.7195121Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:49:08.7195195Z res = mod(**inputs) 2025-12-04T08:49:08.7195477Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T08:49:08.7195561Z outputs = self.bert( 2025-12-04T08:49:08.7195843Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T08:49:08.7195930Z encoder_outputs = self.encoder( 2025-12-04T08:49:08.7196248Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T08:49:08.7196328Z layer_outputs = layer_module( 2025-12-04T08:49:08.7196585Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:49:08.7196670Z return super().__call__(*args, **kwargs) 2025-12-04T08:49:08.7196941Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.7197025Z return func(*args, **kwargs) 2025-12-04T08:49:08.7197302Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 588, in forward 2025-12-04T08:49:08.7197400Z layer_output = apply_chunking_to_forward( 2025-12-04T08:49:08.7197689Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:49:08.7197807Z return forward_fn(*input_tensors) 2025-12-04T08:49:08.7198125Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 596, in feed_forward_chunk 2025-12-04T08:49:08.7198533Z intermediate_output = self.intermediate(attention_output) 2025-12-04T08:49:08.7198814Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 513, in forward 2025-12-04T08:49:08.7198947Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:49:08.7199197Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T08:49:08.7199285Z return self.act(input) 2025-12-04T08:49:08.7199289Z 2025-12-04T08:49:08.7199402Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:49:08.7199635Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:49:08.7199744Z res = mod(**inputs) 2025-12-04T08:49:08.7200025Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T08:49:08.7200108Z outputs = self.bert( 2025-12-04T08:49:08.7200390Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T08:49:08.7200473Z encoder_outputs = self.encoder( 2025-12-04T08:49:08.7200767Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T08:49:08.7200848Z layer_outputs = layer_module( 2025-12-04T08:49:08.7201097Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:49:08.7201192Z return super().__call__(*args, **kwargs) 2025-12-04T08:49:08.7201472Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.7201563Z return func(*args, **kwargs) 2025-12-04T08:49:08.7201844Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 588, in forward 2025-12-04T08:49:08.7201938Z layer_output = apply_chunking_to_forward( 2025-12-04T08:49:08.7202239Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:49:08.7202324Z return forward_fn(*input_tensors) 2025-12-04T08:49:08.7202636Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 597, in feed_forward_chunk 2025-12-04T08:49:08.7202794Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T08:49:08.7203093Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 525, in forward 2025-12-04T08:49:08.7203195Z hidden_states = self.dense(hidden_states) 2025-12-04T08:49:08.7203198Z 2025-12-04T08:49:08.7203351Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:49:08.7203574Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:49:08.7203652Z res = mod(**inputs) 2025-12-04T08:49:08.7203934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T08:49:08.7204015Z outputs = self.bert( 2025-12-04T08:49:08.7204297Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T08:49:08.7204378Z encoder_outputs = self.encoder( 2025-12-04T08:49:08.7204668Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T08:49:08.7204746Z layer_outputs = layer_module( 2025-12-04T08:49:08.7205029Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:49:08.7205123Z return super().__call__(*args, **kwargs) 2025-12-04T08:49:08.7205405Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.7205489Z return func(*args, **kwargs) 2025-12-04T08:49:08.7205767Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T08:49:08.7205859Z self_attention_outputs = self.attention( 2025-12-04T08:49:08.7206139Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.7206225Z return func(*args, **kwargs) 2025-12-04T08:49:08.7206495Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 488, in forward 2025-12-04T08:49:08.7206582Z self_outputs = self.self( 2025-12-04T08:49:08.7206850Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.7206931Z return func(*args, **kwargs) 2025-12-04T08:49:08.7207200Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 363, in forward 2025-12-04T08:49:08.7207429Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-12-04T08:49:08.7207434Z 2025-12-04T08:49:08.7207562Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:49:08.7207766Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:49:08.7207838Z res = mod(**inputs) 2025-12-04T08:49:08.7208096Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T08:49:08.7208171Z outputs = self.bert( 2025-12-04T08:49:08.7208451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T08:49:08.7208528Z encoder_outputs = self.encoder( 2025-12-04T08:49:08.7208797Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T08:49:08.7208880Z layer_outputs = layer_module( 2025-12-04T08:49:08.7209122Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:49:08.7209213Z return super().__call__(*args, **kwargs) 2025-12-04T08:49:08.7209473Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.7209546Z return func(*args, **kwargs) 2025-12-04T08:49:08.7209820Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T08:49:08.7209912Z self_attention_outputs = self.attention( 2025-12-04T08:49:08.7210210Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.7210285Z return func(*args, **kwargs) 2025-12-04T08:49:08.7210557Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 488, in forward 2025-12-04T08:49:08.7210638Z self_outputs = self.self( 2025-12-04T08:49:08.7210904Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.7210978Z return func(*args, **kwargs) 2025-12-04T08:49:08.7211276Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 387, in forward 2025-12-04T08:49:08.7211350Z self.key(current_states) 2025-12-04T08:49:08.7211353Z 2025-12-04T08:49:08.7211505Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:49:08.7211712Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:49:08.7211779Z res = mod(**inputs) 2025-12-04T08:49:08.7212042Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T08:49:08.7212113Z outputs = self.bert( 2025-12-04T08:49:08.7212368Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T08:49:08.7212450Z encoder_outputs = self.encoder( 2025-12-04T08:49:08.7212704Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T08:49:08.7212784Z layer_outputs = layer_module( 2025-12-04T08:49:08.7213011Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:49:08.7213094Z return super().__call__(*args, **kwargs) 2025-12-04T08:49:08.7213353Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.7213422Z return func(*args, **kwargs) 2025-12-04T08:49:08.7213693Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T08:49:08.7213784Z self_attention_outputs = self.attention( 2025-12-04T08:49:08.7214047Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.7214128Z return func(*args, **kwargs) 2025-12-04T08:49:08.7214405Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 488, in forward 2025-12-04T08:49:08.7214479Z self_outputs = self.self( 2025-12-04T08:49:08.7214749Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.7214829Z return func(*args, **kwargs) 2025-12-04T08:49:08.7215105Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 392, in forward 2025-12-04T08:49:08.7215183Z self.value(current_states) 2025-12-04T08:49:08.7215187Z 2025-12-04T08:49:08.7215275Z cudagraph partition due to non gpu ops 2025-12-04T08:49:08.7215392Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:49:08.7215608Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:49:08.7215677Z res = mod(**inputs) 2025-12-04T08:49:08.7215955Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T08:49:08.7216025Z outputs = self.bert( 2025-12-04T08:49:08.7216308Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T08:49:08.7216392Z encoder_outputs = self.encoder( 2025-12-04T08:49:08.7216695Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T08:49:08.7216781Z layer_outputs = layer_module( 2025-12-04T08:49:08.7217022Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:49:08.7217113Z return super().__call__(*args, **kwargs) 2025-12-04T08:49:08.7217373Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.7217445Z return func(*args, **kwargs) 2025-12-04T08:49:08.7217722Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T08:49:08.7217810Z self_attention_outputs = self.attention( 2025-12-04T08:49:08.7218107Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.7218189Z return func(*args, **kwargs) 2025-12-04T08:49:08.7218456Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 488, in forward 2025-12-04T08:49:08.7218540Z self_outputs = self.self( 2025-12-04T08:49:08.7218801Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.7218875Z return func(*args, **kwargs) 2025-12-04T08:49:08.7219148Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 413, in forward 2025-12-04T08:49:08.7219291Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T08:49:08.7219295Z 2025-12-04T08:49:08.7219411Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:49:08.7219629Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:49:08.7219701Z res = mod(**inputs) 2025-12-04T08:49:08.7219981Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T08:49:08.7220051Z outputs = self.bert( 2025-12-04T08:49:08.7220319Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T08:49:08.7220405Z encoder_outputs = self.encoder( 2025-12-04T08:49:08.7220673Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T08:49:08.7220906Z layer_outputs = layer_module( 2025-12-04T08:49:08.7221154Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:49:08.7221239Z return super().__call__(*args, **kwargs) 2025-12-04T08:49:08.7221518Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.7221592Z return func(*args, **kwargs) 2025-12-04T08:49:08.7221862Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T08:49:08.7221961Z self_attention_outputs = self.attention( 2025-12-04T08:49:08.7222222Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.7222305Z return func(*args, **kwargs) 2025-12-04T08:49:08.7222574Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 497, in forward 2025-12-04T08:49:08.7222714Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T08:49:08.7222991Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 436, in forward 2025-12-04T08:49:08.7223088Z hidden_states = self.dense(hidden_states) 2025-12-04T08:49:08.7223161Z 2025-12-04T08:49:08.7223282Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:49:08.7223497Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:49:08.7223565Z res = mod(**inputs) 2025-12-04T08:49:08.7223848Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T08:49:08.7223922Z outputs = self.bert( 2025-12-04T08:49:08.7224199Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T08:49:08.7224289Z encoder_outputs = self.encoder( 2025-12-04T08:49:08.7224563Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T08:49:08.7224707Z layer_outputs = layer_module( 2025-12-04T08:49:08.7224959Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:49:08.7225047Z return super().__call__(*args, **kwargs) 2025-12-04T08:49:08.7225330Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.7225404Z return func(*args, **kwargs) 2025-12-04T08:49:08.7225692Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 588, in forward 2025-12-04T08:49:08.7225793Z layer_output = apply_chunking_to_forward( 2025-12-04T08:49:08.7226084Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:49:08.7226178Z return forward_fn(*input_tensors) 2025-12-04T08:49:08.7226486Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 596, in feed_forward_chunk 2025-12-04T08:49:08.7226623Z intermediate_output = self.intermediate(attention_output) 2025-12-04T08:49:08.7226907Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 512, in forward 2025-12-04T08:49:08.7226999Z hidden_states = self.dense(hidden_states) 2025-12-04T08:49:08.7227003Z 2025-12-04T08:49:08.7227121Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:49:08.7227339Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:49:08.7227407Z res = mod(**inputs) 2025-12-04T08:49:08.7227694Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T08:49:08.7227767Z outputs = self.bert( 2025-12-04T08:49:08.7228046Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T08:49:08.7228134Z encoder_outputs = self.encoder( 2025-12-04T08:49:08.7228423Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T08:49:08.7228512Z layer_outputs = layer_module( 2025-12-04T08:49:08.7228760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:49:08.7228843Z return super().__call__(*args, **kwargs) 2025-12-04T08:49:08.7229123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.7229197Z return func(*args, **kwargs) 2025-12-04T08:49:08.7229485Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 588, in forward 2025-12-04T08:49:08.7229574Z layer_output = apply_chunking_to_forward( 2025-12-04T08:49:08.7229866Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:49:08.7229996Z return forward_fn(*input_tensors) 2025-12-04T08:49:08.7230313Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 596, in feed_forward_chunk 2025-12-04T08:49:08.7230446Z intermediate_output = self.intermediate(attention_output) 2025-12-04T08:49:08.7230760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 513, in forward 2025-12-04T08:49:08.7230886Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:49:08.7231140Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T08:49:08.7231219Z return self.act(input) 2025-12-04T08:49:08.7231223Z 2025-12-04T08:49:08.7231337Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:49:08.7231573Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:49:08.7231683Z res = mod(**inputs) 2025-12-04T08:49:08.7231963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T08:49:08.7232044Z outputs = self.bert( 2025-12-04T08:49:08.7232317Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T08:49:08.7232403Z encoder_outputs = self.encoder( 2025-12-04T08:49:08.7232677Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T08:49:08.7232754Z layer_outputs = layer_module( 2025-12-04T08:49:08.7233006Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:49:08.7233091Z return super().__call__(*args, **kwargs) 2025-12-04T08:49:08.7233372Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.7233451Z return func(*args, **kwargs) 2025-12-04T08:49:08.7233725Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 588, in forward 2025-12-04T08:49:08.7233822Z layer_output = apply_chunking_to_forward( 2025-12-04T08:49:08.7234107Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:49:08.7234189Z return forward_fn(*input_tensors) 2025-12-04T08:49:08.7234497Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 597, in feed_forward_chunk 2025-12-04T08:49:08.7234642Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T08:49:08.7234919Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 525, in forward 2025-12-04T08:49:08.7235011Z hidden_states = self.dense(hidden_states) 2025-12-04T08:49:08.7235018Z 2025-12-04T08:49:08.7235129Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:49:08.7235352Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:49:08.7235421Z res = mod(**inputs) 2025-12-04T08:49:08.7235701Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T08:49:08.7235772Z outputs = self.bert( 2025-12-04T08:49:08.7236046Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T08:49:08.7236132Z encoder_outputs = self.encoder( 2025-12-04T08:49:08.7236403Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T08:49:08.7236483Z layer_outputs = layer_module( 2025-12-04T08:49:08.7236776Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:49:08.7236862Z return super().__call__(*args, **kwargs) 2025-12-04T08:49:08.7237137Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.7237214Z return func(*args, **kwargs) 2025-12-04T08:49:08.7237482Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T08:49:08.7237579Z self_attention_outputs = self.attention( 2025-12-04T08:49:08.7237843Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.7237917Z return func(*args, **kwargs) 2025-12-04T08:49:08.7238251Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 488, in forward 2025-12-04T08:49:08.7238387Z self_outputs = self.self( 2025-12-04T08:49:08.7238668Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.7238744Z return func(*args, **kwargs) 2025-12-04T08:49:08.7239022Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 363, in forward 2025-12-04T08:49:08.7239271Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-12-04T08:49:08.7239275Z 2025-12-04T08:49:08.7239390Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:49:08.7239629Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:49:08.7239698Z res = mod(**inputs) 2025-12-04T08:49:08.7239973Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T08:49:08.7240057Z outputs = self.bert( 2025-12-04T08:49:08.7240332Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T08:49:08.7240411Z encoder_outputs = self.encoder( 2025-12-04T08:49:08.7240696Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T08:49:08.7240774Z layer_outputs = layer_module( 2025-12-04T08:49:08.7241032Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:49:08.7241119Z return super().__call__(*args, **kwargs) 2025-12-04T08:49:08.7241389Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.7241474Z return func(*args, **kwargs) 2025-12-04T08:49:08.7241750Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T08:49:08.7241857Z self_attention_outputs = self.attention( 2025-12-04T08:49:08.7242128Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.7242204Z return func(*args, **kwargs) 2025-12-04T08:49:08.7242486Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 488, in forward 2025-12-04T08:49:08.7242564Z self_outputs = self.self( 2025-12-04T08:49:08.7242834Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.7242914Z return func(*args, **kwargs) 2025-12-04T08:49:08.7243189Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 387, in forward 2025-12-04T08:49:08.7243274Z self.key(current_states) 2025-12-04T08:49:08.7243281Z 2025-12-04T08:49:08.7243396Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:49:08.7243657Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:49:08.7243737Z res = mod(**inputs) 2025-12-04T08:49:08.7244022Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T08:49:08.7244095Z outputs = self.bert( 2025-12-04T08:49:08.7244384Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T08:49:08.7244464Z encoder_outputs = self.encoder( 2025-12-04T08:49:08.7244752Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T08:49:08.7244830Z layer_outputs = layer_module( 2025-12-04T08:49:08.7245081Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:49:08.7245214Z return super().__call__(*args, **kwargs) 2025-12-04T08:49:08.7245484Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.7245567Z return func(*args, **kwargs) 2025-12-04T08:49:08.7245853Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T08:49:08.7245945Z self_attention_outputs = self.attention( 2025-12-04T08:49:08.7246220Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.7246296Z return func(*args, **kwargs) 2025-12-04T08:49:08.7246569Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 488, in forward 2025-12-04T08:49:08.7246654Z self_outputs = self.self( 2025-12-04T08:49:08.7246928Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.7247013Z return func(*args, **kwargs) 2025-12-04T08:49:08.7247288Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 392, in forward 2025-12-04T08:49:08.7247369Z self.value(current_states) 2025-12-04T08:49:08.7247373Z 2025-12-04T08:49:08.7247473Z cudagraph partition due to non gpu ops 2025-12-04T08:49:08.7247587Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:49:08.7247807Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:49:08.7247886Z res = mod(**inputs) 2025-12-04T08:49:08.7248167Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T08:49:08.7248247Z outputs = self.bert( 2025-12-04T08:49:08.7248524Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T08:49:08.7248615Z encoder_outputs = self.encoder( 2025-12-04T08:49:08.7248907Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T08:49:08.7248983Z layer_outputs = layer_module( 2025-12-04T08:49:08.7249232Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:49:08.7249317Z return super().__call__(*args, **kwargs) 2025-12-04T08:49:08.7249579Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.7249661Z return func(*args, **kwargs) 2025-12-04T08:49:08.7249928Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T08:49:08.7250020Z self_attention_outputs = self.attention( 2025-12-04T08:49:08.7250323Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.7250398Z return func(*args, **kwargs) 2025-12-04T08:49:08.7250676Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 488, in forward 2025-12-04T08:49:08.7250752Z self_outputs = self.self( 2025-12-04T08:49:08.7251021Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.7251102Z return func(*args, **kwargs) 2025-12-04T08:49:08.7251385Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 413, in forward 2025-12-04T08:49:08.7251531Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T08:49:08.7251542Z 2025-12-04T08:49:08.7251653Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:49:08.7251909Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:49:08.7251988Z res = mod(**inputs) 2025-12-04T08:49:08.7252273Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T08:49:08.7252339Z outputs = self.bert( 2025-12-04T08:49:08.7252606Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T08:49:08.7252679Z encoder_outputs = self.encoder( 2025-12-04T08:49:08.7252938Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T08:49:08.7253011Z layer_outputs = layer_module( 2025-12-04T08:49:08.7253235Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:49:08.7253323Z return super().__call__(*args, **kwargs) 2025-12-04T08:49:08.7253571Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.7253641Z return func(*args, **kwargs) 2025-12-04T08:49:08.7253902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T08:49:08.7253986Z self_attention_outputs = self.attention( 2025-12-04T08:49:08.7254240Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.7254309Z return func(*args, **kwargs) 2025-12-04T08:49:08.7254560Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 497, in forward 2025-12-04T08:49:08.7254700Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T08:49:08.7254951Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 436, in forward 2025-12-04T08:49:08.7255050Z hidden_states = self.dense(hidden_states) 2025-12-04T08:49:08.7255053Z 2025-12-04T08:49:08.7255158Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:49:08.7255361Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:49:08.7255434Z res = mod(**inputs) 2025-12-04T08:49:08.7255688Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T08:49:08.7255757Z outputs = self.bert( 2025-12-04T08:49:08.7256022Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T08:49:08.7256096Z encoder_outputs = self.encoder( 2025-12-04T08:49:08.7256355Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T08:49:08.7256430Z layer_outputs = layer_module( 2025-12-04T08:49:08.7256687Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:49:08.7256775Z return super().__call__(*args, **kwargs) 2025-12-04T08:49:08.7257023Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.7257093Z return func(*args, **kwargs) 2025-12-04T08:49:08.7257360Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 588, in forward 2025-12-04T08:49:08.7257446Z layer_output = apply_chunking_to_forward( 2025-12-04T08:49:08.7257724Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:49:08.7257804Z return forward_fn(*input_tensors) 2025-12-04T08:49:08.7258087Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 596, in feed_forward_chunk 2025-12-04T08:49:08.7258255Z intermediate_output = self.intermediate(attention_output) 2025-12-04T08:49:08.7258509Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 512, in forward 2025-12-04T08:49:08.7258602Z hidden_states = self.dense(hidden_states) 2025-12-04T08:49:08.7258605Z 2025-12-04T08:49:08.7258708Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:49:08.7258912Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:49:08.7258983Z res = mod(**inputs) 2025-12-04T08:49:08.7259246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T08:49:08.7259313Z outputs = self.bert( 2025-12-04T08:49:08.7259579Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T08:49:08.7259656Z encoder_outputs = self.encoder( 2025-12-04T08:49:08.7259922Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T08:49:08.7259995Z layer_outputs = layer_module( 2025-12-04T08:49:08.7260223Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:49:08.7260308Z return super().__call__(*args, **kwargs) 2025-12-04T08:49:08.7260559Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.7260636Z return func(*args, **kwargs) 2025-12-04T08:49:08.7260889Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 588, in forward 2025-12-04T08:49:08.7260974Z layer_output = apply_chunking_to_forward( 2025-12-04T08:49:08.7261259Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:49:08.7261338Z return forward_fn(*input_tensors) 2025-12-04T08:49:08.7261623Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 596, in feed_forward_chunk 2025-12-04T08:49:08.7261753Z intermediate_output = self.intermediate(attention_output) 2025-12-04T08:49:08.7262008Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 513, in forward 2025-12-04T08:49:08.7262131Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:49:08.7262356Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T08:49:08.7262428Z return self.act(input) 2025-12-04T08:49:08.7262432Z 2025-12-04T08:49:08.7262547Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:49:08.7262752Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:49:08.7262861Z res = mod(**inputs) 2025-12-04T08:49:08.7263129Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T08:49:08.7263199Z outputs = self.bert( 2025-12-04T08:49:08.7263463Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T08:49:08.7263538Z encoder_outputs = self.encoder( 2025-12-04T08:49:08.7263793Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T08:49:08.7263874Z layer_outputs = layer_module( 2025-12-04T08:49:08.7264100Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:49:08.7264187Z return super().__call__(*args, **kwargs) 2025-12-04T08:49:08.7264473Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.7264545Z return func(*args, **kwargs) 2025-12-04T08:49:08.7264814Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 588, in forward 2025-12-04T08:49:08.7264900Z layer_output = apply_chunking_to_forward( 2025-12-04T08:49:08.7265172Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:49:08.7265256Z return forward_fn(*input_tensors) 2025-12-04T08:49:08.7265545Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 597, in feed_forward_chunk 2025-12-04T08:49:08.7265690Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T08:49:08.7265949Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 525, in forward 2025-12-04T08:49:08.7266040Z hidden_states = self.dense(hidden_states) 2025-12-04T08:49:08.7266043Z 2025-12-04T08:49:08.7266156Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:49:08.7266365Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:49:08.7266438Z res = mod(**inputs) 2025-12-04T08:49:08.7266700Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T08:49:08.7266768Z outputs = self.bert( 2025-12-04T08:49:08.7267036Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T08:49:08.7267110Z encoder_outputs = self.encoder( 2025-12-04T08:49:08.7267365Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T08:49:08.7267451Z layer_outputs = layer_module( 2025-12-04T08:49:08.7267684Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:49:08.7267771Z return super().__call__(*args, **kwargs) 2025-12-04T08:49:08.7268024Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.7268095Z return func(*args, **kwargs) 2025-12-04T08:49:08.7268361Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T08:49:08.7268446Z self_attention_outputs = self.attention( 2025-12-04T08:49:08.7268702Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.7268779Z return func(*args, **kwargs) 2025-12-04T08:49:08.7269038Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 488, in forward 2025-12-04T08:49:08.7269149Z self_outputs = self.self( 2025-12-04T08:49:08.7269408Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.7269477Z return func(*args, **kwargs) 2025-12-04T08:49:08.7269746Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 363, in forward 2025-12-04T08:49:08.7269963Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-12-04T08:49:08.7269967Z 2025-12-04T08:49:08.7270078Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:49:08.7270294Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:49:08.7270362Z res = mod(**inputs) 2025-12-04T08:49:08.7270648Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T08:49:08.7270749Z outputs = self.bert( 2025-12-04T08:49:08.7271027Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T08:49:08.7271112Z encoder_outputs = self.encoder( 2025-12-04T08:49:08.7271385Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T08:49:08.7271470Z layer_outputs = layer_module( 2025-12-04T08:49:08.7271715Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:49:08.7271793Z return super().__call__(*args, **kwargs) 2025-12-04T08:49:08.7272061Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.7272131Z return func(*args, **kwargs) 2025-12-04T08:49:08.7272406Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T08:49:08.7272492Z self_attention_outputs = self.attention( 2025-12-04T08:49:08.7272742Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.7272818Z return func(*args, **kwargs) 2025-12-04T08:49:08.7273074Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 488, in forward 2025-12-04T08:49:08.7273145Z self_outputs = self.self( 2025-12-04T08:49:08.7273412Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.7273485Z return func(*args, **kwargs) 2025-12-04T08:49:08.7273762Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 387, in forward 2025-12-04T08:49:08.7273842Z self.key(current_states) 2025-12-04T08:49:08.7273845Z 2025-12-04T08:49:08.7273958Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:49:08.7274181Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:49:08.7274248Z res = mod(**inputs) 2025-12-04T08:49:08.7274521Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T08:49:08.7274597Z outputs = self.bert( 2025-12-04T08:49:08.7274871Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T08:49:08.7274957Z encoder_outputs = self.encoder( 2025-12-04T08:49:08.7275229Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T08:49:08.7275305Z layer_outputs = layer_module( 2025-12-04T08:49:08.7275553Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:49:08.7275670Z return super().__call__(*args, **kwargs) 2025-12-04T08:49:08.7275940Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.7276013Z return func(*args, **kwargs) 2025-12-04T08:49:08.7276282Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T08:49:08.7276378Z self_attention_outputs = self.attention( 2025-12-04T08:49:08.7276642Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.7276713Z return func(*args, **kwargs) 2025-12-04T08:49:08.7276987Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 488, in forward 2025-12-04T08:49:08.7277097Z self_outputs = self.self( 2025-12-04T08:49:08.7277369Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.7277443Z return func(*args, **kwargs) 2025-12-04T08:49:08.7277712Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 392, in forward 2025-12-04T08:49:08.7277799Z self.value(current_states) 2025-12-04T08:49:08.7277803Z 2025-12-04T08:49:08.7277891Z cudagraph partition due to non gpu ops 2025-12-04T08:49:08.7278001Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:49:08.7278296Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:49:08.7278371Z res = mod(**inputs) 2025-12-04T08:49:08.7278655Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T08:49:08.7278727Z outputs = self.bert( 2025-12-04T08:49:08.7279023Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T08:49:08.7279118Z encoder_outputs = self.encoder( 2025-12-04T08:49:08.7279397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T08:49:08.7279484Z layer_outputs = layer_module( 2025-12-04T08:49:08.7279731Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:49:08.7279818Z return super().__call__(*args, **kwargs) 2025-12-04T08:49:08.7280108Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.7280185Z return func(*args, **kwargs) 2025-12-04T08:49:08.7280473Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T08:49:08.7280574Z self_attention_outputs = self.attention( 2025-12-04T08:49:08.7280851Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.7280931Z return func(*args, **kwargs) 2025-12-04T08:49:08.7281212Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 488, in forward 2025-12-04T08:49:08.7281288Z self_outputs = self.self( 2025-12-04T08:49:08.7281560Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.7281634Z return func(*args, **kwargs) 2025-12-04T08:49:08.7281904Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 413, in forward 2025-12-04T08:49:08.7282057Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T08:49:08.7282061Z 2025-12-04T08:49:08.7282175Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:49:08.7282433Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:49:08.7282502Z res = mod(**inputs) 2025-12-04T08:49:08.7282778Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T08:49:08.7282857Z outputs = self.bert( 2025-12-04T08:49:08.7283129Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T08:49:08.7283215Z encoder_outputs = self.encoder( 2025-12-04T08:49:08.7283482Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T08:49:08.7283558Z layer_outputs = layer_module( 2025-12-04T08:49:08.7283805Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:49:08.7283925Z return super().__call__(*args, **kwargs) 2025-12-04T08:49:08.7284193Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.7284274Z return func(*args, **kwargs) 2025-12-04T08:49:08.7284544Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 558, in forward 2025-12-04T08:49:08.7284640Z self_attention_outputs = self.attention( 2025-12-04T08:49:08.7284903Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.7284976Z return func(*args, **kwargs) 2025-12-04T08:49:08.7285280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 497, in forward 2025-12-04T08:49:08.7285419Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T08:49:08.7285692Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 436, in forward 2025-12-04T08:49:08.7285794Z hidden_states = self.dense(hidden_states) 2025-12-04T08:49:08.7285798Z 2025-12-04T08:49:08.7285908Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:49:08.7286132Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:49:08.7286201Z res = mod(**inputs) 2025-12-04T08:49:08.7286522Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T08:49:08.7286600Z outputs = self.bert( 2025-12-04T08:49:08.7286875Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T08:49:08.7286961Z encoder_outputs = self.encoder( 2025-12-04T08:49:08.7287232Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T08:49:08.7287312Z layer_outputs = layer_module( 2025-12-04T08:49:08.7287568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:49:08.7287652Z return super().__call__(*args, **kwargs) 2025-12-04T08:49:08.7287916Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.7287997Z return func(*args, **kwargs) 2025-12-04T08:49:08.7288268Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 588, in forward 2025-12-04T08:49:08.7288366Z layer_output = apply_chunking_to_forward( 2025-12-04T08:49:08.7288655Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:49:08.7288737Z return forward_fn(*input_tensors) 2025-12-04T08:49:08.7289089Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 596, in feed_forward_chunk 2025-12-04T08:49:08.7289220Z intermediate_output = self.intermediate(attention_output) 2025-12-04T08:49:08.7289496Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 512, in forward 2025-12-04T08:49:08.7289586Z hidden_states = self.dense(hidden_states) 2025-12-04T08:49:08.7289590Z 2025-12-04T08:49:08.7289698Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:49:08.7289919Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:49:08.7289988Z res = mod(**inputs) 2025-12-04T08:49:08.7290267Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T08:49:08.7290344Z outputs = self.bert( 2025-12-04T08:49:08.7290610Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T08:49:08.7290726Z encoder_outputs = self.encoder( 2025-12-04T08:49:08.7290981Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T08:49:08.7291053Z layer_outputs = layer_module( 2025-12-04T08:49:08.7291286Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:49:08.7291363Z return super().__call__(*args, **kwargs) 2025-12-04T08:49:08.7291612Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.7291690Z return func(*args, **kwargs) 2025-12-04T08:49:08.7291945Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 588, in forward 2025-12-04T08:49:08.7292036Z layer_output = apply_chunking_to_forward( 2025-12-04T08:49:08.7292314Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:49:08.7292390Z return forward_fn(*input_tensors) 2025-12-04T08:49:08.7292687Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 596, in feed_forward_chunk 2025-12-04T08:49:08.7292808Z intermediate_output = self.intermediate(attention_output) 2025-12-04T08:49:08.7293079Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 513, in forward 2025-12-04T08:49:08.7293195Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:49:08.7293418Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T08:49:08.7293499Z return self.act(input) 2025-12-04T08:49:08.7293503Z 2025-12-04T08:49:08.7293605Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:49:08.7293815Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:49:08.7293887Z res = mod(**inputs) 2025-12-04T08:49:08.7294148Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1280, in forward 2025-12-04T08:49:08.7294224Z outputs = self.bert( 2025-12-04T08:49:08.7294480Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1000, in forward 2025-12-04T08:49:08.7294554Z encoder_outputs = self.encoder( 2025-12-04T08:49:08.7294826Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 650, in forward 2025-12-04T08:49:08.7294897Z layer_outputs = layer_module( 2025-12-04T08:49:08.7295135Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:49:08.7295217Z return super().__call__(*args, **kwargs) 2025-12-04T08:49:08.7295502Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:49:08.7295582Z return func(*args, **kwargs) 2025-12-04T08:49:08.7295832Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 588, in forward 2025-12-04T08:49:08.7295917Z layer_output = apply_chunking_to_forward( 2025-12-04T08:49:08.7296194Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:49:08.7296271Z return forward_fn(*input_tensors) 2025-12-04T08:49:08.7296561Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 597, in feed_forward_chunk 2025-12-04T08:49:08.7296699Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T08:49:08.7297005Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 525, in forward 2025-12-04T08:49:08.7297108Z hidden_states = self.dense(hidden_states) 2025-12-04T08:49:08.7297112Z 2025-12-04T08:49:08.7297224Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:49:08.7297453Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:49:08.7297523Z res = mod(**inputs) 2025-12-04T08:49:08.7297801Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1295, in forward 2025-12-04T08:49:08.7297911Z prediction_scores = self.cls(sequence_output) 2025-12-04T08:49:08.7298182Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 752, in forward 2025-12-04T08:49:08.7298298Z prediction_scores = self.predictions(sequence_output) 2025-12-04T08:49:08.7298567Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 741, in forward 2025-12-04T08:49:08.7298668Z hidden_states = self.transform(hidden_states) 2025-12-04T08:49:08.7298934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 717, in forward 2025-12-04T08:49:08.7299018Z hidden_states = self.dense(hidden_states) 2025-12-04T08:49:08.7299021Z 2025-12-04T08:49:08.7299125Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:49:08.7299337Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:49:08.7299402Z res = mod(**inputs) 2025-12-04T08:49:08.7299667Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1295, in forward 2025-12-04T08:49:08.7299761Z prediction_scores = self.cls(sequence_output) 2025-12-04T08:49:08.7300028Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 752, in forward 2025-12-04T08:49:08.7300160Z prediction_scores = self.predictions(sequence_output) 2025-12-04T08:49:08.7300434Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 742, in forward 2025-12-04T08:49:08.7300533Z hidden_states = self.decoder(hidden_states) 2025-12-04T08:49:08.7300544Z 2025-12-04T08:49:08.7300655Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:49:08.7300867Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:49:08.7300943Z res = mod(**inputs) 2025-12-04T08:49:08.7301218Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1300, in forward 2025-12-04T08:49:08.7301426Z masked_lm_loss = loss_fct(prediction_scores.view(-1, self.config.vocab_size), labels.view(-1)) 2025-12-04T08:49:08.7301430Z 2025-12-04T08:49:19.5051691Z Compilation time (from dynamo_timed): 17.269699269 2025-12-04T08:49:19.5119018Z pass 2025-12-04T08:49:19.5126007Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T08:49:19.5126934Z TIMING: _recursive_pre_grad_passes:0.00709 _recursive_joint_graph_passes:0.66702 _recursive_post_grad_passes:0.07038 async_compile.wait:0.78855 code_gen:9.8619 inductor_compile:11.20472 backend_compile:14.33152 gc:0.0005 entire_frame_compile:17.2697 total_wall_time:17.2697 2025-12-04T08:49:19.5127909Z STATS: call_* op count: 289 | FakeTensorMode.__torch_dispatch__:7149 | FakeTensor.__torch_dispatch__:4342 | ProxyTorchDispatchMode.__torch_dispatch__:1972 2025-12-04T08:49:19.5128452Z Dynamo produced 1 graphs covering 289 ops with 0 graph breaks (0 unique) 2025-12-04T08:49:21.8888387Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/__init__.py:65: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-12-04T08:49:21.8889746Z import pynvml # type: ignore[import] 2025-12-04T08:49:25.4933763Z 2025-12-04T08:49:44.8378965Z loading model: 0it [00:00, ?it/s] 2025-12-04T08:49:44.8379273Z loading model: 0it [00:19, ?it/s] 2025-12-04T08:49:44.8409887Z cpu eval BlenderbotForCausalLM 2025-12-04T08:49:44.8806153Z Compilation time (from dynamo_timed): 0 2025-12-04T08:49:44.8806490Z pass_due_to_skip 2025-12-04T08:49:44.8806893Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T08:49:44.8807254Z TIMING: total_wall_time:0 2025-12-04T08:49:44.8807461Z STATS: call_* op count: 0 2025-12-04T08:49:44.8809261Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique) 2025-12-04T08:49:47.3719752Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/__init__.py:65: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-12-04T08:49:47.3720939Z import pynvml # type: ignore[import] 2025-12-04T08:49:51.7907449Z 2025-12-04T08:50:01.5897766Z loading model: 0it [00:00, ?it/s] 2025-12-04T08:50:01.5898085Z loading model: 0it [00:09, ?it/s] 2025-12-04T08:50:01.5919780Z cpu eval DebertaV2ForMaskedLM 2025-12-04T08:50:01.6116364Z Compilation time (from dynamo_timed): 0 2025-12-04T08:50:01.6116669Z pass_due_to_skip 2025-12-04T08:50:01.6116998Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T08:50:01.6117360Z TIMING: total_wall_time:0 2025-12-04T08:50:01.6117568Z STATS: call_* op count: 0 2025-12-04T08:50:01.6117855Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique) 2025-12-04T08:50:03.5540007Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/__init__.py:65: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-12-04T08:50:03.5540847Z import pynvml # type: ignore[import] 2025-12-04T08:50:06.9718751Z 2025-12-04T08:50:07.6989895Z loading model: 0it [00:00, ?it/s] 2025-12-04T08:50:07.6995298Z loading model: 0it [00:00, ?it/s] 2025-12-04T08:50:07.6996512Z cpu eval DistilBertForMaskedLM 2025-12-04T08:50:08.0086522Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T08:50:08.1468391Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T08:50:08.2817912Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T08:50:13.2070875Z cudagraph partition due to non gpu ops 2025-12-04T08:50:13.2071298Z cudagraph partition due to non gpu ops 2025-12-04T08:50:13.2071591Z cudagraph partition due to non gpu ops 2025-12-04T08:50:13.2071831Z cudagraph partition due to non gpu ops 2025-12-04T08:50:13.2072473Z cudagraph partition due to non gpu ops 2025-12-04T08:50:13.2072710Z cudagraph partition due to non gpu ops 2025-12-04T08:50:13.2072999Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:50:13.2073471Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:50:13.2073842Z res = mod(**inputs) 2025-12-04T08:50:13.2074329Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2025-12-04T08:50:13.2074803Z dlbrt_output = self.distilbert( 2025-12-04T08:50:13.2075300Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 724, in forward 2025-12-04T08:50:13.2075760Z return self.transformer( 2025-12-04T08:50:13.2076301Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-12-04T08:50:13.2076773Z layer_outputs = layer_module( 2025-12-04T08:50:13.2077166Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:50:13.2077573Z return super().__call__(*args, **kwargs) 2025-12-04T08:50:13.2078045Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 466, in forward 2025-12-04T08:50:13.2078644Z sa_output = self.attention( 2025-12-04T08:50:13.2079111Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 388, in forward 2025-12-04T08:50:13.2079648Z q = shape(self.q_lin(query)) # (bs, n_heads, q_length, dim_per_head) 2025-12-04T08:50:13.2079864Z 2025-12-04T08:50:13.2079984Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:50:13.2080411Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:50:13.2080819Z res = mod(**inputs) 2025-12-04T08:50:13.2081259Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2025-12-04T08:50:13.2081724Z dlbrt_output = self.distilbert( 2025-12-04T08:50:13.2082187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 724, in forward 2025-12-04T08:50:13.2082645Z return self.transformer( 2025-12-04T08:50:13.2083094Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-12-04T08:50:13.2083560Z layer_outputs = layer_module( 2025-12-04T08:50:13.2083973Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:50:13.2084383Z return super().__call__(*args, **kwargs) 2025-12-04T08:50:13.2084858Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 466, in forward 2025-12-04T08:50:13.2085324Z sa_output = self.attention( 2025-12-04T08:50:13.2085770Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 389, in forward 2025-12-04T08:50:13.2086294Z k = shape(self.k_lin(key)) # (bs, n_heads, k_length, dim_per_head) 2025-12-04T08:50:13.2086497Z 2025-12-04T08:50:13.2086612Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:50:13.2087014Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:50:13.2087373Z res = mod(**inputs) 2025-12-04T08:50:13.2087819Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2025-12-04T08:50:13.2088284Z dlbrt_output = self.distilbert( 2025-12-04T08:50:13.2088856Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 724, in forward 2025-12-04T08:50:13.2089321Z return self.transformer( 2025-12-04T08:50:13.2089777Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-12-04T08:50:13.2090238Z layer_outputs = layer_module( 2025-12-04T08:50:13.2090619Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:50:13.2091031Z return super().__call__(*args, **kwargs) 2025-12-04T08:50:13.2091499Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 466, in forward 2025-12-04T08:50:13.2091966Z sa_output = self.attention( 2025-12-04T08:50:13.2092397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 390, in forward 2025-12-04T08:50:13.2092953Z v = shape(self.v_lin(value)) # (bs, n_heads, k_length, dim_per_head) 2025-12-04T08:50:13.2093153Z 2025-12-04T08:50:13.2093242Z cudagraph partition due to non gpu ops 2025-12-04T08:50:13.2093504Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:50:13.2093894Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:50:13.2094252Z res = mod(**inputs) 2025-12-04T08:50:13.2094672Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2025-12-04T08:50:13.2095127Z dlbrt_output = self.distilbert( 2025-12-04T08:50:13.2095576Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 724, in forward 2025-12-04T08:50:13.2096027Z return self.transformer( 2025-12-04T08:50:13.2096484Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-12-04T08:50:13.2096942Z layer_outputs = layer_module( 2025-12-04T08:50:13.2097326Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:50:13.2097731Z return super().__call__(*args, **kwargs) 2025-12-04T08:50:13.2098187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 466, in forward 2025-12-04T08:50:13.2098634Z sa_output = self.attention( 2025-12-04T08:50:13.2099079Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 392, in forward 2025-12-04T08:50:13.2099598Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T08:50:13.2099803Z 2025-12-04T08:50:13.2099917Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:50:13.2100318Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:50:13.2100675Z res = mod(**inputs) 2025-12-04T08:50:13.2101093Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2025-12-04T08:50:13.2101543Z dlbrt_output = self.distilbert( 2025-12-04T08:50:13.2102000Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 724, in forward 2025-12-04T08:50:13.2102463Z return self.transformer( 2025-12-04T08:50:13.2102900Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-12-04T08:50:13.2103394Z layer_outputs = layer_module( 2025-12-04T08:50:13.2103785Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:50:13.2104187Z return super().__call__(*args, **kwargs) 2025-12-04T08:50:13.2104676Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 466, in forward 2025-12-04T08:50:13.2105132Z sa_output = self.attention( 2025-12-04T08:50:13.2105568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 402, in forward 2025-12-04T08:50:13.2106032Z attn_output = self.out_lin(attn_output) 2025-12-04T08:50:13.2106184Z 2025-12-04T08:50:13.2106297Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:50:13.2106697Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:50:13.2107051Z res = mod(**inputs) 2025-12-04T08:50:13.2107464Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2025-12-04T08:50:13.2108028Z dlbrt_output = self.distilbert( 2025-12-04T08:50:13.2108486Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 724, in forward 2025-12-04T08:50:13.2108937Z return self.transformer( 2025-12-04T08:50:13.2109365Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-12-04T08:50:13.2109823Z layer_outputs = layer_module( 2025-12-04T08:50:13.2110205Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:50:13.2110614Z return super().__call__(*args, **kwargs) 2025-12-04T08:50:13.2111071Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 484, in forward 2025-12-04T08:50:13.2111570Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-12-04T08:50:13.2112068Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 418, in forward 2025-12-04T08:50:13.2112668Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-12-04T08:50:13.2113246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:50:13.2113682Z return forward_fn(*input_tensors) 2025-12-04T08:50:13.2114134Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 421, in ff_chunk 2025-12-04T08:50:13.2114660Z x = self.lin1(input) 2025-12-04T08:50:13.2114791Z 2025-12-04T08:50:13.2114908Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:50:13.2115316Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:50:13.2115678Z res = mod(**inputs) 2025-12-04T08:50:13.2116113Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2025-12-04T08:50:13.2116597Z dlbrt_output = self.distilbert( 2025-12-04T08:50:13.2117059Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 724, in forward 2025-12-04T08:50:13.2117511Z return self.transformer( 2025-12-04T08:50:13.2117956Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-12-04T08:50:13.2118691Z layer_outputs = layer_module( 2025-12-04T08:50:13.2119088Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:50:13.2119498Z return super().__call__(*args, **kwargs) 2025-12-04T08:50:13.2119972Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 484, in forward 2025-12-04T08:50:13.2120499Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-12-04T08:50:13.2121289Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 418, in forward 2025-12-04T08:50:13.2121904Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-12-04T08:50:13.2122491Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:50:13.2122948Z return forward_fn(*input_tensors) 2025-12-04T08:50:13.2123422Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 422, in ff_chunk 2025-12-04T08:50:13.2123883Z x = self.activation(x) 2025-12-04T08:50:13.2124260Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T08:50:13.2124702Z return self.act(input) 2025-12-04T08:50:13.2124828Z 2025-12-04T08:50:13.2125103Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:50:13.2125519Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:50:13.2125932Z res = mod(**inputs) 2025-12-04T08:50:13.2126373Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2025-12-04T08:50:13.2126838Z dlbrt_output = self.distilbert( 2025-12-04T08:50:13.2127304Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 724, in forward 2025-12-04T08:50:13.2127775Z return self.transformer( 2025-12-04T08:50:13.2128287Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-12-04T08:50:13.2128764Z layer_outputs = layer_module( 2025-12-04T08:50:13.2129165Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:50:13.2129577Z return super().__call__(*args, **kwargs) 2025-12-04T08:50:13.2130048Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 484, in forward 2025-12-04T08:50:13.2130567Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-12-04T08:50:13.2131075Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 418, in forward 2025-12-04T08:50:13.2131682Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-12-04T08:50:13.2132246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:50:13.2132693Z return forward_fn(*input_tensors) 2025-12-04T08:50:13.2133159Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 423, in ff_chunk 2025-12-04T08:50:13.2133603Z x = self.lin2(x) 2025-12-04T08:50:13.2133711Z 2025-12-04T08:50:13.2133826Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:50:13.2134224Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:50:13.2134579Z res = mod(**inputs) 2025-12-04T08:50:13.2134992Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2025-12-04T08:50:13.2135443Z dlbrt_output = self.distilbert( 2025-12-04T08:50:13.2135889Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 724, in forward 2025-12-04T08:50:13.2136334Z return self.transformer( 2025-12-04T08:50:13.2136761Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-12-04T08:50:13.2137252Z layer_outputs = layer_module( 2025-12-04T08:50:13.2137635Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:50:13.2138027Z return super().__call__(*args, **kwargs) 2025-12-04T08:50:13.2138478Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 466, in forward 2025-12-04T08:50:13.2138935Z sa_output = self.attention( 2025-12-04T08:50:13.2139370Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 388, in forward 2025-12-04T08:50:13.2139865Z q = shape(self.q_lin(query)) # (bs, n_heads, q_length, dim_per_head) 2025-12-04T08:50:13.2140071Z 2025-12-04T08:50:13.2140184Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:50:13.2140628Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:50:13.2140984Z res = mod(**inputs) 2025-12-04T08:50:13.2141407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2025-12-04T08:50:13.2141865Z dlbrt_output = self.distilbert( 2025-12-04T08:50:13.2142318Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 724, in forward 2025-12-04T08:50:13.2142766Z return self.transformer( 2025-12-04T08:50:13.2143205Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-12-04T08:50:13.2143661Z layer_outputs = layer_module( 2025-12-04T08:50:13.2144049Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:50:13.2144449Z return super().__call__(*args, **kwargs) 2025-12-04T08:50:13.2144914Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 466, in forward 2025-12-04T08:50:13.2145371Z sa_output = self.attention( 2025-12-04T08:50:13.2145832Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 389, in forward 2025-12-04T08:50:13.2146356Z k = shape(self.k_lin(key)) # (bs, n_heads, k_length, dim_per_head) 2025-12-04T08:50:13.2146559Z 2025-12-04T08:50:13.2146678Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:50:13.2147087Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:50:13.2147451Z res = mod(**inputs) 2025-12-04T08:50:13.2147878Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2025-12-04T08:50:13.2148337Z dlbrt_output = self.distilbert( 2025-12-04T08:50:13.2148799Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 724, in forward 2025-12-04T08:50:13.2149258Z return self.transformer( 2025-12-04T08:50:13.2149719Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-12-04T08:50:13.2150187Z layer_outputs = layer_module( 2025-12-04T08:50:13.2150583Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:50:13.2150995Z return super().__call__(*args, **kwargs) 2025-12-04T08:50:13.2151474Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 466, in forward 2025-12-04T08:50:13.2151951Z sa_output = self.attention( 2025-12-04T08:50:13.2152406Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 390, in forward 2025-12-04T08:50:13.2152988Z v = shape(self.v_lin(value)) # (bs, n_heads, k_length, dim_per_head) 2025-12-04T08:50:13.2153197Z 2025-12-04T08:50:13.2153289Z cudagraph partition due to non gpu ops 2025-12-04T08:50:13.2153558Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:50:13.2153960Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:50:13.2154327Z res = mod(**inputs) 2025-12-04T08:50:13.2154760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2025-12-04T08:50:13.2155228Z dlbrt_output = self.distilbert( 2025-12-04T08:50:13.2155687Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 724, in forward 2025-12-04T08:50:13.2156145Z return self.transformer( 2025-12-04T08:50:13.2156632Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-12-04T08:50:13.2157094Z layer_outputs = layer_module( 2025-12-04T08:50:13.2157494Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:50:13.2157911Z return super().__call__(*args, **kwargs) 2025-12-04T08:50:13.2158483Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 466, in forward 2025-12-04T08:50:13.2158949Z sa_output = self.attention( 2025-12-04T08:50:13.2159403Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 392, in forward 2025-12-04T08:50:13.2159995Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T08:50:13.2160203Z 2025-12-04T08:50:13.2160318Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:50:13.2160735Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:50:13.2161101Z res = mod(**inputs) 2025-12-04T08:50:13.2161532Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2025-12-04T08:50:13.2161974Z dlbrt_output = self.distilbert( 2025-12-04T08:50:13.2162416Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 724, in forward 2025-12-04T08:50:13.2162861Z return self.transformer( 2025-12-04T08:50:13.2163293Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-12-04T08:50:13.2163736Z layer_outputs = layer_module( 2025-12-04T08:50:13.2164116Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:50:13.2164516Z return super().__call__(*args, **kwargs) 2025-12-04T08:50:13.2164973Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 466, in forward 2025-12-04T08:50:13.2165419Z sa_output = self.attention( 2025-12-04T08:50:13.2165858Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 402, in forward 2025-12-04T08:50:13.2166325Z attn_output = self.out_lin(attn_output) 2025-12-04T08:50:13.2166477Z 2025-12-04T08:50:13.2166586Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:50:13.2166976Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:50:13.2167326Z res = mod(**inputs) 2025-12-04T08:50:13.2167732Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2025-12-04T08:50:13.2168181Z dlbrt_output = self.distilbert( 2025-12-04T08:50:13.2168688Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 724, in forward 2025-12-04T08:50:13.2169138Z return self.transformer( 2025-12-04T08:50:13.2169573Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-12-04T08:50:13.2170033Z layer_outputs = layer_module( 2025-12-04T08:50:13.2170415Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:50:13.2170820Z return super().__call__(*args, **kwargs) 2025-12-04T08:50:13.2171266Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 484, in forward 2025-12-04T08:50:13.2171762Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-12-04T08:50:13.2172255Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 418, in forward 2025-12-04T08:50:13.2172879Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-12-04T08:50:13.2173457Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:50:13.2173894Z return forward_fn(*input_tensors) 2025-12-04T08:50:13.2174353Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 421, in ff_chunk 2025-12-04T08:50:13.2174822Z x = self.lin1(input) 2025-12-04T08:50:13.2174949Z 2025-12-04T08:50:13.2175061Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:50:13.2175454Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:50:13.2175804Z res = mod(**inputs) 2025-12-04T08:50:13.2176213Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2025-12-04T08:50:13.2176671Z dlbrt_output = self.distilbert( 2025-12-04T08:50:13.2177112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 724, in forward 2025-12-04T08:50:13.2177552Z return self.transformer( 2025-12-04T08:50:13.2177985Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-12-04T08:50:13.2178432Z layer_outputs = layer_module( 2025-12-04T08:50:13.2178810Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:50:13.2179199Z return super().__call__(*args, **kwargs) 2025-12-04T08:50:13.2179649Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 484, in forward 2025-12-04T08:50:13.2180141Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-12-04T08:50:13.2180629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 418, in forward 2025-12-04T08:50:13.2181204Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-12-04T08:50:13.2181767Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:50:13.2182201Z return forward_fn(*input_tensors) 2025-12-04T08:50:13.2182655Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 422, in ff_chunk 2025-12-04T08:50:13.2183099Z x = self.activation(x) 2025-12-04T08:50:13.2183462Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T08:50:13.2183843Z return self.act(input) 2025-12-04T08:50:13.2183962Z 2025-12-04T08:50:13.2184077Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:50:13.2184512Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:50:13.2184867Z res = mod(**inputs) 2025-12-04T08:50:13.2185292Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2025-12-04T08:50:13.2185739Z dlbrt_output = self.distilbert( 2025-12-04T08:50:13.2186186Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 724, in forward 2025-12-04T08:50:13.2186643Z return self.transformer( 2025-12-04T08:50:13.2187071Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-12-04T08:50:13.2187520Z layer_outputs = layer_module( 2025-12-04T08:50:13.2187939Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:50:13.2188348Z return super().__call__(*args, **kwargs) 2025-12-04T08:50:13.2188795Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 484, in forward 2025-12-04T08:50:13.2189302Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-12-04T08:50:13.2189799Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 418, in forward 2025-12-04T08:50:13.2190393Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-12-04T08:50:13.2190948Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:50:13.2191380Z return forward_fn(*input_tensors) 2025-12-04T08:50:13.2191839Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 423, in ff_chunk 2025-12-04T08:50:13.2192299Z x = self.lin2(x) 2025-12-04T08:50:13.2192410Z 2025-12-04T08:50:13.2192521Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:50:13.2192917Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:50:13.2193267Z res = mod(**inputs) 2025-12-04T08:50:13.2193681Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2025-12-04T08:50:13.2194145Z dlbrt_output = self.distilbert( 2025-12-04T08:50:13.2194593Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 724, in forward 2025-12-04T08:50:13.2195045Z return self.transformer( 2025-12-04T08:50:13.2195484Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-12-04T08:50:13.2195938Z layer_outputs = layer_module( 2025-12-04T08:50:13.2196323Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:50:13.2196717Z return super().__call__(*args, **kwargs) 2025-12-04T08:50:13.2197173Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 466, in forward 2025-12-04T08:50:13.2197627Z sa_output = self.attention( 2025-12-04T08:50:13.2198060Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 388, in forward 2025-12-04T08:50:13.2198655Z q = shape(self.q_lin(query)) # (bs, n_heads, q_length, dim_per_head) 2025-12-04T08:50:13.2198867Z 2025-12-04T08:50:13.2198984Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:50:13.2199393Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:50:13.2199774Z res = mod(**inputs) 2025-12-04T08:50:13.2200238Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2025-12-04T08:50:13.2200694Z dlbrt_output = self.distilbert( 2025-12-04T08:50:13.2201140Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 724, in forward 2025-12-04T08:50:13.2201584Z return self.transformer( 2025-12-04T08:50:13.2202022Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-12-04T08:50:13.2202470Z layer_outputs = layer_module( 2025-12-04T08:50:13.2202858Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:50:13.2203253Z return super().__call__(*args, **kwargs) 2025-12-04T08:50:13.2203752Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 466, in forward 2025-12-04T08:50:13.2204209Z sa_output = self.attention( 2025-12-04T08:50:13.2204647Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 389, in forward 2025-12-04T08:50:13.2205158Z k = shape(self.k_lin(key)) # (bs, n_heads, k_length, dim_per_head) 2025-12-04T08:50:13.2205361Z 2025-12-04T08:50:13.2205475Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:50:13.2205876Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:50:13.2206240Z res = mod(**inputs) 2025-12-04T08:50:13.2206671Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2025-12-04T08:50:13.2207130Z dlbrt_output = self.distilbert( 2025-12-04T08:50:13.2207622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 724, in forward 2025-12-04T08:50:13.2208078Z return self.transformer( 2025-12-04T08:50:13.2208498Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-12-04T08:50:13.2208964Z layer_outputs = layer_module( 2025-12-04T08:50:13.2209345Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:50:13.2209756Z return super().__call__(*args, **kwargs) 2025-12-04T08:50:13.2210227Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 466, in forward 2025-12-04T08:50:13.2210696Z sa_output = self.attention( 2025-12-04T08:50:13.2211143Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 390, in forward 2025-12-04T08:50:13.2211661Z v = shape(self.v_lin(value)) # (bs, n_heads, k_length, dim_per_head) 2025-12-04T08:50:13.2211868Z 2025-12-04T08:50:13.2211961Z cudagraph partition due to non gpu ops 2025-12-04T08:50:13.2212232Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:50:13.2212625Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:50:13.2212989Z res = mod(**inputs) 2025-12-04T08:50:13.2213417Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2025-12-04T08:50:13.2213878Z dlbrt_output = self.distilbert( 2025-12-04T08:50:13.2214340Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 724, in forward 2025-12-04T08:50:13.2214803Z return self.transformer( 2025-12-04T08:50:13.2215249Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-12-04T08:50:13.2215743Z layer_outputs = layer_module( 2025-12-04T08:50:13.2216127Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:50:13.2216529Z return super().__call__(*args, **kwargs) 2025-12-04T08:50:13.2216985Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 466, in forward 2025-12-04T08:50:13.2217431Z sa_output = self.attention( 2025-12-04T08:50:13.2217878Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 392, in forward 2025-12-04T08:50:13.2218409Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T08:50:13.2218616Z 2025-12-04T08:50:13.2218729Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:50:13.2219172Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:50:13.2219540Z res = mod(**inputs) 2025-12-04T08:50:13.2219962Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2025-12-04T08:50:13.2220423Z dlbrt_output = self.distilbert( 2025-12-04T08:50:13.2221018Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 724, in forward 2025-12-04T08:50:13.2221474Z return self.transformer( 2025-12-04T08:50:13.2221899Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-12-04T08:50:13.2222348Z layer_outputs = layer_module( 2025-12-04T08:50:13.2222731Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:50:13.2223132Z return super().__call__(*args, **kwargs) 2025-12-04T08:50:13.2223587Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 466, in forward 2025-12-04T08:50:13.2224040Z sa_output = self.attention( 2025-12-04T08:50:13.2224480Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 402, in forward 2025-12-04T08:50:13.2224941Z attn_output = self.out_lin(attn_output) 2025-12-04T08:50:13.2225090Z 2025-12-04T08:50:13.2225204Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:50:13.2225593Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:50:13.2225945Z res = mod(**inputs) 2025-12-04T08:50:13.2226357Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2025-12-04T08:50:13.2226808Z dlbrt_output = self.distilbert( 2025-12-04T08:50:13.2227258Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 724, in forward 2025-12-04T08:50:13.2227723Z return self.transformer( 2025-12-04T08:50:13.2228152Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-12-04T08:50:13.2228610Z layer_outputs = layer_module( 2025-12-04T08:50:13.2228990Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:50:13.2229396Z return super().__call__(*args, **kwargs) 2025-12-04T08:50:13.2229848Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 484, in forward 2025-12-04T08:50:13.2230341Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-12-04T08:50:13.2230830Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 418, in forward 2025-12-04T08:50:13.2231524Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-12-04T08:50:13.2232090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:50:13.2232527Z return forward_fn(*input_tensors) 2025-12-04T08:50:13.2232992Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 421, in ff_chunk 2025-12-04T08:50:13.2233444Z x = self.lin1(input) 2025-12-04T08:50:13.2248025Z 2025-12-04T08:50:13.2248253Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:50:13.2248732Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:50:13.2249113Z res = mod(**inputs) 2025-12-04T08:50:13.2249603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2025-12-04T08:50:13.2250333Z dlbrt_output = self.distilbert( 2025-12-04T08:50:13.2250808Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 724, in forward 2025-12-04T08:50:13.2251287Z return self.transformer( 2025-12-04T08:50:13.2251753Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-12-04T08:50:13.2252219Z layer_outputs = layer_module( 2025-12-04T08:50:13.2252589Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:50:13.2253003Z return super().__call__(*args, **kwargs) 2025-12-04T08:50:13.2253474Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 484, in forward 2025-12-04T08:50:13.2253958Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-12-04T08:50:13.2254428Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 418, in forward 2025-12-04T08:50:13.2255028Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-12-04T08:50:13.2255605Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:50:13.2256045Z return forward_fn(*input_tensors) 2025-12-04T08:50:13.2256511Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 422, in ff_chunk 2025-12-04T08:50:13.2256945Z x = self.activation(x) 2025-12-04T08:50:13.2257293Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T08:50:13.2257649Z return self.act(input) 2025-12-04T08:50:13.2257782Z 2025-12-04T08:50:13.2257903Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:50:13.2258311Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:50:13.2258671Z res = mod(**inputs) 2025-12-04T08:50:13.2259089Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2025-12-04T08:50:13.2259553Z dlbrt_output = self.distilbert( 2025-12-04T08:50:13.2259978Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 724, in forward 2025-12-04T08:50:13.2260419Z return self.transformer( 2025-12-04T08:50:13.2260855Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-12-04T08:50:13.2261308Z layer_outputs = layer_module( 2025-12-04T08:50:13.2261695Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:50:13.2262172Z return super().__call__(*args, **kwargs) 2025-12-04T08:50:13.2262652Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 484, in forward 2025-12-04T08:50:13.2263129Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-12-04T08:50:13.2263591Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 418, in forward 2025-12-04T08:50:13.2264160Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-12-04T08:50:13.2264708Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:50:13.2265135Z return forward_fn(*input_tensors) 2025-12-04T08:50:13.2265568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 423, in ff_chunk 2025-12-04T08:50:13.2266039Z x = self.lin2(x) 2025-12-04T08:50:13.2266155Z 2025-12-04T08:50:13.2266273Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:50:13.2266686Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:50:13.2267037Z res = mod(**inputs) 2025-12-04T08:50:13.2267471Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2025-12-04T08:50:13.2267904Z dlbrt_output = self.distilbert( 2025-12-04T08:50:13.2268330Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 724, in forward 2025-12-04T08:50:13.2268747Z return self.transformer( 2025-12-04T08:50:13.2269160Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-12-04T08:50:13.2269607Z layer_outputs = layer_module( 2025-12-04T08:50:13.2269987Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:50:13.2270393Z return super().__call__(*args, **kwargs) 2025-12-04T08:50:13.2270851Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 466, in forward 2025-12-04T08:50:13.2271314Z sa_output = self.attention( 2025-12-04T08:50:13.2271727Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 388, in forward 2025-12-04T08:50:13.2272242Z q = shape(self.q_lin(query)) # (bs, n_heads, q_length, dim_per_head) 2025-12-04T08:50:13.2272442Z 2025-12-04T08:50:13.2272567Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:50:13.2272969Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:50:13.2273320Z res = mod(**inputs) 2025-12-04T08:50:13.2273747Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2025-12-04T08:50:13.2274201Z dlbrt_output = self.distilbert( 2025-12-04T08:50:13.2274650Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 724, in forward 2025-12-04T08:50:13.2275104Z return self.transformer( 2025-12-04T08:50:13.2275551Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-12-04T08:50:13.2276005Z layer_outputs = layer_module( 2025-12-04T08:50:13.2276381Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:50:13.2276781Z return super().__call__(*args, **kwargs) 2025-12-04T08:50:13.2277287Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 466, in forward 2025-12-04T08:50:13.2277736Z sa_output = self.attention( 2025-12-04T08:50:13.2278280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 389, in forward 2025-12-04T08:50:13.2278802Z k = shape(self.k_lin(key)) # (bs, n_heads, k_length, dim_per_head) 2025-12-04T08:50:13.2278995Z 2025-12-04T08:50:13.2279119Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:50:13.2279508Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:50:13.2279868Z res = mod(**inputs) 2025-12-04T08:50:13.2280310Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2025-12-04T08:50:13.2280771Z dlbrt_output = self.distilbert( 2025-12-04T08:50:13.2281350Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 724, in forward 2025-12-04T08:50:13.2281807Z return self.transformer( 2025-12-04T08:50:13.2282245Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-12-04T08:50:13.2282690Z layer_outputs = layer_module( 2025-12-04T08:50:13.2283075Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:50:13.2283476Z return super().__call__(*args, **kwargs) 2025-12-04T08:50:13.2283929Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 466, in forward 2025-12-04T08:50:13.2284371Z sa_output = self.attention( 2025-12-04T08:50:13.2284812Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 390, in forward 2025-12-04T08:50:13.2285314Z v = shape(self.v_lin(value)) # (bs, n_heads, k_length, dim_per_head) 2025-12-04T08:50:13.2285506Z 2025-12-04T08:50:13.2285603Z cudagraph partition due to non gpu ops 2025-12-04T08:50:13.2285847Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:50:13.2286222Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:50:13.2286577Z res = mod(**inputs) 2025-12-04T08:50:13.2287184Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2025-12-04T08:50:13.2287649Z dlbrt_output = self.distilbert( 2025-12-04T08:50:13.2288107Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 724, in forward 2025-12-04T08:50:13.2288561Z return self.transformer( 2025-12-04T08:50:13.2288989Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-12-04T08:50:13.2289460Z layer_outputs = layer_module( 2025-12-04T08:50:13.2289825Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:50:13.2290223Z return super().__call__(*args, **kwargs) 2025-12-04T08:50:13.2290683Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 466, in forward 2025-12-04T08:50:13.2291138Z sa_output = self.attention( 2025-12-04T08:50:13.2291565Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 392, in forward 2025-12-04T08:50:13.2292085Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T08:50:13.2292297Z 2025-12-04T08:50:13.2292407Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:50:13.2292791Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:50:13.2293122Z res = mod(**inputs) 2025-12-04T08:50:13.2293565Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2025-12-04T08:50:13.2293993Z dlbrt_output = self.distilbert( 2025-12-04T08:50:13.2294418Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 724, in forward 2025-12-04T08:50:13.2294838Z return self.transformer( 2025-12-04T08:50:13.2295253Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-12-04T08:50:13.2295681Z layer_outputs = layer_module( 2025-12-04T08:50:13.2296036Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:50:13.2296440Z return super().__call__(*args, **kwargs) 2025-12-04T08:50:13.2296953Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 466, in forward 2025-12-04T08:50:13.2297412Z sa_output = self.attention( 2025-12-04T08:50:13.2297815Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 402, in forward 2025-12-04T08:50:13.2298252Z attn_output = self.out_lin(attn_output) 2025-12-04T08:50:13.2298394Z 2025-12-04T08:50:13.2298506Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:50:13.2298881Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:50:13.2299207Z res = mod(**inputs) 2025-12-04T08:50:13.2299601Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2025-12-04T08:50:13.2300029Z dlbrt_output = self.distilbert( 2025-12-04T08:50:13.2300445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 724, in forward 2025-12-04T08:50:13.2300873Z return self.transformer( 2025-12-04T08:50:13.2301304Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-12-04T08:50:13.2301776Z layer_outputs = layer_module( 2025-12-04T08:50:13.2302125Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:50:13.2302499Z return super().__call__(*args, **kwargs) 2025-12-04T08:50:13.2302925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 484, in forward 2025-12-04T08:50:13.2303383Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-12-04T08:50:13.2303870Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 418, in forward 2025-12-04T08:50:13.2304454Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-12-04T08:50:13.2305016Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:50:13.2305444Z return forward_fn(*input_tensors) 2025-12-04T08:50:13.2305899Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 421, in ff_chunk 2025-12-04T08:50:13.2306347Z x = self.lin1(input) 2025-12-04T08:50:13.2306464Z 2025-12-04T08:50:13.2306583Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:50:13.2306966Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:50:13.2307319Z res = mod(**inputs) 2025-12-04T08:50:13.2307801Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2025-12-04T08:50:13.2308270Z dlbrt_output = self.distilbert( 2025-12-04T08:50:13.2308763Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 724, in forward 2025-12-04T08:50:13.2309218Z return self.transformer( 2025-12-04T08:50:13.2309684Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-12-04T08:50:13.2310125Z layer_outputs = layer_module( 2025-12-04T08:50:13.2310507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:50:13.2310905Z return super().__call__(*args, **kwargs) 2025-12-04T08:50:13.2311361Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 484, in forward 2025-12-04T08:50:13.2311843Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-12-04T08:50:13.2312387Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 418, in forward 2025-12-04T08:50:13.2312984Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-12-04T08:50:13.2313555Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:50:13.2313984Z return forward_fn(*input_tensors) 2025-12-04T08:50:13.2314441Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 422, in ff_chunk 2025-12-04T08:50:13.2314894Z x = self.activation(x) 2025-12-04T08:50:13.2315251Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T08:50:13.2315631Z return self.act(input) 2025-12-04T08:50:13.2315762Z 2025-12-04T08:50:13.2315877Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:50:13.2316279Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:50:13.2316630Z res = mod(**inputs) 2025-12-04T08:50:13.2317058Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2025-12-04T08:50:13.2317525Z dlbrt_output = self.distilbert( 2025-12-04T08:50:13.2317970Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 724, in forward 2025-12-04T08:50:13.2318523Z return self.transformer( 2025-12-04T08:50:13.2318983Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-12-04T08:50:13.2319461Z layer_outputs = layer_module( 2025-12-04T08:50:13.2319850Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:50:13.2320258Z return super().__call__(*args, **kwargs) 2025-12-04T08:50:13.2320990Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 484, in forward 2025-12-04T08:50:13.2321501Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-12-04T08:50:13.2321997Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 418, in forward 2025-12-04T08:50:13.2322594Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-12-04T08:50:13.2323168Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:50:13.2323609Z return forward_fn(*input_tensors) 2025-12-04T08:50:13.2324059Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 423, in ff_chunk 2025-12-04T08:50:13.2324513Z x = self.lin2(x) 2025-12-04T08:50:13.2324710Z 2025-12-04T08:50:13.2324837Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:50:13.2325226Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:50:13.2325583Z res = mod(**inputs) 2025-12-04T08:50:13.2326010Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2025-12-04T08:50:13.2326465Z dlbrt_output = self.distilbert( 2025-12-04T08:50:13.2326907Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 724, in forward 2025-12-04T08:50:13.2327357Z return self.transformer( 2025-12-04T08:50:13.2327800Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-12-04T08:50:13.2328305Z layer_outputs = layer_module( 2025-12-04T08:50:13.2328689Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:50:13.2329090Z return super().__call__(*args, **kwargs) 2025-12-04T08:50:13.2329544Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 466, in forward 2025-12-04T08:50:13.2329990Z sa_output = self.attention( 2025-12-04T08:50:13.2330444Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 388, in forward 2025-12-04T08:50:13.2330963Z q = shape(self.q_lin(query)) # (bs, n_heads, q_length, dim_per_head) 2025-12-04T08:50:13.2331342Z 2025-12-04T08:50:13.2331467Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:50:13.2331879Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:50:13.2332237Z res = mod(**inputs) 2025-12-04T08:50:13.2332664Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2025-12-04T08:50:13.2333112Z dlbrt_output = self.distilbert( 2025-12-04T08:50:13.2333571Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 724, in forward 2025-12-04T08:50:13.2334037Z return self.transformer( 2025-12-04T08:50:13.2334475Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-12-04T08:50:13.2334931Z layer_outputs = layer_module( 2025-12-04T08:50:13.2335328Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:50:13.2335733Z return super().__call__(*args, **kwargs) 2025-12-04T08:50:13.2336192Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 466, in forward 2025-12-04T08:50:13.2336639Z sa_output = self.attention( 2025-12-04T08:50:13.2337086Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 389, in forward 2025-12-04T08:50:13.2337606Z k = shape(self.k_lin(key)) # (bs, n_heads, k_length, dim_per_head) 2025-12-04T08:50:13.2337801Z 2025-12-04T08:50:13.2337917Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:50:13.2338321Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:50:13.2338684Z res = mod(**inputs) 2025-12-04T08:50:13.2339117Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2025-12-04T08:50:13.2339580Z dlbrt_output = self.distilbert( 2025-12-04T08:50:13.2340028Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 724, in forward 2025-12-04T08:50:13.2340516Z return self.transformer( 2025-12-04T08:50:13.2340954Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-12-04T08:50:13.2345738Z layer_outputs = layer_module( 2025-12-04T08:50:13.2346143Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:50:13.2346572Z return super().__call__(*args, **kwargs) 2025-12-04T08:50:13.2347052Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 466, in forward 2025-12-04T08:50:13.2347527Z sa_output = self.attention( 2025-12-04T08:50:13.2347987Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 390, in forward 2025-12-04T08:50:13.2348541Z v = shape(self.v_lin(value)) # (bs, n_heads, k_length, dim_per_head) 2025-12-04T08:50:13.2348750Z 2025-12-04T08:50:13.2348849Z cudagraph partition due to non gpu ops 2025-12-04T08:50:13.2349123Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:50:13.2349529Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:50:13.2349927Z res = mod(**inputs) 2025-12-04T08:50:13.2350366Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2025-12-04T08:50:13.2350831Z dlbrt_output = self.distilbert( 2025-12-04T08:50:13.2351283Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 724, in forward 2025-12-04T08:50:13.2351745Z return self.transformer( 2025-12-04T08:50:13.2352194Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-12-04T08:50:13.2352649Z layer_outputs = layer_module( 2025-12-04T08:50:13.2353043Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:50:13.2353452Z return super().__call__(*args, **kwargs) 2025-12-04T08:50:13.2353907Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 466, in forward 2025-12-04T08:50:13.2354352Z sa_output = self.attention( 2025-12-04T08:50:13.2354790Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 392, in forward 2025-12-04T08:50:13.2355313Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T08:50:13.2355522Z 2025-12-04T08:50:13.2355646Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:50:13.2356049Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:50:13.2356412Z res = mod(**inputs) 2025-12-04T08:50:13.2356847Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2025-12-04T08:50:13.2357306Z dlbrt_output = self.distilbert( 2025-12-04T08:50:13.2357761Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 724, in forward 2025-12-04T08:50:13.2358300Z return self.transformer( 2025-12-04T08:50:13.2358756Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-12-04T08:50:13.2359213Z layer_outputs = layer_module( 2025-12-04T08:50:13.2359605Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:50:13.2360022Z return super().__call__(*args, **kwargs) 2025-12-04T08:50:13.2360483Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 466, in forward 2025-12-04T08:50:13.2360979Z sa_output = self.attention( 2025-12-04T08:50:13.2361420Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 402, in forward 2025-12-04T08:50:13.2361972Z attn_output = self.out_lin(attn_output) 2025-12-04T08:50:13.2362125Z 2025-12-04T08:50:13.2362248Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:50:13.2362639Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:50:13.2362982Z res = mod(**inputs) 2025-12-04T08:50:13.2363402Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2025-12-04T08:50:13.2363849Z dlbrt_output = self.distilbert( 2025-12-04T08:50:13.2364285Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 724, in forward 2025-12-04T08:50:13.2364756Z return self.transformer( 2025-12-04T08:50:13.2365193Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-12-04T08:50:13.2365642Z layer_outputs = layer_module( 2025-12-04T08:50:13.2366015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:50:13.2366411Z return super().__call__(*args, **kwargs) 2025-12-04T08:50:13.2366871Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 484, in forward 2025-12-04T08:50:13.2367364Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-12-04T08:50:13.2367850Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 418, in forward 2025-12-04T08:50:13.2368441Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-12-04T08:50:13.2369006Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:50:13.2369433Z return forward_fn(*input_tensors) 2025-12-04T08:50:13.2369885Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 421, in ff_chunk 2025-12-04T08:50:13.2370331Z x = self.lin1(input) 2025-12-04T08:50:13.2370448Z 2025-12-04T08:50:13.2370567Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:50:13.2370959Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:50:13.2371311Z res = mod(**inputs) 2025-12-04T08:50:13.2371728Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2025-12-04T08:50:13.2372182Z dlbrt_output = self.distilbert( 2025-12-04T08:50:13.2372622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 724, in forward 2025-12-04T08:50:13.2373077Z return self.transformer( 2025-12-04T08:50:13.2373509Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-12-04T08:50:13.2373952Z layer_outputs = layer_module( 2025-12-04T08:50:13.2374332Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:50:13.2374728Z return super().__call__(*args, **kwargs) 2025-12-04T08:50:13.2375180Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 484, in forward 2025-12-04T08:50:13.2375659Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-12-04T08:50:13.2376220Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 418, in forward 2025-12-04T08:50:13.2376824Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-12-04T08:50:13.2377387Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:50:13.2377876Z return forward_fn(*input_tensors) 2025-12-04T08:50:13.2378345Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 422, in ff_chunk 2025-12-04T08:50:13.2378823Z x = self.activation(x) 2025-12-04T08:50:13.2379191Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T08:50:13.2379578Z return self.act(input) 2025-12-04T08:50:13.2379710Z 2025-12-04T08:50:13.2379843Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:50:13.2380242Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:50:13.2380587Z res = mod(**inputs) 2025-12-04T08:50:13.2381008Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2025-12-04T08:50:13.2381470Z dlbrt_output = self.distilbert( 2025-12-04T08:50:13.2381917Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 724, in forward 2025-12-04T08:50:13.2382376Z return self.transformer( 2025-12-04T08:50:13.2382817Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-12-04T08:50:13.2383272Z layer_outputs = layer_module( 2025-12-04T08:50:13.2383642Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:50:13.2384040Z return super().__call__(*args, **kwargs) 2025-12-04T08:50:13.2384506Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 484, in forward 2025-12-04T08:50:13.2384995Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-12-04T08:50:13.2385485Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 418, in forward 2025-12-04T08:50:13.2386073Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-12-04T08:50:13.2386636Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:50:13.2387080Z return forward_fn(*input_tensors) 2025-12-04T08:50:13.2387534Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 423, in ff_chunk 2025-12-04T08:50:13.2387991Z x = self.lin2(x) 2025-12-04T08:50:13.2388102Z 2025-12-04T08:50:13.2388231Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:50:13.2388627Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:50:13.2388993Z res = mod(**inputs) 2025-12-04T08:50:13.2389418Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2025-12-04T08:50:13.2389867Z dlbrt_output = self.distilbert( 2025-12-04T08:50:13.2390318Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 724, in forward 2025-12-04T08:50:13.2390762Z return self.transformer( 2025-12-04T08:50:13.2391198Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-12-04T08:50:13.2391641Z layer_outputs = layer_module( 2025-12-04T08:50:13.2392071Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:50:13.2392470Z return super().__call__(*args, **kwargs) 2025-12-04T08:50:13.2392931Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 466, in forward 2025-12-04T08:50:13.2393396Z sa_output = self.attention( 2025-12-04T08:50:13.2393840Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 388, in forward 2025-12-04T08:50:13.2394357Z q = shape(self.q_lin(query)) # (bs, n_heads, q_length, dim_per_head) 2025-12-04T08:50:13.2394560Z 2025-12-04T08:50:13.2394684Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:50:13.2395084Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:50:13.2395460Z res = mod(**inputs) 2025-12-04T08:50:13.2395909Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2025-12-04T08:50:13.2396378Z dlbrt_output = self.distilbert( 2025-12-04T08:50:13.2396854Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 724, in forward 2025-12-04T08:50:13.2397321Z return self.transformer( 2025-12-04T08:50:13.2397773Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-12-04T08:50:13.2398325Z layer_outputs = layer_module( 2025-12-04T08:50:13.2398727Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:50:13.2399145Z return super().__call__(*args, **kwargs) 2025-12-04T08:50:13.2399627Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 466, in forward 2025-12-04T08:50:13.2400077Z sa_output = self.attention( 2025-12-04T08:50:13.2400518Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 389, in forward 2025-12-04T08:50:13.2401032Z k = shape(self.k_lin(key)) # (bs, n_heads, k_length, dim_per_head) 2025-12-04T08:50:13.2401224Z 2025-12-04T08:50:13.2401336Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:50:13.2401732Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:50:13.2402083Z res = mod(**inputs) 2025-12-04T08:50:13.2402521Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2025-12-04T08:50:13.2402980Z dlbrt_output = self.distilbert( 2025-12-04T08:50:13.2403427Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 724, in forward 2025-12-04T08:50:13.2403875Z return self.transformer( 2025-12-04T08:50:13.2404311Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-12-04T08:50:13.2404749Z layer_outputs = layer_module( 2025-12-04T08:50:13.2405128Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:50:13.2405531Z return super().__call__(*args, **kwargs) 2025-12-04T08:50:13.2405993Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 466, in forward 2025-12-04T08:50:13.2406430Z sa_output = self.attention( 2025-12-04T08:50:13.2406864Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 390, in forward 2025-12-04T08:50:13.2407365Z v = shape(self.v_lin(value)) # (bs, n_heads, k_length, dim_per_head) 2025-12-04T08:50:13.2407562Z 2025-12-04T08:50:13.2407697Z cudagraph partition due to non gpu ops 2025-12-04T08:50:13.2407958Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:50:13.2408357Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:50:13.2408755Z res = mod(**inputs) 2025-12-04T08:50:13.2409168Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2025-12-04T08:50:13.2409619Z dlbrt_output = self.distilbert( 2025-12-04T08:50:13.2410063Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 724, in forward 2025-12-04T08:50:13.2410511Z return self.transformer( 2025-12-04T08:50:13.2410935Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-12-04T08:50:13.2411411Z layer_outputs = layer_module( 2025-12-04T08:50:13.2411796Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:50:13.2412186Z return super().__call__(*args, **kwargs) 2025-12-04T08:50:13.2412642Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 466, in forward 2025-12-04T08:50:13.2413094Z sa_output = self.attention( 2025-12-04T08:50:13.2413545Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 392, in forward 2025-12-04T08:50:13.2414082Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T08:50:13.2414290Z 2025-12-04T08:50:13.2414403Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:50:13.2414797Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:50:13.2415162Z res = mod(**inputs) 2025-12-04T08:50:13.2415580Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2025-12-04T08:50:13.2416050Z dlbrt_output = self.distilbert( 2025-12-04T08:50:13.2416566Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 724, in forward 2025-12-04T08:50:13.2417023Z return self.transformer( 2025-12-04T08:50:13.2417468Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-12-04T08:50:13.2417926Z layer_outputs = layer_module( 2025-12-04T08:50:13.2418309Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:50:13.2418704Z return super().__call__(*args, **kwargs) 2025-12-04T08:50:13.2419170Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 466, in forward 2025-12-04T08:50:13.2419628Z sa_output = self.attention( 2025-12-04T08:50:13.2420065Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 402, in forward 2025-12-04T08:50:13.2420520Z attn_output = self.out_lin(attn_output) 2025-12-04T08:50:13.2420680Z 2025-12-04T08:50:13.2420921Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:50:13.2421320Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:50:13.2421667Z res = mod(**inputs) 2025-12-04T08:50:13.2422090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2025-12-04T08:50:13.2422542Z dlbrt_output = self.distilbert( 2025-12-04T08:50:13.2422994Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 724, in forward 2025-12-04T08:50:13.2423444Z return self.transformer( 2025-12-04T08:50:13.2423999Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-12-04T08:50:13.2424457Z layer_outputs = layer_module( 2025-12-04T08:50:13.2424870Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:50:13.2425268Z return super().__call__(*args, **kwargs) 2025-12-04T08:50:13.2425732Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 484, in forward 2025-12-04T08:50:13.2426236Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-12-04T08:50:13.2426721Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 418, in forward 2025-12-04T08:50:13.2427340Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-12-04T08:50:13.2427912Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:50:13.2428347Z return forward_fn(*input_tensors) 2025-12-04T08:50:13.2428798Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 421, in ff_chunk 2025-12-04T08:50:13.2429243Z x = self.lin1(input) 2025-12-04T08:50:13.2429365Z 2025-12-04T08:50:13.2429485Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:50:13.2429877Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:50:13.2430222Z res = mod(**inputs) 2025-12-04T08:50:13.2430640Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2025-12-04T08:50:13.2431091Z dlbrt_output = self.distilbert( 2025-12-04T08:50:13.2431531Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 724, in forward 2025-12-04T08:50:13.2431977Z return self.transformer( 2025-12-04T08:50:13.2432413Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-12-04T08:50:13.2432860Z layer_outputs = layer_module( 2025-12-04T08:50:13.2433231Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:50:13.2433628Z return super().__call__(*args, **kwargs) 2025-12-04T08:50:13.2434078Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 484, in forward 2025-12-04T08:50:13.2434572Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-12-04T08:50:13.2435077Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 418, in forward 2025-12-04T08:50:13.2435683Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-12-04T08:50:13.2436267Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:50:13.2436710Z return forward_fn(*input_tensors) 2025-12-04T08:50:13.2437174Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 422, in ff_chunk 2025-12-04T08:50:13.2437647Z x = self.activation(x) 2025-12-04T08:50:13.2438019Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T08:50:13.2438469Z return self.act(input) 2025-12-04T08:50:13.2438605Z 2025-12-04T08:50:13.2438723Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:50:13.2439176Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:50:13.2439535Z res = mod(**inputs) 2025-12-04T08:50:13.2439970Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2025-12-04T08:50:13.2440462Z dlbrt_output = self.distilbert( 2025-12-04T08:50:13.2440922Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 724, in forward 2025-12-04T08:50:13.2441377Z return self.transformer( 2025-12-04T08:50:13.2441827Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 531, in forward 2025-12-04T08:50:13.2443016Z layer_outputs = layer_module( 2025-12-04T08:50:13.2443412Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:50:13.2443854Z return super().__call__(*args, **kwargs) 2025-12-04T08:50:13.2444338Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 484, in forward 2025-12-04T08:50:13.2444856Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-12-04T08:50:13.2445372Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 418, in forward 2025-12-04T08:50:13.2445998Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-12-04T08:50:13.2446602Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:50:13.2447061Z return forward_fn(*input_tensors) 2025-12-04T08:50:13.2447526Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 423, in ff_chunk 2025-12-04T08:50:13.2447994Z x = self.lin2(x) 2025-12-04T08:50:13.2448116Z 2025-12-04T08:50:13.2448239Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:50:13.2448656Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:50:13.2449231Z res = mod(**inputs) 2025-12-04T08:50:13.2449727Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 824, in forward 2025-12-04T08:50:13.2450291Z prediction_logits = self.vocab_transform(hidden_states) # (bs, seq_length, dim) 2025-12-04T08:50:13.2450541Z 2025-12-04T08:50:13.2450668Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:50:13.2451063Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:50:13.2451423Z res = mod(**inputs) 2025-12-04T08:50:13.2451853Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 827, in forward 2025-12-04T08:50:13.2452434Z prediction_logits = self.vocab_projector(prediction_logits) # (bs, seq_length, vocab_size) 2025-12-04T08:50:13.2452708Z 2025-12-04T08:50:13.2452818Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:50:13.2453207Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:50:13.2453556Z res = mod(**inputs) 2025-12-04T08:50:13.2453963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 831, in forward 2025-12-04T08:50:13.2454549Z mlm_loss = self.mlm_loss_fct(prediction_logits.view(-1, prediction_logits.size(-1)), labels.view(-1)) 2025-12-04T08:50:13.2454814Z 2025-12-04T08:50:22.6153479Z Compilation time (from dynamo_timed): 13.718644428 2025-12-04T08:50:22.6153793Z pass 2025-12-04T08:50:22.6154339Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T08:50:22.6155629Z TIMING: _recursive_pre_grad_passes:0.00569 _recursive_joint_graph_passes:0.27047 _recursive_post_grad_passes:0.04794 async_compile.wait:0.84318 code_gen:9.39958 inductor_compile:10.15599 backend_compile:12.01761 gc:0.00098 entire_frame_compile:13.71864 total_wall_time:13.71864 2025-12-04T08:50:22.6156812Z STATS: call_* op count: 153 | FakeTensorMode.__torch_dispatch__:3980 | FakeTensor.__torch_dispatch__:2344 | ProxyTorchDispatchMode.__torch_dispatch__:1053 2025-12-04T08:50:22.6157364Z Dynamo produced 1 graphs covering 153 ops with 0 graph breaks (0 unique) 2025-12-04T08:50:25.0098908Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/__init__.py:65: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-12-04T08:50:25.0100732Z import pynvml # type: ignore[import] 2025-12-04T08:50:28.5507790Z 2025-12-04T08:50:29.6996055Z loading model: 0it [00:00, ?it/s]`loss_type=None` was set in the config but it is unrecognized. Using the default loss: `ForCausalLMLoss`. 2025-12-04T08:50:29.6996858Z WARNING:transformers.modeling_utils:`loss_type=None` was set in the config but it is unrecognized. Using the default loss: `ForCausalLMLoss`. 2025-12-04T08:50:29.7410533Z 2025-12-04T08:50:29.7413270Z loading model: 0it [00:01, ?it/s] 2025-12-04T08:50:29.7419234Z cpu eval DistillGPT2 2025-12-04T08:50:30.8311390Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T08:50:31.3529461Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T08:50:31.8831004Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T08:50:38.7681517Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:50:38.7682150Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1068, in forward 2025-12-04T08:50:38.7682692Z transformer_outputs = self.transformer( 2025-12-04T08:50:38.7683165Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 873, in forward 2025-12-04T08:50:38.7683638Z causal_mask = create_causal_mask( 2025-12-04T08:50:38.7684086Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/masking_utils.py", line 788, in create_causal_mask 2025-12-04T08:50:38.7684682Z early_exit, attention_mask, packed_sequence_mask, kv_length, kv_offset = _preprocess_mask_arguments( 2025-12-04T08:50:38.7685274Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/masking_utils.py", line 740, in _preprocess_mask_arguments 2025-12-04T08:50:38.7685801Z packed_sequence_mask = find_packed_sequence_indices(position_ids) 2025-12-04T08:50:38.7686382Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/masking_utils.py", line 655, in find_packed_sequence_indices 2025-12-04T08:50:38.7686991Z first_dummy_value = position_ids[:, :1] - 1 # We just need the diff on this first value to be 1 2025-12-04T08:50:38.7687261Z 2025-12-04T08:50:38.7687359Z cudagraph partition due to non gpu ops 2025-12-04T08:50:38.7687643Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:50:38.7688141Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1068, in forward 2025-12-04T08:50:38.7688617Z transformer_outputs = self.transformer( 2025-12-04T08:50:38.7689056Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T08:50:38.7689492Z outputs = block( 2025-12-04T08:50:38.7689869Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:50:38.7690288Z return super().__call__(*args, **kwargs) 2025-12-04T08:50:38.7691053Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:50:38.7691493Z return func(*args, **kwargs) 2025-12-04T08:50:38.7691934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T08:50:38.7692475Z attn_output, self_attn_weights = self.attn( 2025-12-04T08:50:38.7692930Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:50:38.7693367Z return func(*args, **kwargs) 2025-12-04T08:50:38.7693810Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 313, in forward 2025-12-04T08:50:38.7694428Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-12-04T08:50:38.7695021Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T08:50:38.7695493Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T08:50:38.7695697Z 2025-12-04T08:50:38.7695826Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:50:38.7696307Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1068, in forward 2025-12-04T08:50:38.7696769Z transformer_outputs = self.transformer( 2025-12-04T08:50:38.7697210Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 873, in forward 2025-12-04T08:50:38.7697654Z causal_mask = create_causal_mask( 2025-12-04T08:50:38.7698084Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/masking_utils.py", line 788, in create_causal_mask 2025-12-04T08:50:38.7698657Z early_exit, attention_mask, packed_sequence_mask, kv_length, kv_offset = _preprocess_mask_arguments( 2025-12-04T08:50:38.7699260Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/masking_utils.py", line 740, in _preprocess_mask_arguments 2025-12-04T08:50:38.7699767Z packed_sequence_mask = find_packed_sequence_indices(position_ids) 2025-12-04T08:50:38.7700289Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/masking_utils.py", line 656, in find_packed_sequence_indices 2025-12-04T08:50:38.7700829Z position_diff = torch.diff(position_ids, prepend=first_dummy_value, dim=-1) 2025-12-04T08:50:38.7701064Z 2025-12-04T08:50:38.7701165Z cudagraph partition due to non gpu ops 2025-12-04T08:50:38.7701432Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:50:38.7701908Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1068, in forward 2025-12-04T08:50:38.7702393Z transformer_outputs = self.transformer( 2025-12-04T08:50:38.7702841Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 873, in forward 2025-12-04T08:50:38.7703284Z causal_mask = create_causal_mask( 2025-12-04T08:50:38.7703706Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/masking_utils.py", line 788, in create_causal_mask 2025-12-04T08:50:38.7704285Z early_exit, attention_mask, packed_sequence_mask, kv_length, kv_offset = _preprocess_mask_arguments( 2025-12-04T08:50:38.7704867Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/masking_utils.py", line 740, in _preprocess_mask_arguments 2025-12-04T08:50:38.7705374Z packed_sequence_mask = find_packed_sequence_indices(position_ids) 2025-12-04T08:50:38.7705885Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/masking_utils.py", line 657, in find_packed_sequence_indices 2025-12-04T08:50:38.7706388Z packed_sequence_mask = (position_diff != 1).cumsum(-1) 2025-12-04T08:50:38.7706577Z 2025-12-04T08:50:38.7706695Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:50:38.7707226Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1068, in forward 2025-12-04T08:50:38.7707681Z transformer_outputs = self.transformer( 2025-12-04T08:50:38.7708127Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 873, in forward 2025-12-04T08:50:38.7708596Z causal_mask = create_causal_mask( 2025-12-04T08:50:38.7709018Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/masking_utils.py", line 788, in create_causal_mask 2025-12-04T08:50:38.7709589Z early_exit, attention_mask, packed_sequence_mask, kv_length, kv_offset = _preprocess_mask_arguments( 2025-12-04T08:50:38.7710383Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/masking_utils.py", line 740, in _preprocess_mask_arguments 2025-12-04T08:50:38.7710923Z packed_sequence_mask = find_packed_sequence_indices(position_ids) 2025-12-04T08:50:38.7711539Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/masking_utils.py", line 657, in find_packed_sequence_indices 2025-12-04T08:50:38.7712031Z packed_sequence_mask = (position_diff != 1).cumsum(-1) 2025-12-04T08:50:38.7712215Z 2025-12-04T08:50:38.7712307Z cudagraph partition due to non gpu ops 2025-12-04T08:50:38.7712575Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:50:38.7713047Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1068, in forward 2025-12-04T08:50:38.7713502Z transformer_outputs = self.transformer( 2025-12-04T08:50:38.7713933Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T08:50:38.7714369Z outputs = block( 2025-12-04T08:50:38.7714742Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:50:38.7715164Z return super().__call__(*args, **kwargs) 2025-12-04T08:50:38.7715593Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:50:38.7716032Z return func(*args, **kwargs) 2025-12-04T08:50:38.7716457Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T08:50:38.7716911Z attn_output, self_attn_weights = self.attn( 2025-12-04T08:50:38.7717353Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:50:38.7717788Z return func(*args, **kwargs) 2025-12-04T08:50:38.7718499Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 345, in forward 2025-12-04T08:50:38.7718976Z attn_output, attn_weights = attention_interface( 2025-12-04T08:50:38.7719494Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T08:50:38.7720046Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T08:50:38.7720253Z 2025-12-04T08:50:38.7720379Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:50:38.7721039Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1068, in forward 2025-12-04T08:50:38.7721500Z transformer_outputs = self.transformer( 2025-12-04T08:50:38.7722059Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T08:50:38.7722485Z outputs = block( 2025-12-04T08:50:38.7722860Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:50:38.7723384Z return super().__call__(*args, **kwargs) 2025-12-04T08:50:38.7723914Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:50:38.7724335Z return func(*args, **kwargs) 2025-12-04T08:50:38.7724758Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T08:50:38.7725261Z attn_output, self_attn_weights = self.attn( 2025-12-04T08:50:38.7725693Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:50:38.7726119Z return func(*args, **kwargs) 2025-12-04T08:50:38.7726540Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 358, in forward 2025-12-04T08:50:38.7726986Z attn_output = self.c_proj(attn_output) 2025-12-04T08:50:38.7727390Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T08:50:38.7727882Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T08:50:38.7728083Z 2025-12-04T08:50:38.7728210Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:50:38.7728683Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1068, in forward 2025-12-04T08:50:38.7729129Z transformer_outputs = self.transformer( 2025-12-04T08:50:38.7729568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T08:50:38.7729993Z outputs = block( 2025-12-04T08:50:38.7730353Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:50:38.7730764Z return super().__call__(*args, **kwargs) 2025-12-04T08:50:38.7731192Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:50:38.7731733Z return func(*args, **kwargs) 2025-12-04T08:50:38.7732149Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T08:50:38.7732615Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T08:50:38.7733074Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 374, in forward 2025-12-04T08:50:38.7733519Z hidden_states = self.c_fc(hidden_states) 2025-12-04T08:50:38.7733915Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T08:50:38.7734369Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T08:50:38.7734564Z 2025-12-04T08:50:38.7734715Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:50:38.7735179Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1068, in forward 2025-12-04T08:50:38.7735634Z transformer_outputs = self.transformer( 2025-12-04T08:50:38.7736091Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T08:50:38.7736505Z outputs = block( 2025-12-04T08:50:38.7736870Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:50:38.7737284Z return super().__call__(*args, **kwargs) 2025-12-04T08:50:38.7737708Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:50:38.7738131Z return func(*args, **kwargs) 2025-12-04T08:50:38.7738544Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T08:50:38.7739019Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T08:50:38.7739465Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 375, in forward 2025-12-04T08:50:38.7739951Z hidden_states = self.act(hidden_states) 2025-12-04T08:50:38.7740355Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 62, in forward 2025-12-04T08:50:38.7740878Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-12-04T08:50:38.7741188Z 2025-12-04T08:50:38.7741306Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:50:38.7741783Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1068, in forward 2025-12-04T08:50:38.7742231Z transformer_outputs = self.transformer( 2025-12-04T08:50:38.7742669Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T08:50:38.7743080Z outputs = block( 2025-12-04T08:50:38.7743467Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:50:38.7743882Z return super().__call__(*args, **kwargs) 2025-12-04T08:50:38.7744307Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:50:38.7744720Z return func(*args, **kwargs) 2025-12-04T08:50:38.7745135Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T08:50:38.7745600Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T08:50:38.7746049Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 376, in forward 2025-12-04T08:50:38.7746494Z hidden_states = self.c_proj(hidden_states) 2025-12-04T08:50:38.7746900Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T08:50:38.7747344Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T08:50:38.7747540Z 2025-12-04T08:50:38.7747788Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:50:38.7748244Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1068, in forward 2025-12-04T08:50:38.7748675Z transformer_outputs = self.transformer( 2025-12-04T08:50:38.7749098Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T08:50:38.7749502Z outputs = block( 2025-12-04T08:50:38.7749855Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:50:38.7750249Z return super().__call__(*args, **kwargs) 2025-12-04T08:50:38.7750669Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:50:38.7751087Z return func(*args, **kwargs) 2025-12-04T08:50:38.7751495Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T08:50:38.7751942Z attn_output, self_attn_weights = self.attn( 2025-12-04T08:50:38.7752373Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:50:38.7752805Z return func(*args, **kwargs) 2025-12-04T08:50:38.7753208Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 313, in forward 2025-12-04T08:50:38.7753759Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-12-04T08:50:38.7754263Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T08:50:38.7754716Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T08:50:38.7754910Z 2025-12-04T08:50:38.7755009Z cudagraph partition due to non gpu ops 2025-12-04T08:50:38.7755310Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:50:38.7755799Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1068, in forward 2025-12-04T08:50:38.7756270Z transformer_outputs = self.transformer( 2025-12-04T08:50:38.7756706Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T08:50:38.7757128Z outputs = block( 2025-12-04T08:50:38.7757492Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:50:38.7757901Z return super().__call__(*args, **kwargs) 2025-12-04T08:50:38.7758389Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:50:38.7758849Z return func(*args, **kwargs) 2025-12-04T08:50:38.7759278Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T08:50:38.7759736Z attn_output, self_attn_weights = self.attn( 2025-12-04T08:50:38.7760177Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:50:38.7760609Z return func(*args, **kwargs) 2025-12-04T08:50:38.7761056Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 345, in forward 2025-12-04T08:50:38.7761526Z attn_output, attn_weights = attention_interface( 2025-12-04T08:50:38.7762033Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T08:50:38.7762587Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T08:50:38.7762798Z 2025-12-04T08:50:38.7762928Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:50:38.7763403Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1068, in forward 2025-12-04T08:50:38.7763856Z transformer_outputs = self.transformer( 2025-12-04T08:50:38.7764299Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T08:50:38.7764729Z outputs = block( 2025-12-04T08:50:38.7765091Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:50:38.7765507Z return super().__call__(*args, **kwargs) 2025-12-04T08:50:38.7765943Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:50:38.7766367Z return func(*args, **kwargs) 2025-12-04T08:50:38.7766783Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T08:50:38.7767244Z attn_output, self_attn_weights = self.attn( 2025-12-04T08:50:38.7767688Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:50:38.7768107Z return func(*args, **kwargs) 2025-12-04T08:50:38.7768533Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 358, in forward 2025-12-04T08:50:38.7768981Z attn_output = self.c_proj(attn_output) 2025-12-04T08:50:38.7769392Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T08:50:38.7769841Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T08:50:38.7770044Z 2025-12-04T08:50:38.7770167Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:50:38.7770645Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1068, in forward 2025-12-04T08:50:38.7771147Z transformer_outputs = self.transformer( 2025-12-04T08:50:38.7771581Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T08:50:38.7772008Z outputs = block( 2025-12-04T08:50:38.7772401Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:50:38.7772809Z return super().__call__(*args, **kwargs) 2025-12-04T08:50:38.7773235Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:50:38.7773664Z return func(*args, **kwargs) 2025-12-04T08:50:38.7774077Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T08:50:38.7774550Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T08:50:38.7775055Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 374, in forward 2025-12-04T08:50:38.7775494Z hidden_states = self.c_fc(hidden_states) 2025-12-04T08:50:38.7775889Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T08:50:38.7776336Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T08:50:38.7776536Z 2025-12-04T08:50:38.7776655Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:50:38.7777136Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1068, in forward 2025-12-04T08:50:38.7777593Z transformer_outputs = self.transformer( 2025-12-04T08:50:38.7778065Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T08:50:38.7778503Z outputs = block( 2025-12-04T08:50:38.7778878Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:50:38.7779282Z return super().__call__(*args, **kwargs) 2025-12-04T08:50:38.7779710Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:50:38.7780137Z return func(*args, **kwargs) 2025-12-04T08:50:38.7780546Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T08:50:38.7781024Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T08:50:38.7781481Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 375, in forward 2025-12-04T08:50:38.7781930Z hidden_states = self.act(hidden_states) 2025-12-04T08:50:38.7782319Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 62, in forward 2025-12-04T08:50:38.7782844Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-12-04T08:50:38.7783116Z 2025-12-04T08:50:38.7783232Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:50:38.7783705Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1068, in forward 2025-12-04T08:50:38.7784148Z transformer_outputs = self.transformer( 2025-12-04T08:50:38.7784588Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T08:50:38.7785006Z outputs = block( 2025-12-04T08:50:38.7785362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:50:38.7785766Z return super().__call__(*args, **kwargs) 2025-12-04T08:50:38.7786191Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:50:38.7786613Z return func(*args, **kwargs) 2025-12-04T08:50:38.7787067Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T08:50:38.7787533Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T08:50:38.7788015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 376, in forward 2025-12-04T08:50:38.7788457Z hidden_states = self.c_proj(hidden_states) 2025-12-04T08:50:38.7788855Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T08:50:38.7789303Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T08:50:38.7789493Z 2025-12-04T08:50:38.7789619Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:50:38.7790081Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1068, in forward 2025-12-04T08:50:38.7790559Z transformer_outputs = self.transformer( 2025-12-04T08:50:38.7790999Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T08:50:38.7791423Z outputs = block( 2025-12-04T08:50:38.7791784Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:50:38.7792192Z return super().__call__(*args, **kwargs) 2025-12-04T08:50:38.7792624Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:50:38.7793057Z return func(*args, **kwargs) 2025-12-04T08:50:38.7793465Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T08:50:38.7793915Z attn_output, self_attn_weights = self.attn( 2025-12-04T08:50:38.7794354Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:50:38.7794766Z return func(*args, **kwargs) 2025-12-04T08:50:38.7795181Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 313, in forward 2025-12-04T08:50:38.7795735Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-12-04T08:50:38.7796255Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T08:50:38.7796694Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T08:50:38.7796891Z 2025-12-04T08:50:38.7796985Z cudagraph partition due to non gpu ops 2025-12-04T08:50:38.7797258Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:50:38.7797743Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1068, in forward 2025-12-04T08:50:38.7798274Z transformer_outputs = self.transformer( 2025-12-04T08:50:38.7798731Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T08:50:38.7799152Z outputs = block( 2025-12-04T08:50:38.7799511Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:50:38.7799921Z return super().__call__(*args, **kwargs) 2025-12-04T08:50:38.7800346Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:50:38.7800765Z return func(*args, **kwargs) 2025-12-04T08:50:38.7801175Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T08:50:38.7801624Z attn_output, self_attn_weights = self.attn( 2025-12-04T08:50:38.7802068Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:50:38.7802557Z return func(*args, **kwargs) 2025-12-04T08:50:38.7802974Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 345, in forward 2025-12-04T08:50:38.7804019Z attn_output, attn_weights = attention_interface( 2025-12-04T08:50:38.7804526Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T08:50:38.7805066Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T08:50:38.7805279Z 2025-12-04T08:50:38.7805395Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:50:38.7805864Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1068, in forward 2025-12-04T08:50:38.7806306Z transformer_outputs = self.transformer( 2025-12-04T08:50:38.7806770Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T08:50:38.7807200Z outputs = block( 2025-12-04T08:50:38.7807572Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:50:38.7807986Z return super().__call__(*args, **kwargs) 2025-12-04T08:50:38.7808420Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:50:38.7808852Z return func(*args, **kwargs) 2025-12-04T08:50:38.7809270Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T08:50:38.7809716Z attn_output, self_attn_weights = self.attn( 2025-12-04T08:50:38.7810159Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:50:38.7810590Z return func(*args, **kwargs) 2025-12-04T08:50:38.7811006Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 358, in forward 2025-12-04T08:50:38.7811442Z attn_output = self.c_proj(attn_output) 2025-12-04T08:50:38.7811847Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T08:50:38.7812302Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T08:50:38.7812498Z 2025-12-04T08:50:38.7812618Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:50:38.7813113Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1068, in forward 2025-12-04T08:50:38.7813571Z transformer_outputs = self.transformer( 2025-12-04T08:50:38.7814009Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T08:50:38.7814428Z outputs = block( 2025-12-04T08:50:38.7814800Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:50:38.7815220Z return super().__call__(*args, **kwargs) 2025-12-04T08:50:38.7815649Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:50:38.7816097Z return func(*args, **kwargs) 2025-12-04T08:50:38.7816518Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T08:50:38.7816996Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T08:50:38.7817460Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 374, in forward 2025-12-04T08:50:38.7817918Z hidden_states = self.c_fc(hidden_states) 2025-12-04T08:50:38.7818328Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T08:50:38.7818835Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T08:50:38.7819032Z 2025-12-04T08:50:38.7819149Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:50:38.7819639Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1068, in forward 2025-12-04T08:50:38.7820113Z transformer_outputs = self.transformer( 2025-12-04T08:50:38.7820546Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T08:50:38.7821159Z outputs = block( 2025-12-04T08:50:38.7821525Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:50:38.7821936Z return super().__call__(*args, **kwargs) 2025-12-04T08:50:38.7822360Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:50:38.7822865Z return func(*args, **kwargs) 2025-12-04T08:50:38.7823278Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T08:50:38.7823738Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T08:50:38.7824189Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 375, in forward 2025-12-04T08:50:38.7824623Z hidden_states = self.act(hidden_states) 2025-12-04T08:50:38.7825018Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 62, in forward 2025-12-04T08:50:38.7825526Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-12-04T08:50:38.7825797Z 2025-12-04T08:50:38.7825915Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:50:38.7826388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1068, in forward 2025-12-04T08:50:38.7826833Z transformer_outputs = self.transformer( 2025-12-04T08:50:38.7827260Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T08:50:38.7827682Z outputs = block( 2025-12-04T08:50:38.7828043Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:50:38.7828447Z return super().__call__(*args, **kwargs) 2025-12-04T08:50:38.7828864Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:50:38.7829284Z return func(*args, **kwargs) 2025-12-04T08:50:38.7829696Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T08:50:38.7830154Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T08:50:38.7830611Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 376, in forward 2025-12-04T08:50:38.7831058Z hidden_states = self.c_proj(hidden_states) 2025-12-04T08:50:38.7831463Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T08:50:38.7831904Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T08:50:38.7832103Z 2025-12-04T08:50:38.7832219Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:50:38.7832691Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1068, in forward 2025-12-04T08:50:38.7833140Z transformer_outputs = self.transformer( 2025-12-04T08:50:38.7833570Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T08:50:38.7833990Z outputs = block( 2025-12-04T08:50:38.7834424Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:50:38.7834829Z return super().__call__(*args, **kwargs) 2025-12-04T08:50:38.7835262Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:50:38.7835751Z return func(*args, **kwargs) 2025-12-04T08:50:38.7836170Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 451, in forward 2025-12-04T08:50:38.7836646Z hidden_states = residual + feed_forward_hidden_states 2025-12-04T08:50:38.7836835Z 2025-12-04T08:50:38.7836950Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:50:38.7837416Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1068, in forward 2025-12-04T08:50:38.7837885Z transformer_outputs = self.transformer( 2025-12-04T08:50:38.7838403Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T08:50:38.7838834Z outputs = block( 2025-12-04T08:50:38.7839197Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:50:38.7839601Z return super().__call__(*args, **kwargs) 2025-12-04T08:50:38.7840034Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:50:38.7840466Z return func(*args, **kwargs) 2025-12-04T08:50:38.7840887Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T08:50:38.7841327Z attn_output, self_attn_weights = self.attn( 2025-12-04T08:50:38.7841765Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:50:38.7842190Z return func(*args, **kwargs) 2025-12-04T08:50:38.7842603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 313, in forward 2025-12-04T08:50:38.7843177Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-12-04T08:50:38.7843703Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T08:50:38.7844160Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T08:50:38.7844351Z 2025-12-04T08:50:38.7844443Z cudagraph partition due to non gpu ops 2025-12-04T08:50:38.7844714Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:50:38.7845192Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1068, in forward 2025-12-04T08:50:38.7845646Z transformer_outputs = self.transformer( 2025-12-04T08:50:38.7846095Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T08:50:38.7846524Z outputs = block( 2025-12-04T08:50:38.7846890Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:50:38.7847304Z return super().__call__(*args, **kwargs) 2025-12-04T08:50:38.7847733Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:50:38.7848156Z return func(*args, **kwargs) 2025-12-04T08:50:38.7848582Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T08:50:38.7849034Z attn_output, self_attn_weights = self.attn( 2025-12-04T08:50:38.7849471Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:50:38.7850063Z return func(*args, **kwargs) 2025-12-04T08:50:38.7850547Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 345, in forward 2025-12-04T08:50:38.7851019Z attn_output, attn_weights = attention_interface( 2025-12-04T08:50:38.7851528Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T08:50:38.7852097Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T08:50:38.7852315Z 2025-12-04T08:50:38.7852434Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:50:38.7852915Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1068, in forward 2025-12-04T08:50:38.7853351Z transformer_outputs = self.transformer( 2025-12-04T08:50:38.7853776Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T08:50:38.7854202Z outputs = block( 2025-12-04T08:50:38.7854559Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:50:38.7854957Z return super().__call__(*args, **kwargs) 2025-12-04T08:50:38.7855367Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:50:38.7855776Z return func(*args, **kwargs) 2025-12-04T08:50:38.7856188Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T08:50:38.7856645Z attn_output, self_attn_weights = self.attn( 2025-12-04T08:50:38.7857075Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:50:38.7857510Z return func(*args, **kwargs) 2025-12-04T08:50:38.7857914Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 358, in forward 2025-12-04T08:50:38.7858353Z attn_output = self.c_proj(attn_output) 2025-12-04T08:50:38.7858737Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T08:50:38.7859169Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T08:50:38.7859359Z 2025-12-04T08:50:38.7859482Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:50:38.7859930Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1068, in forward 2025-12-04T08:50:38.7860375Z transformer_outputs = self.transformer( 2025-12-04T08:50:38.7860796Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T08:50:38.7861212Z outputs = block( 2025-12-04T08:50:38.7861559Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:50:38.7861979Z return super().__call__(*args, **kwargs) 2025-12-04T08:50:38.7862404Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:50:38.7862830Z return func(*args, **kwargs) 2025-12-04T08:50:38.7863262Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T08:50:38.7863716Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T08:50:38.7864157Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 374, in forward 2025-12-04T08:50:38.7864581Z hidden_states = self.c_fc(hidden_states) 2025-12-04T08:50:38.7864981Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T08:50:38.7865424Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T08:50:38.7865624Z 2025-12-04T08:50:38.7865815Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:50:38.7866270Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1068, in forward 2025-12-04T08:50:38.7866735Z transformer_outputs = self.transformer( 2025-12-04T08:50:38.7867174Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T08:50:38.7867596Z outputs = block( 2025-12-04T08:50:38.7867956Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:50:38.7868367Z return super().__call__(*args, **kwargs) 2025-12-04T08:50:38.7868794Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:50:38.7869240Z return func(*args, **kwargs) 2025-12-04T08:50:38.7869664Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T08:50:38.7870132Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T08:50:38.7870592Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 375, in forward 2025-12-04T08:50:38.7871028Z hidden_states = self.act(hidden_states) 2025-12-04T08:50:38.7871426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 62, in forward 2025-12-04T08:50:38.7871945Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-12-04T08:50:38.7872210Z 2025-12-04T08:50:38.7872336Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:50:38.7872803Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1068, in forward 2025-12-04T08:50:38.7873256Z transformer_outputs = self.transformer( 2025-12-04T08:50:38.7873699Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T08:50:38.7874116Z outputs = block( 2025-12-04T08:50:38.7874485Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:50:38.7874900Z return super().__call__(*args, **kwargs) 2025-12-04T08:50:38.7875327Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:50:38.7875744Z return func(*args, **kwargs) 2025-12-04T08:50:38.7876159Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T08:50:38.7876642Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T08:50:38.7877092Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 376, in forward 2025-12-04T08:50:38.7877549Z hidden_states = self.c_proj(hidden_states) 2025-12-04T08:50:38.7877958Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T08:50:38.7878484Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T08:50:38.7878682Z 2025-12-04T08:50:38.7878801Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:50:38.7879269Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1068, in forward 2025-12-04T08:50:38.7879725Z transformer_outputs = self.transformer( 2025-12-04T08:50:38.7880161Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T08:50:38.7880586Z outputs = block( 2025-12-04T08:50:38.7880957Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:50:38.7881421Z return super().__call__(*args, **kwargs) 2025-12-04T08:50:38.7881847Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:50:38.7882298Z return func(*args, **kwargs) 2025-12-04T08:50:38.7882715Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T08:50:38.7883173Z attn_output, self_attn_weights = self.attn( 2025-12-04T08:50:38.7883601Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:50:38.7884018Z return func(*args, **kwargs) 2025-12-04T08:50:38.7884431Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 313, in forward 2025-12-04T08:50:38.7885018Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-12-04T08:50:38.7885537Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T08:50:38.7885988Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T08:50:38.7886181Z 2025-12-04T08:50:38.7886283Z cudagraph partition due to non gpu ops 2025-12-04T08:50:38.7886546Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:50:38.7887017Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1068, in forward 2025-12-04T08:50:38.7887466Z transformer_outputs = self.transformer( 2025-12-04T08:50:38.7887909Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T08:50:38.7888323Z outputs = block( 2025-12-04T08:50:38.7888688Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:50:38.7889102Z return super().__call__(*args, **kwargs) 2025-12-04T08:50:38.7889525Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:50:38.7889947Z return func(*args, **kwargs) 2025-12-04T08:50:38.7890366Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T08:50:38.7890809Z attn_output, self_attn_weights = self.attn( 2025-12-04T08:50:38.7891239Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:50:38.7891660Z return func(*args, **kwargs) 2025-12-04T08:50:38.7892078Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 345, in forward 2025-12-04T08:50:38.7892539Z attn_output, attn_weights = attention_interface( 2025-12-04T08:50:38.7893046Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T08:50:38.7893596Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T08:50:38.7893801Z 2025-12-04T08:50:38.7893929Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:50:38.7894424Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1068, in forward 2025-12-04T08:50:38.7894874Z transformer_outputs = self.transformer( 2025-12-04T08:50:38.7895317Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T08:50:38.7895744Z outputs = block( 2025-12-04T08:50:38.7896102Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:50:38.7896517Z return super().__call__(*args, **kwargs) 2025-12-04T08:50:38.7897016Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:50:38.7897441Z return func(*args, **kwargs) 2025-12-04T08:50:38.7897849Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T08:50:38.7898326Z attn_output, self_attn_weights = self.attn( 2025-12-04T08:50:38.7898763Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:50:38.7899177Z return func(*args, **kwargs) 2025-12-04T08:50:38.7899599Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 358, in forward 2025-12-04T08:50:38.7900052Z attn_output = self.c_proj(attn_output) 2025-12-04T08:50:38.7900460Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T08:50:38.7900920Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T08:50:38.7901118Z 2025-12-04T08:50:38.7901237Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:50:38.7901725Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1068, in forward 2025-12-04T08:50:38.7902182Z transformer_outputs = self.transformer( 2025-12-04T08:50:38.7902604Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T08:50:38.7903012Z outputs = block( 2025-12-04T08:50:38.7903374Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:50:38.7903777Z return super().__call__(*args, **kwargs) 2025-12-04T08:50:38.7904206Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:50:38.7904629Z return func(*args, **kwargs) 2025-12-04T08:50:38.7905076Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T08:50:38.7905529Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T08:50:38.7905997Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 374, in forward 2025-12-04T08:50:38.7906446Z hidden_states = self.c_fc(hidden_states) 2025-12-04T08:50:38.7906841Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T08:50:38.7907282Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T08:50:38.7907472Z 2025-12-04T08:50:38.7907589Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:50:38.7908059Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1068, in forward 2025-12-04T08:50:38.7908508Z transformer_outputs = self.transformer( 2025-12-04T08:50:38.7908952Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T08:50:38.7909380Z outputs = block( 2025-12-04T08:50:38.7909747Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:50:38.7910160Z return super().__call__(*args, **kwargs) 2025-12-04T08:50:38.7910593Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:50:38.7911024Z return func(*args, **kwargs) 2025-12-04T08:50:38.7911447Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T08:50:38.7911923Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T08:50:38.7912397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 375, in forward 2025-12-04T08:50:38.7912897Z hidden_states = self.act(hidden_states) 2025-12-04T08:50:38.7913293Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 62, in forward 2025-12-04T08:50:38.7913831Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-12-04T08:50:38.7914103Z 2025-12-04T08:50:38.7914220Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:50:38.7914689Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1068, in forward 2025-12-04T08:50:38.7915167Z transformer_outputs = self.transformer( 2025-12-04T08:50:38.7915613Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T08:50:38.7916083Z outputs = block( 2025-12-04T08:50:38.7916447Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:50:38.7916859Z return super().__call__(*args, **kwargs) 2025-12-04T08:50:38.7917289Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:50:38.7917711Z return func(*args, **kwargs) 2025-12-04T08:50:38.7918116Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T08:50:38.7918671Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T08:50:38.7919137Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 376, in forward 2025-12-04T08:50:38.7919593Z hidden_states = self.c_proj(hidden_states) 2025-12-04T08:50:38.7919996Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T08:50:38.7920453Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T08:50:38.7920648Z 2025-12-04T08:50:38.7920922Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:50:38.7921395Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1068, in forward 2025-12-04T08:50:38.7921846Z transformer_outputs = self.transformer( 2025-12-04T08:50:38.7922289Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T08:50:38.7922727Z outputs = block( 2025-12-04T08:50:38.7923089Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:50:38.7923506Z return super().__call__(*args, **kwargs) 2025-12-04T08:50:38.7923941Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:50:38.7924369Z return func(*args, **kwargs) 2025-12-04T08:50:38.7924802Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 451, in forward 2025-12-04T08:50:38.7925272Z hidden_states = residual + feed_forward_hidden_states 2025-12-04T08:50:38.7925454Z 2025-12-04T08:50:38.7925576Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:50:38.7926040Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1068, in forward 2025-12-04T08:50:38.7926488Z transformer_outputs = self.transformer( 2025-12-04T08:50:38.7926922Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T08:50:38.7927345Z outputs = block( 2025-12-04T08:50:38.7927701Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:50:38.7928116Z return super().__call__(*args, **kwargs) 2025-12-04T08:50:38.7928636Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:50:38.7929050Z return func(*args, **kwargs) 2025-12-04T08:50:38.7929459Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T08:50:38.7929936Z attn_output, self_attn_weights = self.attn( 2025-12-04T08:50:38.7930367Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:50:38.7930771Z return func(*args, **kwargs) 2025-12-04T08:50:38.7931178Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 313, in forward 2025-12-04T08:50:38.7931722Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-12-04T08:50:38.7932290Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T08:50:38.7932723Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T08:50:38.7932920Z 2025-12-04T08:50:38.7933010Z cudagraph partition due to non gpu ops 2025-12-04T08:50:38.7933253Z cudagraph partition due to non gpu ops 2025-12-04T08:50:38.7933508Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:50:38.7933969Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1068, in forward 2025-12-04T08:50:38.7934408Z transformer_outputs = self.transformer( 2025-12-04T08:50:38.7934836Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T08:50:38.7935247Z outputs = block( 2025-12-04T08:50:38.7935605Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:50:38.7936007Z return super().__call__(*args, **kwargs) 2025-12-04T08:50:38.7936426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:50:38.7936831Z return func(*args, **kwargs) 2025-12-04T08:50:38.7937240Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T08:50:38.7937689Z attn_output, self_attn_weights = self.attn( 2025-12-04T08:50:38.7938109Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:50:38.7938523Z return func(*args, **kwargs) 2025-12-04T08:50:38.7938924Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 345, in forward 2025-12-04T08:50:38.7939381Z attn_output, attn_weights = attention_interface( 2025-12-04T08:50:38.7939869Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T08:50:38.7940398Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T08:50:38.7940596Z 2025-12-04T08:50:38.7940716Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:50:38.7941175Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1068, in forward 2025-12-04T08:50:38.7941616Z transformer_outputs = self.transformer( 2025-12-04T08:50:38.7942041Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T08:50:38.7942466Z outputs = block( 2025-12-04T08:50:38.7942823Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:50:38.7943241Z return super().__call__(*args, **kwargs) 2025-12-04T08:50:38.7943712Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:50:38.7944124Z return func(*args, **kwargs) 2025-12-04T08:50:38.7944520Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T08:50:38.7944984Z attn_output, self_attn_weights = self.attn( 2025-12-04T08:50:38.7945410Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:50:38.7945808Z return func(*args, **kwargs) 2025-12-04T08:50:38.7946211Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 358, in forward 2025-12-04T08:50:38.7946647Z attn_output = self.c_proj(attn_output) 2025-12-04T08:50:38.7947036Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T08:50:38.7947492Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T08:50:38.7947688Z 2025-12-04T08:50:38.7947806Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:50:38.7948275Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1068, in forward 2025-12-04T08:50:38.7948719Z transformer_outputs = self.transformer( 2025-12-04T08:50:38.7949147Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T08:50:38.7949561Z outputs = block( 2025-12-04T08:50:38.7949913Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:50:38.7950308Z return super().__call__(*args, **kwargs) 2025-12-04T08:50:38.7950724Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:50:38.7951139Z return func(*args, **kwargs) 2025-12-04T08:50:38.7951553Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T08:50:38.7952007Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T08:50:38.7952462Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 374, in forward 2025-12-04T08:50:38.7952903Z hidden_states = self.c_fc(hidden_states) 2025-12-04T08:50:38.7953290Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T08:50:38.7953735Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T08:50:38.7953928Z 2025-12-04T08:50:38.7954045Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:50:38.7954507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1068, in forward 2025-12-04T08:50:38.7954944Z transformer_outputs = self.transformer( 2025-12-04T08:50:38.7955384Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T08:50:38.7955803Z outputs = block( 2025-12-04T08:50:38.7956154Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:50:38.7956545Z return super().__call__(*args, **kwargs) 2025-12-04T08:50:38.7956960Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:50:38.7957377Z return func(*args, **kwargs) 2025-12-04T08:50:38.7957768Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T08:50:38.7958311Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T08:50:38.7958800Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 375, in forward 2025-12-04T08:50:38.7959290Z hidden_states = self.act(hidden_states) 2025-12-04T08:50:38.7959684Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 62, in forward 2025-12-04T08:50:38.7960207Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-12-04T08:50:38.7960498Z 2025-12-04T08:50:38.7960615Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:50:38.7961093Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1068, in forward 2025-12-04T08:50:38.7961533Z transformer_outputs = self.transformer( 2025-12-04T08:50:38.7961972Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T08:50:38.7962410Z outputs = block( 2025-12-04T08:50:38.7962789Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:50:38.7963206Z return super().__call__(*args, **kwargs) 2025-12-04T08:50:38.7963634Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:50:38.7964058Z return func(*args, **kwargs) 2025-12-04T08:50:38.7964465Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T08:50:38.7964936Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T08:50:38.7965390Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 376, in forward 2025-12-04T08:50:38.7965845Z hidden_states = self.c_proj(hidden_states) 2025-12-04T08:50:38.7966245Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T08:50:38.7966690Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T08:50:38.7966881Z 2025-12-04T08:50:38.7967007Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:50:38.7967469Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1092, in forward 2025-12-04T08:50:38.7967952Z logits = self.lm_head(hidden_states[:, slice_indices, :]) 2025-12-04T08:50:38.7968140Z 2025-12-04T08:50:49.8742739Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:50:49.8743300Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/loss/loss_utils.py", line 67, in ForCausalLMLoss 2025-12-04T08:50:49.8743850Z loss = fixed_cross_entropy(logits, shift_labels, num_items_in_batch, ignore_index, **kwargs) 2025-12-04T08:50:49.8744392Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/loss/loss_utils.py", line 36, in fixed_cross_entropy 2025-12-04T08:50:49.8744990Z loss = nn.functional.cross_entropy(source, target, ignore_index=ignore_index, reduction=reduction) 2025-12-04T08:50:49.8745288Z 2025-12-04T08:50:51.1862959Z Compilation time (from dynamo_timed): 18.270922663 2025-12-04T08:50:51.2069157Z pass 2025-12-04T08:50:51.2069620Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T08:50:51.2070553Z TIMING: gc:0.00349 entire_frame_compile:18.27092 _recursive_pre_grad_passes:0.00798 _recursive_joint_graph_passes:0.24387 _recursive_post_grad_passes:0.05253 async_compile.wait:2.25994 code_gen:11.2307 inductor_compile:12.00418 backend_compile:14.05538 total_wall_time:18.27092 2025-12-04T08:50:51.2071609Z STATS: call_* op count: 311 | FakeTensorMode.__torch_dispatch__:4780 | FakeTensor.__torch_dispatch__:2291 | ProxyTorchDispatchMode.__torch_dispatch__:916 2025-12-04T08:50:51.2072160Z Dynamo produced 2 graphs covering 311 ops with 2 graph breaks (1 unique) 2025-12-04T08:50:53.6645580Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/__init__.py:65: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-12-04T08:50:53.6648173Z import pynvml # type: ignore[import] 2025-12-04T08:50:57.4366560Z 2025-12-04T08:50:57.4381795Z loading model: 0it [00:00, ?it/s]If you want to use `ElectraForCausalLM` as a standalone, add `is_decoder=True.` 2025-12-04T08:50:57.4382450Z WARNING:transformers.models.electra.modeling_electra:If you want to use `ElectraForCausalLM` as a standalone, add `is_decoder=True.` 2025-12-04T08:50:57.6945838Z 2025-12-04T08:50:57.6950831Z loading model: 0it [00:00, ?it/s] 2025-12-04T08:50:57.6960414Z cpu eval ElectraForCausalLM 2025-12-04T08:50:58.0079676Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T08:50:58.1745000Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T08:50:58.3339410Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T08:51:06.5478671Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:06.5479201Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:06.5479643Z res = mod(**inputs) 2025-12-04T08:51:06.5480105Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T08:51:06.5486906Z outputs = self.electra( 2025-12-04T08:51:06.5487881Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 787, in forward 2025-12-04T08:51:06.5488427Z hidden_states = self.embeddings_project(hidden_states) 2025-12-04T08:51:06.5488619Z 2025-12-04T08:51:06.5488746Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:06.5489198Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:06.5489568Z res = mod(**inputs) 2025-12-04T08:51:06.5489999Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T08:51:06.5490446Z outputs = self.electra( 2025-12-04T08:51:06.5490874Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T08:51:06.5491342Z hidden_states = self.encoder( 2025-12-04T08:51:06.5491779Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T08:51:06.5492244Z layer_outputs = layer_module( 2025-12-04T08:51:06.5492634Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:06.5493057Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:06.5493499Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.5493941Z return func(*args, **kwargs) 2025-12-04T08:51:06.5494421Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-12-04T08:51:06.5494878Z self_attention_outputs = self.attention( 2025-12-04T08:51:06.5495317Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.5495742Z return func(*args, **kwargs) 2025-12-04T08:51:06.5496179Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-12-04T08:51:06.5496865Z self_outputs = self.self( 2025-12-04T08:51:06.5497283Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.5498085Z return func(*args, **kwargs) 2025-12-04T08:51:06.5498522Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 239, in forward 2025-12-04T08:51:06.5498971Z query_layer = self.query(hidden_states) 2025-12-04T08:51:06.5499194Z 2025-12-04T08:51:06.5499317Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:06.5499733Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:06.5500096Z res = mod(**inputs) 2025-12-04T08:51:06.5500521Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T08:51:06.5500967Z outputs = self.electra( 2025-12-04T08:51:06.5501388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T08:51:06.5501945Z hidden_states = self.encoder( 2025-12-04T08:51:06.5502395Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T08:51:06.5502863Z layer_outputs = layer_module( 2025-12-04T08:51:06.5503254Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:06.5503673Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:06.5504111Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.5504546Z return func(*args, **kwargs) 2025-12-04T08:51:06.5504974Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-12-04T08:51:06.5505455Z self_attention_outputs = self.attention( 2025-12-04T08:51:06.5505912Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.5506346Z return func(*args, **kwargs) 2025-12-04T08:51:06.5506774Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-12-04T08:51:06.5507236Z self_outputs = self.self( 2025-12-04T08:51:06.5507659Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.5508074Z return func(*args, **kwargs) 2025-12-04T08:51:06.5508502Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 263, in forward 2025-12-04T08:51:06.5509150Z key_layer = self.key(current_states) 2025-12-04T08:51:06.5509301Z 2025-12-04T08:51:06.5509428Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:06.5509825Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:06.5510191Z res = mod(**inputs) 2025-12-04T08:51:06.5510613Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T08:51:06.5511059Z outputs = self.electra( 2025-12-04T08:51:06.5511479Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T08:51:06.5511961Z hidden_states = self.encoder( 2025-12-04T08:51:06.5512392Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T08:51:06.5512829Z layer_outputs = layer_module( 2025-12-04T08:51:06.5513208Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:06.5513614Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:06.5514034Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.5514491Z return func(*args, **kwargs) 2025-12-04T08:51:06.5514934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-12-04T08:51:06.5515407Z self_attention_outputs = self.attention( 2025-12-04T08:51:06.5515845Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.5516243Z return func(*args, **kwargs) 2025-12-04T08:51:06.5516664Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-12-04T08:51:06.5517107Z self_outputs = self.self( 2025-12-04T08:51:06.5517509Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.5517956Z return func(*args, **kwargs) 2025-12-04T08:51:06.5518626Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 267, in forward 2025-12-04T08:51:06.5519092Z value_layer = self.value(current_states) 2025-12-04T08:51:06.5519246Z 2025-12-04T08:51:06.5519340Z cudagraph partition due to non gpu ops 2025-12-04T08:51:06.5519596Z cudagraph partition due to non gpu ops 2025-12-04T08:51:06.5519858Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:06.5520247Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:06.5520602Z res = mod(**inputs) 2025-12-04T08:51:06.5521195Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T08:51:06.5521643Z outputs = self.electra( 2025-12-04T08:51:06.5522050Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T08:51:06.5522490Z hidden_states = self.encoder( 2025-12-04T08:51:06.5522927Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T08:51:06.5523363Z layer_outputs = layer_module( 2025-12-04T08:51:06.5523745Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:06.5524153Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:06.5524577Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.5524989Z return func(*args, **kwargs) 2025-12-04T08:51:06.5525406Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-12-04T08:51:06.5525846Z self_attention_outputs = self.attention( 2025-12-04T08:51:06.5526264Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.5526662Z return func(*args, **kwargs) 2025-12-04T08:51:06.5527081Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 402, in forward 2025-12-04T08:51:06.5527570Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T08:51:06.5528058Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 341, in forward 2025-12-04T08:51:06.5528494Z hidden_states = self.dense(hidden_states) 2025-12-04T08:51:06.5528652Z 2025-12-04T08:51:06.5528765Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:06.5529161Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:06.5529507Z res = mod(**inputs) 2025-12-04T08:51:06.5529914Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T08:51:06.5530416Z outputs = self.electra( 2025-12-04T08:51:06.5530834Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T08:51:06.5531284Z hidden_states = self.encoder( 2025-12-04T08:51:06.5532007Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T08:51:06.5532438Z layer_outputs = layer_module( 2025-12-04T08:51:06.5532816Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:06.5533204Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:06.5533618Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.5534048Z return func(*args, **kwargs) 2025-12-04T08:51:06.5534457Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-12-04T08:51:06.5534905Z layer_output = apply_chunking_to_forward( 2025-12-04T08:51:06.5535345Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:51:06.5535776Z return forward_fn(*input_tensors) 2025-12-04T08:51:06.5536239Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 504, in feed_forward_chunk 2025-12-04T08:51:06.5536760Z intermediate_output = self.intermediate(attention_output) 2025-12-04T08:51:06.5537247Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 418, in forward 2025-12-04T08:51:06.5537691Z hidden_states = self.dense(hidden_states) 2025-12-04T08:51:06.5537843Z 2025-12-04T08:51:06.5537957Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:06.5538355Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:06.5538710Z res = mod(**inputs) 2025-12-04T08:51:06.5539112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T08:51:06.5539550Z outputs = self.electra( 2025-12-04T08:51:06.5539961Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T08:51:06.5540389Z hidden_states = self.encoder( 2025-12-04T08:51:06.5540829Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T08:51:06.5541259Z layer_outputs = layer_module( 2025-12-04T08:51:06.5541634Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:06.5542029Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:06.5542447Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.5542858Z return func(*args, **kwargs) 2025-12-04T08:51:06.5543287Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-12-04T08:51:06.5543697Z layer_output = apply_chunking_to_forward( 2025-12-04T08:51:06.5544112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:51:06.5544519Z return forward_fn(*input_tensors) 2025-12-04T08:51:06.5544955Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 504, in feed_forward_chunk 2025-12-04T08:51:06.5545446Z intermediate_output = self.intermediate(attention_output) 2025-12-04T08:51:06.5545941Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 419, in forward 2025-12-04T08:51:06.5546393Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:51:06.5546787Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T08:51:06.5547168Z return self.act(input) 2025-12-04T08:51:06.5547283Z 2025-12-04T08:51:06.5547399Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:06.5547779Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:06.5548107Z res = mod(**inputs) 2025-12-04T08:51:06.5548488Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T08:51:06.5548897Z outputs = self.electra( 2025-12-04T08:51:06.5549296Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T08:51:06.5549707Z hidden_states = self.encoder( 2025-12-04T08:51:06.5550107Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T08:51:06.5550512Z layer_outputs = layer_module( 2025-12-04T08:51:06.5550866Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:06.5551261Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:06.5551678Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.5552081Z return func(*args, **kwargs) 2025-12-04T08:51:06.5552498Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-12-04T08:51:06.5552922Z layer_output = apply_chunking_to_forward( 2025-12-04T08:51:06.5553344Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:51:06.5553763Z return forward_fn(*input_tensors) 2025-12-04T08:51:06.5554234Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 505, in feed_forward_chunk 2025-12-04T08:51:06.5554772Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T08:51:06.5555268Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 432, in forward 2025-12-04T08:51:06.5555705Z hidden_states = self.dense(hidden_states) 2025-12-04T08:51:06.5555864Z 2025-12-04T08:51:06.5555978Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:06.5556376Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:06.5556727Z res = mod(**inputs) 2025-12-04T08:51:06.5557142Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T08:51:06.5557576Z outputs = self.electra( 2025-12-04T08:51:06.5557989Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T08:51:06.5558565Z hidden_states = self.encoder( 2025-12-04T08:51:06.5559004Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T08:51:06.5559457Z layer_outputs = layer_module( 2025-12-04T08:51:06.5559828Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:06.5560229Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:06.5560648Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.5561107Z return func(*args, **kwargs) 2025-12-04T08:51:06.5561525Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-12-04T08:51:06.5561970Z self_attention_outputs = self.attention( 2025-12-04T08:51:06.5562431Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.5562841Z return func(*args, **kwargs) 2025-12-04T08:51:06.5563253Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-12-04T08:51:06.5563687Z self_outputs = self.self( 2025-12-04T08:51:06.5564084Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.5564501Z return func(*args, **kwargs) 2025-12-04T08:51:06.5564921Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 239, in forward 2025-12-04T08:51:06.5565363Z query_layer = self.query(hidden_states) 2025-12-04T08:51:06.5565513Z 2025-12-04T08:51:06.5565636Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:06.5566028Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:06.5566382Z res = mod(**inputs) 2025-12-04T08:51:06.5566792Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T08:51:06.5567220Z outputs = self.electra( 2025-12-04T08:51:06.5567633Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T08:51:06.5568061Z hidden_states = self.encoder( 2025-12-04T08:51:06.5568481Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T08:51:06.5568901Z layer_outputs = layer_module( 2025-12-04T08:51:06.5569281Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:06.5569680Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:06.5570091Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.5570489Z return func(*args, **kwargs) 2025-12-04T08:51:06.5570906Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-12-04T08:51:06.5571363Z self_attention_outputs = self.attention( 2025-12-04T08:51:06.5571773Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.5572180Z return func(*args, **kwargs) 2025-12-04T08:51:06.5572596Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-12-04T08:51:06.5573025Z self_outputs = self.self( 2025-12-04T08:51:06.5573413Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.5573820Z return func(*args, **kwargs) 2025-12-04T08:51:06.5574237Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 263, in forward 2025-12-04T08:51:06.5574665Z key_layer = self.key(current_states) 2025-12-04T08:51:06.5574819Z 2025-12-04T08:51:06.5574935Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:06.5575330Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:06.5575683Z res = mod(**inputs) 2025-12-04T08:51:06.5576124Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T08:51:06.5576560Z outputs = self.electra( 2025-12-04T08:51:06.5576975Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T08:51:06.5577424Z hidden_states = self.encoder( 2025-12-04T08:51:06.5577837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T08:51:06.5578267Z layer_outputs = layer_module( 2025-12-04T08:51:06.5578643Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:06.5579037Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:06.5579458Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.5579889Z return func(*args, **kwargs) 2025-12-04T08:51:06.5580310Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-12-04T08:51:06.5580753Z self_attention_outputs = self.attention( 2025-12-04T08:51:06.5581175Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.5581597Z return func(*args, **kwargs) 2025-12-04T08:51:06.5582022Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-12-04T08:51:06.5582431Z self_outputs = self.self( 2025-12-04T08:51:06.5582806Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.5583189Z return func(*args, **kwargs) 2025-12-04T08:51:06.5583577Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 267, in forward 2025-12-04T08:51:06.5584014Z value_layer = self.value(current_states) 2025-12-04T08:51:06.5584164Z 2025-12-04T08:51:06.5584262Z cudagraph partition due to non gpu ops 2025-12-04T08:51:06.5584502Z cudagraph partition due to non gpu ops 2025-12-04T08:51:06.5584741Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:06.5585116Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:06.5585453Z res = mod(**inputs) 2025-12-04T08:51:06.5585836Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T08:51:06.5586307Z outputs = self.electra( 2025-12-04T08:51:06.5586687Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T08:51:06.5587083Z hidden_states = self.encoder( 2025-12-04T08:51:06.5587469Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T08:51:06.5587865Z layer_outputs = layer_module( 2025-12-04T08:51:06.5588209Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:06.5588572Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:06.5588966Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.5589351Z return func(*args, **kwargs) 2025-12-04T08:51:06.5589759Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-12-04T08:51:06.5590186Z self_attention_outputs = self.attention( 2025-12-04T08:51:06.5590583Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.5590967Z return func(*args, **kwargs) 2025-12-04T08:51:06.5591397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 402, in forward 2025-12-04T08:51:06.5591869Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T08:51:06.5592377Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 341, in forward 2025-12-04T08:51:06.5592794Z hidden_states = self.dense(hidden_states) 2025-12-04T08:51:06.5592938Z 2025-12-04T08:51:06.5593045Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:06.5593416Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:06.5593753Z res = mod(**inputs) 2025-12-04T08:51:06.5594141Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T08:51:06.5594561Z outputs = self.electra( 2025-12-04T08:51:06.5594976Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T08:51:06.5595409Z hidden_states = self.encoder( 2025-12-04T08:51:06.5595798Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T08:51:06.5596231Z layer_outputs = layer_module( 2025-12-04T08:51:06.5596610Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:06.5597005Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:06.5597412Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.5597817Z return func(*args, **kwargs) 2025-12-04T08:51:06.5598342Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-12-04T08:51:06.5598811Z layer_output = apply_chunking_to_forward( 2025-12-04T08:51:06.5599267Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:51:06.5599711Z return forward_fn(*input_tensors) 2025-12-04T08:51:06.5600198Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 504, in feed_forward_chunk 2025-12-04T08:51:06.5600727Z intermediate_output = self.intermediate(attention_output) 2025-12-04T08:51:06.5601212Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 418, in forward 2025-12-04T08:51:06.5601663Z hidden_states = self.dense(hidden_states) 2025-12-04T08:51:06.5601816Z 2025-12-04T08:51:06.5601936Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:06.5602324Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:06.5602685Z res = mod(**inputs) 2025-12-04T08:51:06.5603098Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T08:51:06.5603521Z outputs = self.electra( 2025-12-04T08:51:06.5603930Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T08:51:06.5604360Z hidden_states = self.encoder( 2025-12-04T08:51:06.5604780Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T08:51:06.5605201Z layer_outputs = layer_module( 2025-12-04T08:51:06.5605578Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:06.5605970Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:06.5606427Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.5606836Z return func(*args, **kwargs) 2025-12-04T08:51:06.5607272Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-12-04T08:51:06.5607771Z layer_output = apply_chunking_to_forward( 2025-12-04T08:51:06.5608236Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:51:06.5608665Z return forward_fn(*input_tensors) 2025-12-04T08:51:06.5609132Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 504, in feed_forward_chunk 2025-12-04T08:51:06.5609710Z intermediate_output = self.intermediate(attention_output) 2025-12-04T08:51:06.5610184Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 419, in forward 2025-12-04T08:51:06.5610694Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:51:06.5611122Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T08:51:06.5611517Z return self.act(input) 2025-12-04T08:51:06.5611641Z 2025-12-04T08:51:06.5611755Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:06.5612156Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:06.5612515Z res = mod(**inputs) 2025-12-04T08:51:06.5612895Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T08:51:06.5613305Z outputs = self.electra( 2025-12-04T08:51:06.5613695Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T08:51:06.5614103Z hidden_states = self.encoder( 2025-12-04T08:51:06.5614501Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T08:51:06.5614908Z layer_outputs = layer_module( 2025-12-04T08:51:06.5615267Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:06.5615635Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:06.5616045Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.5616443Z return func(*args, **kwargs) 2025-12-04T08:51:06.5616838Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-12-04T08:51:06.5617251Z layer_output = apply_chunking_to_forward( 2025-12-04T08:51:06.5617681Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:51:06.5618091Z return forward_fn(*input_tensors) 2025-12-04T08:51:06.5618536Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 505, in feed_forward_chunk 2025-12-04T08:51:06.5619037Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T08:51:06.5619503Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 432, in forward 2025-12-04T08:51:06.5619921Z hidden_states = self.dense(hidden_states) 2025-12-04T08:51:06.5620064Z 2025-12-04T08:51:06.5620171Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:06.5620541Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:06.5621011Z res = mod(**inputs) 2025-12-04T08:51:06.5621406Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T08:51:06.5622328Z outputs = self.electra( 2025-12-04T08:51:06.5622718Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T08:51:06.5623163Z hidden_states = self.encoder( 2025-12-04T08:51:06.5623564Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T08:51:06.5623965Z layer_outputs = layer_module( 2025-12-04T08:51:06.5624325Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:06.5624701Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:06.5625091Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.5625503Z return func(*args, **kwargs) 2025-12-04T08:51:06.5625897Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-12-04T08:51:06.5626318Z self_attention_outputs = self.attention( 2025-12-04T08:51:06.5626705Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.5627091Z return func(*args, **kwargs) 2025-12-04T08:51:06.5627483Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-12-04T08:51:06.5627878Z self_outputs = self.self( 2025-12-04T08:51:06.5628248Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.5628646Z return func(*args, **kwargs) 2025-12-04T08:51:06.5629069Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 239, in forward 2025-12-04T08:51:06.5629507Z query_layer = self.query(hidden_states) 2025-12-04T08:51:06.5629668Z 2025-12-04T08:51:06.5629791Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:06.5630154Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:06.5630487Z res = mod(**inputs) 2025-12-04T08:51:06.5630864Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T08:51:06.5631270Z outputs = self.electra( 2025-12-04T08:51:06.5631681Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T08:51:06.5632111Z hidden_states = self.encoder( 2025-12-04T08:51:06.5632548Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T08:51:06.5632975Z layer_outputs = layer_module( 2025-12-04T08:51:06.5633354Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:06.5633746Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:06.5634160Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.5634569Z return func(*args, **kwargs) 2025-12-04T08:51:06.5634992Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-12-04T08:51:06.5635434Z self_attention_outputs = self.attention( 2025-12-04T08:51:06.5635853Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.5636283Z return func(*args, **kwargs) 2025-12-04T08:51:06.5636702Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-12-04T08:51:06.5637149Z self_outputs = self.self( 2025-12-04T08:51:06.5637594Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.5638004Z return func(*args, **kwargs) 2025-12-04T08:51:06.5638574Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 263, in forward 2025-12-04T08:51:06.5639032Z key_layer = self.key(current_states) 2025-12-04T08:51:06.5639185Z 2025-12-04T08:51:06.5639316Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:06.5639713Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:06.5640081Z res = mod(**inputs) 2025-12-04T08:51:06.5640503Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T08:51:06.5640979Z outputs = self.electra( 2025-12-04T08:51:06.5641400Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T08:51:06.5641847Z hidden_states = self.encoder( 2025-12-04T08:51:06.5642284Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T08:51:06.5642722Z layer_outputs = layer_module( 2025-12-04T08:51:06.5643110Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:06.5643520Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:06.5643949Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.5644363Z return func(*args, **kwargs) 2025-12-04T08:51:06.5644791Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-12-04T08:51:06.5645251Z self_attention_outputs = self.attention( 2025-12-04T08:51:06.5645671Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.5646087Z return func(*args, **kwargs) 2025-12-04T08:51:06.5646479Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-12-04T08:51:06.5646884Z self_outputs = self.self( 2025-12-04T08:51:06.5647253Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.5647636Z return func(*args, **kwargs) 2025-12-04T08:51:06.5648038Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 267, in forward 2025-12-04T08:51:06.5648453Z value_layer = self.value(current_states) 2025-12-04T08:51:06.5648595Z 2025-12-04T08:51:06.5648678Z cudagraph partition due to non gpu ops 2025-12-04T08:51:06.5648902Z cudagraph partition due to non gpu ops 2025-12-04T08:51:06.5649148Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:06.5649512Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:06.5649850Z res = mod(**inputs) 2025-12-04T08:51:06.5650242Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T08:51:06.5650656Z outputs = self.electra( 2025-12-04T08:51:06.5651036Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T08:51:06.5651445Z hidden_states = self.encoder( 2025-12-04T08:51:06.5651844Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T08:51:06.5652244Z layer_outputs = layer_module( 2025-12-04T08:51:06.5652640Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:06.5653017Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:06.5653411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.5653811Z return func(*args, **kwargs) 2025-12-04T08:51:06.5654204Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-12-04T08:51:06.5654625Z self_attention_outputs = self.attention( 2025-12-04T08:51:06.5655012Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.5655396Z return func(*args, **kwargs) 2025-12-04T08:51:06.5655786Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 402, in forward 2025-12-04T08:51:06.5656277Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T08:51:06.5656729Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 341, in forward 2025-12-04T08:51:06.5657153Z hidden_states = self.dense(hidden_states) 2025-12-04T08:51:06.5657305Z 2025-12-04T08:51:06.5657412Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:06.5657784Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:06.5658112Z res = mod(**inputs) 2025-12-04T08:51:06.5658498Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T08:51:06.5658911Z outputs = self.electra( 2025-12-04T08:51:06.5659293Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T08:51:06.5659708Z hidden_states = self.encoder( 2025-12-04T08:51:06.5660109Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T08:51:06.5660516Z layer_outputs = layer_module( 2025-12-04T08:51:06.5660869Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:06.5661243Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:06.5661639Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.5662026Z return func(*args, **kwargs) 2025-12-04T08:51:06.5662414Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-12-04T08:51:06.5662829Z layer_output = apply_chunking_to_forward( 2025-12-04T08:51:06.5663255Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:51:06.5663673Z return forward_fn(*input_tensors) 2025-12-04T08:51:06.5664140Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 504, in feed_forward_chunk 2025-12-04T08:51:06.5664627Z intermediate_output = self.intermediate(attention_output) 2025-12-04T08:51:06.5665084Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 418, in forward 2025-12-04T08:51:06.5665497Z hidden_states = self.dense(hidden_states) 2025-12-04T08:51:06.5665644Z 2025-12-04T08:51:06.5665752Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:06.5666125Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:06.5666462Z res = mod(**inputs) 2025-12-04T08:51:06.5666900Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T08:51:06.5667334Z outputs = self.electra( 2025-12-04T08:51:06.5667747Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T08:51:06.5668192Z hidden_states = self.encoder( 2025-12-04T08:51:06.5668618Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T08:51:06.5669053Z layer_outputs = layer_module( 2025-12-04T08:51:06.5669434Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:06.5669826Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:06.5670244Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.5670677Z return func(*args, **kwargs) 2025-12-04T08:51:06.5671093Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-12-04T08:51:06.5671558Z layer_output = apply_chunking_to_forward( 2025-12-04T08:51:06.5671997Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:51:06.5672431Z return forward_fn(*input_tensors) 2025-12-04T08:51:06.5672890Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 504, in feed_forward_chunk 2025-12-04T08:51:06.5673408Z intermediate_output = self.intermediate(attention_output) 2025-12-04T08:51:06.5673891Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 419, in forward 2025-12-04T08:51:06.5674363Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:51:06.5674781Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T08:51:06.5675159Z return self.act(input) 2025-12-04T08:51:06.5675282Z 2025-12-04T08:51:06.5675403Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:06.5675790Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:06.5676149Z res = mod(**inputs) 2025-12-04T08:51:06.5676561Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T08:51:06.5677000Z outputs = self.electra( 2025-12-04T08:51:06.5677407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T08:51:06.5677834Z hidden_states = self.encoder( 2025-12-04T08:51:06.5678310Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T08:51:06.5678773Z layer_outputs = layer_module( 2025-12-04T08:51:06.5679179Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:06.5679593Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:06.5680042Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.5680453Z return func(*args, **kwargs) 2025-12-04T08:51:06.5680874Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-12-04T08:51:06.5681325Z layer_output = apply_chunking_to_forward( 2025-12-04T08:51:06.5681769Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:51:06.5682201Z return forward_fn(*input_tensors) 2025-12-04T08:51:06.5682734Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 505, in feed_forward_chunk 2025-12-04T08:51:06.5683267Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T08:51:06.5683760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 432, in forward 2025-12-04T08:51:06.5684199Z hidden_states = self.dense(hidden_states) 2025-12-04T08:51:06.5684349Z 2025-12-04T08:51:06.5684456Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:06.5684829Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:06.5685155Z res = mod(**inputs) 2025-12-04T08:51:06.5685542Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T08:51:06.5685967Z outputs = self.electra( 2025-12-04T08:51:06.5686360Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T08:51:06.5686776Z hidden_states = self.encoder( 2025-12-04T08:51:06.5687177Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T08:51:06.5687590Z layer_outputs = layer_module( 2025-12-04T08:51:06.5687940Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:06.5688321Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:06.5688715Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.5689104Z return func(*args, **kwargs) 2025-12-04T08:51:06.5689497Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-12-04T08:51:06.5689922Z self_attention_outputs = self.attention( 2025-12-04T08:51:06.5690324Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.5690706Z return func(*args, **kwargs) 2025-12-04T08:51:06.5691106Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-12-04T08:51:06.5691515Z self_outputs = self.self( 2025-12-04T08:51:06.5691891Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.5692268Z return func(*args, **kwargs) 2025-12-04T08:51:06.5692666Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 239, in forward 2025-12-04T08:51:06.5693085Z query_layer = self.query(hidden_states) 2025-12-04T08:51:06.5693233Z 2025-12-04T08:51:06.5693349Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:06.5693751Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:06.5694108Z res = mod(**inputs) 2025-12-04T08:51:06.5694516Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T08:51:06.5694944Z outputs = self.electra( 2025-12-04T08:51:06.5695335Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T08:51:06.5695740Z hidden_states = self.encoder( 2025-12-04T08:51:06.5696139Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T08:51:06.5696538Z layer_outputs = layer_module( 2025-12-04T08:51:06.5696900Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:06.5697354Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:06.5697775Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.5698192Z return func(*args, **kwargs) 2025-12-04T08:51:06.5698649Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-12-04T08:51:06.5699142Z self_attention_outputs = self.attention( 2025-12-04T08:51:06.5699574Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.5700022Z return func(*args, **kwargs) 2025-12-04T08:51:06.5700453Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-12-04T08:51:06.5700901Z self_outputs = self.self( 2025-12-04T08:51:06.5701325Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.5701744Z return func(*args, **kwargs) 2025-12-04T08:51:06.5702165Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 263, in forward 2025-12-04T08:51:06.5702607Z key_layer = self.key(current_states) 2025-12-04T08:51:06.5702763Z 2025-12-04T08:51:06.5702880Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:06.5703281Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:06.5703652Z res = mod(**inputs) 2025-12-04T08:51:06.5704058Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T08:51:06.5704478Z outputs = self.electra( 2025-12-04T08:51:06.5704858Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T08:51:06.5705248Z hidden_states = self.encoder( 2025-12-04T08:51:06.5705634Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T08:51:06.5706035Z layer_outputs = layer_module( 2025-12-04T08:51:06.5706393Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:06.5706763Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:06.5707175Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.5707587Z return func(*args, **kwargs) 2025-12-04T08:51:06.5707992Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-12-04T08:51:06.5708423Z self_attention_outputs = self.attention( 2025-12-04T08:51:06.5708824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.5709207Z return func(*args, **kwargs) 2025-12-04T08:51:06.5709595Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-12-04T08:51:06.5710002Z self_outputs = self.self( 2025-12-04T08:51:06.5710373Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.5710759Z return func(*args, **kwargs) 2025-12-04T08:51:06.5711142Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 267, in forward 2025-12-04T08:51:06.5711559Z value_layer = self.value(current_states) 2025-12-04T08:51:06.5711698Z 2025-12-04T08:51:06.5711795Z cudagraph partition due to non gpu ops 2025-12-04T08:51:06.5712025Z cudagraph partition due to non gpu ops 2025-12-04T08:51:06.5712326Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:06.5712720Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:06.5713056Z res = mod(**inputs) 2025-12-04T08:51:06.5713439Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T08:51:06.5713885Z outputs = self.electra( 2025-12-04T08:51:06.5714294Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T08:51:06.5714727Z hidden_states = self.encoder( 2025-12-04T08:51:06.5715161Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T08:51:06.5715595Z layer_outputs = layer_module( 2025-12-04T08:51:06.5715997Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:06.5716390Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:06.5716804Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.5717214Z return func(*args, **kwargs) 2025-12-04T08:51:06.5717644Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-12-04T08:51:06.5718080Z self_attention_outputs = self.attention( 2025-12-04T08:51:06.5718591Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.5719007Z return func(*args, **kwargs) 2025-12-04T08:51:06.5719432Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 402, in forward 2025-12-04T08:51:06.5719933Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T08:51:06.5720421Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 341, in forward 2025-12-04T08:51:06.5721029Z hidden_states = self.dense(hidden_states) 2025-12-04T08:51:06.5721198Z 2025-12-04T08:51:06.5721309Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:06.5721687Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:06.5722027Z res = mod(**inputs) 2025-12-04T08:51:06.5722412Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T08:51:06.5722826Z outputs = self.electra( 2025-12-04T08:51:06.5723217Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T08:51:06.5723631Z hidden_states = self.encoder( 2025-12-04T08:51:06.5724030Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T08:51:06.5724443Z layer_outputs = layer_module( 2025-12-04T08:51:06.5724808Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:06.5725189Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:06.5725573Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.5725959Z return func(*args, **kwargs) 2025-12-04T08:51:06.5726353Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-12-04T08:51:06.5726770Z layer_output = apply_chunking_to_forward( 2025-12-04T08:51:06.5727185Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:51:06.5727598Z return forward_fn(*input_tensors) 2025-12-04T08:51:06.5728132Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 504, in feed_forward_chunk 2025-12-04T08:51:06.5728616Z intermediate_output = self.intermediate(attention_output) 2025-12-04T08:51:06.5729105Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 418, in forward 2025-12-04T08:51:06.5729533Z hidden_states = self.dense(hidden_states) 2025-12-04T08:51:06.5729675Z 2025-12-04T08:51:06.5729792Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:06.5730160Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:06.5730497Z res = mod(**inputs) 2025-12-04T08:51:06.5730885Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T08:51:06.5731329Z outputs = self.electra( 2025-12-04T08:51:06.5731724Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T08:51:06.5732132Z hidden_states = self.encoder( 2025-12-04T08:51:06.5732527Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T08:51:06.5732923Z layer_outputs = layer_module( 2025-12-04T08:51:06.5733279Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:06.5733655Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:06.5734039Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.5734428Z return func(*args, **kwargs) 2025-12-04T08:51:06.5734820Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-12-04T08:51:06.5735244Z layer_output = apply_chunking_to_forward( 2025-12-04T08:51:06.5735649Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:51:06.5736060Z return forward_fn(*input_tensors) 2025-12-04T08:51:06.5736501Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 504, in feed_forward_chunk 2025-12-04T08:51:06.5736987Z intermediate_output = self.intermediate(attention_output) 2025-12-04T08:51:06.5737441Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 419, in forward 2025-12-04T08:51:06.5737918Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:51:06.5738342Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T08:51:06.5738714Z return self.act(input) 2025-12-04T08:51:06.5738846Z 2025-12-04T08:51:06.5738962Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:06.5739360Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:06.5739715Z res = mod(**inputs) 2025-12-04T08:51:06.5740130Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T08:51:06.5740559Z outputs = self.electra( 2025-12-04T08:51:06.5740967Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T08:51:06.5741394Z hidden_states = self.encoder( 2025-12-04T08:51:06.5741828Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T08:51:06.5742258Z layer_outputs = layer_module( 2025-12-04T08:51:06.5742677Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:06.5743075Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:06.5743505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.5743936Z return func(*args, **kwargs) 2025-12-04T08:51:06.5744359Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-12-04T08:51:06.5744815Z layer_output = apply_chunking_to_forward( 2025-12-04T08:51:06.5745266Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:51:06.5745704Z return forward_fn(*input_tensors) 2025-12-04T08:51:06.5746167Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 505, in feed_forward_chunk 2025-12-04T08:51:06.5746769Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T08:51:06.5747303Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 432, in forward 2025-12-04T08:51:06.5747791Z hidden_states = self.dense(hidden_states) 2025-12-04T08:51:06.5747945Z 2025-12-04T08:51:06.5748065Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:06.5748496Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:06.5748872Z res = mod(**inputs) 2025-12-04T08:51:06.5749316Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T08:51:06.5749763Z outputs = self.electra( 2025-12-04T08:51:06.5750195Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T08:51:06.5750653Z hidden_states = self.encoder( 2025-12-04T08:51:06.5751107Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T08:51:06.5751569Z layer_outputs = layer_module( 2025-12-04T08:51:06.5751929Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:06.5752356Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:06.5752741Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.5753131Z return func(*args, **kwargs) 2025-12-04T08:51:06.5753525Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-12-04T08:51:06.5753937Z self_attention_outputs = self.attention( 2025-12-04T08:51:06.5754336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.5754720Z return func(*args, **kwargs) 2025-12-04T08:51:06.5755110Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-12-04T08:51:06.5755510Z self_outputs = self.self( 2025-12-04T08:51:06.5755885Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.5756270Z return func(*args, **kwargs) 2025-12-04T08:51:06.5756661Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 239, in forward 2025-12-04T08:51:06.5757071Z query_layer = self.query(hidden_states) 2025-12-04T08:51:06.5757220Z 2025-12-04T08:51:06.5757330Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:06.5757702Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:06.5758068Z res = mod(**inputs) 2025-12-04T08:51:06.5758587Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T08:51:06.5759051Z outputs = self.electra( 2025-12-04T08:51:06.5759515Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T08:51:06.5759943Z hidden_states = self.encoder( 2025-12-04T08:51:06.5760373Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T08:51:06.5760902Z layer_outputs = layer_module( 2025-12-04T08:51:06.5761253Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:06.5761630Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:06.5762049Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.5762439Z return func(*args, **kwargs) 2025-12-04T08:51:06.5762824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-12-04T08:51:06.5763245Z self_attention_outputs = self.attention( 2025-12-04T08:51:06.5763642Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.5764025Z return func(*args, **kwargs) 2025-12-04T08:51:06.5764411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-12-04T08:51:06.5764819Z self_outputs = self.self( 2025-12-04T08:51:06.5765195Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.5765566Z return func(*args, **kwargs) 2025-12-04T08:51:06.5765954Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 263, in forward 2025-12-04T08:51:06.5766361Z key_layer = self.key(current_states) 2025-12-04T08:51:06.5766496Z 2025-12-04T08:51:06.5766611Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:06.5766970Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:06.5767298Z res = mod(**inputs) 2025-12-04T08:51:06.5767673Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T08:51:06.5768061Z outputs = self.electra( 2025-12-04T08:51:06.5768442Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T08:51:06.5768839Z hidden_states = self.encoder( 2025-12-04T08:51:06.5769232Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T08:51:06.5769638Z layer_outputs = layer_module( 2025-12-04T08:51:06.5769989Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:06.5770355Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:06.5770741Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.5771113Z return func(*args, **kwargs) 2025-12-04T08:51:06.5771495Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-12-04T08:51:06.5771902Z self_attention_outputs = self.attention( 2025-12-04T08:51:06.5772280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.5772662Z return func(*args, **kwargs) 2025-12-04T08:51:06.5773090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-12-04T08:51:06.5773489Z self_outputs = self.self( 2025-12-04T08:51:06.5773868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.5774241Z return func(*args, **kwargs) 2025-12-04T08:51:06.5774621Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 267, in forward 2025-12-04T08:51:06.5775016Z value_layer = self.value(current_states) 2025-12-04T08:51:06.5775157Z 2025-12-04T08:51:06.5775238Z cudagraph partition due to non gpu ops 2025-12-04T08:51:06.5775452Z cudagraph partition due to non gpu ops 2025-12-04T08:51:06.5775689Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:06.5776068Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:06.5776401Z res = mod(**inputs) 2025-12-04T08:51:06.5776780Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T08:51:06.5777178Z outputs = self.electra( 2025-12-04T08:51:06.5777562Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T08:51:06.5777957Z hidden_states = self.encoder( 2025-12-04T08:51:06.5778351Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T08:51:06.5778754Z layer_outputs = layer_module( 2025-12-04T08:51:06.5779119Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:06.5779503Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:06.5779902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.5780287Z return func(*args, **kwargs) 2025-12-04T08:51:06.5780688Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-12-04T08:51:06.5781111Z self_attention_outputs = self.attention( 2025-12-04T08:51:06.5781518Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.5781900Z return func(*args, **kwargs) 2025-12-04T08:51:06.5782290Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 402, in forward 2025-12-04T08:51:06.5782753Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T08:51:06.5783213Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 341, in forward 2025-12-04T08:51:06.5783642Z hidden_states = self.dense(hidden_states) 2025-12-04T08:51:06.5783787Z 2025-12-04T08:51:06.5783908Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:06.5784280Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:06.5784623Z res = mod(**inputs) 2025-12-04T08:51:06.5785015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T08:51:06.5785425Z outputs = self.electra( 2025-12-04T08:51:06.5785812Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T08:51:06.5786224Z hidden_states = self.encoder( 2025-12-04T08:51:06.5786629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T08:51:06.5787045Z layer_outputs = layer_module( 2025-12-04T08:51:06.5787440Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:06.5787819Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:06.5788252Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.5788659Z return func(*args, **kwargs) 2025-12-04T08:51:06.5789082Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-12-04T08:51:06.5789535Z layer_output = apply_chunking_to_forward( 2025-12-04T08:51:06.5789983Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:51:06.5790404Z return forward_fn(*input_tensors) 2025-12-04T08:51:06.5790879Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 504, in feed_forward_chunk 2025-12-04T08:51:06.5791368Z intermediate_output = self.intermediate(attention_output) 2025-12-04T08:51:06.5791847Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 418, in forward 2025-12-04T08:51:06.5792278Z hidden_states = self.dense(hidden_states) 2025-12-04T08:51:06.5792428Z 2025-12-04T08:51:06.5792534Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:06.5792903Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:06.5793231Z res = mod(**inputs) 2025-12-04T08:51:06.5793617Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T08:51:06.5794027Z outputs = self.electra( 2025-12-04T08:51:06.5794416Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T08:51:06.5794813Z hidden_states = self.encoder( 2025-12-04T08:51:06.5795213Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T08:51:06.5795615Z layer_outputs = layer_module( 2025-12-04T08:51:06.5795976Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:06.5796372Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:06.5796788Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.5797204Z return func(*args, **kwargs) 2025-12-04T08:51:06.5797611Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-12-04T08:51:06.5798056Z layer_output = apply_chunking_to_forward( 2025-12-04T08:51:06.5798609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:51:06.5799057Z return forward_fn(*input_tensors) 2025-12-04T08:51:06.5799529Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 504, in feed_forward_chunk 2025-12-04T08:51:06.5800048Z intermediate_output = self.intermediate(attention_output) 2025-12-04T08:51:06.5800507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 419, in forward 2025-12-04T08:51:06.5800957Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:51:06.5801396Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T08:51:06.5801787Z return self.act(input) 2025-12-04T08:51:06.5801913Z 2025-12-04T08:51:06.5802044Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:06.5802497Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:06.5802868Z res = mod(**inputs) 2025-12-04T08:51:06.5803291Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T08:51:06.5803759Z outputs = self.electra( 2025-12-04T08:51:06.5804194Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T08:51:06.5804648Z hidden_states = self.encoder( 2025-12-04T08:51:06.5805099Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T08:51:06.5805549Z layer_outputs = layer_module( 2025-12-04T08:51:06.5805949Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:06.5806391Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:06.5806824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.5807241Z return func(*args, **kwargs) 2025-12-04T08:51:06.5807691Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-12-04T08:51:06.5808164Z layer_output = apply_chunking_to_forward( 2025-12-04T08:51:06.5808612Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:51:06.5809048Z return forward_fn(*input_tensors) 2025-12-04T08:51:06.5809489Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 505, in feed_forward_chunk 2025-12-04T08:51:06.5809993Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T08:51:06.5810459Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 432, in forward 2025-12-04T08:51:06.5810882Z hidden_states = self.dense(hidden_states) 2025-12-04T08:51:06.5811031Z 2025-12-04T08:51:06.5811139Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:06.5811524Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:06.5811856Z res = mod(**inputs) 2025-12-04T08:51:06.5812241Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T08:51:06.5812656Z outputs = self.electra( 2025-12-04T08:51:06.5813038Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T08:51:06.5813450Z hidden_states = self.encoder( 2025-12-04T08:51:06.5813853Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T08:51:06.5814256Z layer_outputs = layer_module( 2025-12-04T08:51:06.5814613Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:06.5814995Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:06.5815391Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.5815771Z return func(*args, **kwargs) 2025-12-04T08:51:06.5816168Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-12-04T08:51:06.5816585Z self_attention_outputs = self.attention( 2025-12-04T08:51:06.5816981Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.5817363Z return func(*args, **kwargs) 2025-12-04T08:51:06.5817841Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-12-04T08:51:06.5818275Z self_outputs = self.self( 2025-12-04T08:51:06.5818671Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.5819094Z return func(*args, **kwargs) 2025-12-04T08:51:06.5819532Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 239, in forward 2025-12-04T08:51:06.5819978Z query_layer = self.query(hidden_states) 2025-12-04T08:51:06.5820131Z 2025-12-04T08:51:06.5820247Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:06.5820646Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:06.5821141Z res = mod(**inputs) 2025-12-04T08:51:06.5821564Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T08:51:06.5821994Z outputs = self.electra( 2025-12-04T08:51:06.5822386Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T08:51:06.5822796Z hidden_states = self.encoder( 2025-12-04T08:51:06.5823189Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T08:51:06.5823602Z layer_outputs = layer_module( 2025-12-04T08:51:06.5823965Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:06.5824345Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:06.5824732Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.5825124Z return func(*args, **kwargs) 2025-12-04T08:51:06.5825525Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-12-04T08:51:06.5825930Z self_attention_outputs = self.attention( 2025-12-04T08:51:06.5826311Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.5826685Z return func(*args, **kwargs) 2025-12-04T08:51:06.5827065Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-12-04T08:51:06.5827461Z self_outputs = self.self( 2025-12-04T08:51:06.5827835Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.5828221Z return func(*args, **kwargs) 2025-12-04T08:51:06.5828614Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 263, in forward 2025-12-04T08:51:06.5829030Z key_layer = self.key(current_states) 2025-12-04T08:51:06.5829184Z 2025-12-04T08:51:06.5829297Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:06.5829700Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:06.5830031Z res = mod(**inputs) 2025-12-04T08:51:06.5830417Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T08:51:06.5830828Z outputs = self.electra( 2025-12-04T08:51:06.5831218Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T08:51:06.5831615Z hidden_states = self.encoder( 2025-12-04T08:51:06.5832013Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T08:51:06.5832423Z layer_outputs = layer_module( 2025-12-04T08:51:06.5832904Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:06.5833295Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:06.5833764Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.5834178Z return func(*args, **kwargs) 2025-12-04T08:51:06.5834588Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-12-04T08:51:06.5835039Z self_attention_outputs = self.attention( 2025-12-04T08:51:06.5835460Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.5835868Z return func(*args, **kwargs) 2025-12-04T08:51:06.5836307Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-12-04T08:51:06.5836743Z self_outputs = self.self( 2025-12-04T08:51:06.5837136Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.5837537Z return func(*args, **kwargs) 2025-12-04T08:51:06.5837967Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 267, in forward 2025-12-04T08:51:06.5838497Z value_layer = self.value(current_states) 2025-12-04T08:51:06.5838652Z 2025-12-04T08:51:06.5838750Z cudagraph partition due to non gpu ops 2025-12-04T08:51:06.5838985Z cudagraph partition due to non gpu ops 2025-12-04T08:51:06.5839254Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:06.5839665Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:06.5840016Z res = mod(**inputs) 2025-12-04T08:51:06.5840404Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T08:51:06.5840817Z outputs = self.electra( 2025-12-04T08:51:06.5841211Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T08:51:06.5841616Z hidden_states = self.encoder( 2025-12-04T08:51:06.5842013Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T08:51:06.5842419Z layer_outputs = layer_module( 2025-12-04T08:51:06.5842778Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:06.5843149Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:06.5843543Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.5843932Z return func(*args, **kwargs) 2025-12-04T08:51:06.5844320Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-12-04T08:51:06.5844748Z self_attention_outputs = self.attention( 2025-12-04T08:51:06.5845145Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.5845528Z return func(*args, **kwargs) 2025-12-04T08:51:06.5845914Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 402, in forward 2025-12-04T08:51:06.5846380Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T08:51:06.5846837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 341, in forward 2025-12-04T08:51:06.5847258Z hidden_states = self.dense(hidden_states) 2025-12-04T08:51:06.5847403Z 2025-12-04T08:51:06.5847567Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:06.5847941Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:06.5848274Z res = mod(**inputs) 2025-12-04T08:51:06.5848673Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T08:51:06.5849081Z outputs = self.electra( 2025-12-04T08:51:06.5849470Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T08:51:06.5849875Z hidden_states = self.encoder( 2025-12-04T08:51:06.5850266Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T08:51:06.5850669Z layer_outputs = layer_module( 2025-12-04T08:51:06.5851055Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:06.5851433Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:06.5851836Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.5852232Z return func(*args, **kwargs) 2025-12-04T08:51:06.5852632Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-12-04T08:51:06.5853053Z layer_output = apply_chunking_to_forward( 2025-12-04T08:51:06.5853474Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:51:06.5853889Z return forward_fn(*input_tensors) 2025-12-04T08:51:06.5854339Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 504, in feed_forward_chunk 2025-12-04T08:51:06.5854828Z intermediate_output = self.intermediate(attention_output) 2025-12-04T08:51:06.5855293Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 418, in forward 2025-12-04T08:51:06.5855718Z hidden_states = self.dense(hidden_states) 2025-12-04T08:51:06.5855864Z 2025-12-04T08:51:06.5855974Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:06.5856350Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:06.5856694Z res = mod(**inputs) 2025-12-04T08:51:06.5857087Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T08:51:06.5857499Z outputs = self.electra( 2025-12-04T08:51:06.5857895Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T08:51:06.5858317Z hidden_states = self.encoder( 2025-12-04T08:51:06.5858719Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T08:51:06.5859166Z layer_outputs = layer_module( 2025-12-04T08:51:06.5859536Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:06.5859944Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:06.5860334Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.5860718Z return func(*args, **kwargs) 2025-12-04T08:51:06.5861111Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-12-04T08:51:06.5861533Z layer_output = apply_chunking_to_forward( 2025-12-04T08:51:06.5861936Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:51:06.5862384Z return forward_fn(*input_tensors) 2025-12-04T08:51:06.5862828Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 504, in feed_forward_chunk 2025-12-04T08:51:06.5863330Z intermediate_output = self.intermediate(attention_output) 2025-12-04T08:51:06.5863787Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 419, in forward 2025-12-04T08:51:06.5864232Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:51:06.5864631Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T08:51:06.5864985Z return self.act(input) 2025-12-04T08:51:06.5865106Z 2025-12-04T08:51:06.5865215Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:06.5865613Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:06.5865960Z res = mod(**inputs) 2025-12-04T08:51:06.5866359Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T08:51:06.5866797Z outputs = self.electra( 2025-12-04T08:51:06.5867208Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T08:51:06.5867630Z hidden_states = self.encoder( 2025-12-04T08:51:06.5868061Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T08:51:06.5868490Z layer_outputs = layer_module( 2025-12-04T08:51:06.5868847Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:06.5869226Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:06.5869618Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.5869994Z return func(*args, **kwargs) 2025-12-04T08:51:06.5870381Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-12-04T08:51:06.5870808Z layer_output = apply_chunking_to_forward( 2025-12-04T08:51:06.5871228Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:51:06.5871639Z return forward_fn(*input_tensors) 2025-12-04T08:51:06.5872075Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 505, in feed_forward_chunk 2025-12-04T08:51:06.5872585Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T08:51:06.5873058Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 432, in forward 2025-12-04T08:51:06.5873486Z hidden_states = self.dense(hidden_states) 2025-12-04T08:51:06.5873630Z 2025-12-04T08:51:06.5873738Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:06.5874112Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:06.5874452Z res = mod(**inputs) 2025-12-04T08:51:06.5874837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T08:51:06.5875255Z outputs = self.electra( 2025-12-04T08:51:06.5875647Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T08:51:06.5876060Z hidden_states = self.encoder( 2025-12-04T08:51:06.5876466Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T08:51:06.5876896Z layer_outputs = layer_module( 2025-12-04T08:51:06.5877355Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:06.5877734Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:06.5878123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.5878641Z return func(*args, **kwargs) 2025-12-04T08:51:06.5879077Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-12-04T08:51:06.5879543Z self_attention_outputs = self.attention( 2025-12-04T08:51:06.5879983Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.5880403Z return func(*args, **kwargs) 2025-12-04T08:51:06.5880824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-12-04T08:51:06.5881225Z self_outputs = self.self( 2025-12-04T08:51:06.5881601Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.5881991Z return func(*args, **kwargs) 2025-12-04T08:51:06.5882377Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 239, in forward 2025-12-04T08:51:06.5882797Z query_layer = self.query(hidden_states) 2025-12-04T08:51:06.5882946Z 2025-12-04T08:51:06.5883054Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:06.5883433Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:06.5883776Z res = mod(**inputs) 2025-12-04T08:51:06.5884180Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T08:51:06.5884609Z outputs = self.electra( 2025-12-04T08:51:06.5885029Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T08:51:06.5885456Z hidden_states = self.encoder( 2025-12-04T08:51:06.5885890Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T08:51:06.5886334Z layer_outputs = layer_module( 2025-12-04T08:51:06.5886703Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:06.5887098Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:06.5887512Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.5887919Z return func(*args, **kwargs) 2025-12-04T08:51:06.5888328Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-12-04T08:51:06.5888773Z self_attention_outputs = self.attention( 2025-12-04T08:51:06.5889190Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.5889589Z return func(*args, **kwargs) 2025-12-04T08:51:06.5890000Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-12-04T08:51:06.5890427Z self_outputs = self.self( 2025-12-04T08:51:06.5890815Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.5891209Z return func(*args, **kwargs) 2025-12-04T08:51:06.5891620Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 263, in forward 2025-12-04T08:51:06.5892055Z key_layer = self.key(current_states) 2025-12-04T08:51:06.5892200Z 2025-12-04T08:51:06.5892363Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:06.5892731Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:06.5893080Z res = mod(**inputs) 2025-12-04T08:51:06.5893505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T08:51:06.5893925Z outputs = self.electra( 2025-12-04T08:51:06.5894331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T08:51:06.5894758Z hidden_states = self.encoder( 2025-12-04T08:51:06.5895175Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T08:51:06.5895595Z layer_outputs = layer_module( 2025-12-04T08:51:06.5895995Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:06.5896393Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:06.5896800Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.5897212Z return func(*args, **kwargs) 2025-12-04T08:51:06.5897629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-12-04T08:51:06.5898069Z self_attention_outputs = self.attention( 2025-12-04T08:51:06.5898480Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.5898886Z return func(*args, **kwargs) 2025-12-04T08:51:06.5899297Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-12-04T08:51:06.5899717Z self_outputs = self.self( 2025-12-04T08:51:06.5900115Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.5900520Z return func(*args, **kwargs) 2025-12-04T08:51:06.5900932Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 267, in forward 2025-12-04T08:51:06.5901367Z value_layer = self.value(current_states) 2025-12-04T08:51:06.5901522Z 2025-12-04T08:51:06.5901612Z cudagraph partition due to non gpu ops 2025-12-04T08:51:06.5901848Z cudagraph partition due to non gpu ops 2025-12-04T08:51:06.5902107Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:06.5902468Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:06.5902801Z res = mod(**inputs) 2025-12-04T08:51:06.5903184Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T08:51:06.5903586Z outputs = self.electra( 2025-12-04T08:51:06.5903865Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T08:51:06.5903939Z hidden_states = self.encoder( 2025-12-04T08:51:06.5904214Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T08:51:06.5904287Z layer_outputs = layer_module( 2025-12-04T08:51:06.5904517Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:06.5904608Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:06.5904859Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.5904932Z return func(*args, **kwargs) 2025-12-04T08:51:06.5905268Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-12-04T08:51:06.5905354Z self_attention_outputs = self.attention( 2025-12-04T08:51:06.5905609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.5905694Z return func(*args, **kwargs) 2025-12-04T08:51:06.5905965Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 402, in forward 2025-12-04T08:51:06.5906108Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T08:51:06.5906376Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 341, in forward 2025-12-04T08:51:06.5906461Z hidden_states = self.dense(hidden_states) 2025-12-04T08:51:06.5906472Z 2025-12-04T08:51:06.5906579Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:06.5906805Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:06.5906880Z res = mod(**inputs) 2025-12-04T08:51:06.5907154Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T08:51:06.5907226Z outputs = self.electra( 2025-12-04T08:51:06.5907511Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T08:51:06.5907584Z hidden_states = self.encoder( 2025-12-04T08:51:06.5907862Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T08:51:06.5907934Z layer_outputs = layer_module( 2025-12-04T08:51:06.5908164Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:06.5908255Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:06.5908508Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.5908577Z return func(*args, **kwargs) 2025-12-04T08:51:06.5908853Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-12-04T08:51:06.5908940Z layer_output = apply_chunking_to_forward( 2025-12-04T08:51:06.5909213Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:51:06.5909293Z return forward_fn(*input_tensors) 2025-12-04T08:51:06.5909598Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 504, in feed_forward_chunk 2025-12-04T08:51:06.5909727Z intermediate_output = self.intermediate(attention_output) 2025-12-04T08:51:06.5910003Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 418, in forward 2025-12-04T08:51:06.5910095Z hidden_states = self.dense(hidden_states) 2025-12-04T08:51:06.5910099Z 2025-12-04T08:51:06.5910204Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:06.5910412Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:06.5910486Z res = mod(**inputs) 2025-12-04T08:51:06.5910760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T08:51:06.5910830Z outputs = self.electra( 2025-12-04T08:51:06.5911107Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T08:51:06.5911179Z hidden_states = self.encoder( 2025-12-04T08:51:06.5911457Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T08:51:06.5911564Z layer_outputs = layer_module( 2025-12-04T08:51:06.5911795Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:06.5911884Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:06.5912158Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.5912235Z return func(*args, **kwargs) 2025-12-04T08:51:06.5912505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-12-04T08:51:06.5912592Z layer_output = apply_chunking_to_forward( 2025-12-04T08:51:06.5912868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:51:06.5912964Z return forward_fn(*input_tensors) 2025-12-04T08:51:06.5913268Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 504, in feed_forward_chunk 2025-12-04T08:51:06.5913398Z intermediate_output = self.intermediate(attention_output) 2025-12-04T08:51:06.5913667Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 419, in forward 2025-12-04T08:51:06.5913791Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:51:06.5914015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T08:51:06.5914088Z return self.act(input) 2025-12-04T08:51:06.5914091Z 2025-12-04T08:51:06.5914205Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:06.5914415Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:06.5914492Z res = mod(**inputs) 2025-12-04T08:51:06.5914768Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T08:51:06.5914839Z outputs = self.electra( 2025-12-04T08:51:06.5915115Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T08:51:06.5915190Z hidden_states = self.encoder( 2025-12-04T08:51:06.5915459Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T08:51:06.5915540Z layer_outputs = layer_module( 2025-12-04T08:51:06.5915768Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:06.5915856Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:06.5916106Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.5916180Z return func(*args, **kwargs) 2025-12-04T08:51:06.5916461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-12-04T08:51:06.5916545Z layer_output = apply_chunking_to_forward( 2025-12-04T08:51:06.5916808Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:51:06.5916894Z return forward_fn(*input_tensors) 2025-12-04T08:51:06.5917217Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 505, in feed_forward_chunk 2025-12-04T08:51:06.5917367Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T08:51:06.5917660Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 432, in forward 2025-12-04T08:51:06.5917749Z hidden_states = self.dense(hidden_states) 2025-12-04T08:51:06.5917753Z 2025-12-04T08:51:06.5917908Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:06.5918124Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:06.5918268Z res = mod(**inputs) 2025-12-04T08:51:06.5918569Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T08:51:06.5918666Z outputs = self.electra( 2025-12-04T08:51:06.5918962Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T08:51:06.5919040Z hidden_states = self.encoder( 2025-12-04T08:51:06.5919339Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T08:51:06.5919426Z layer_outputs = layer_module( 2025-12-04T08:51:06.5919697Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:06.5919793Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:06.5920058Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.5920136Z return func(*args, **kwargs) 2025-12-04T08:51:06.5920426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-12-04T08:51:06.5920518Z self_attention_outputs = self.attention( 2025-12-04T08:51:06.5920928Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.5921010Z return func(*args, **kwargs) 2025-12-04T08:51:06.5921308Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-12-04T08:51:06.5921409Z self_outputs = self.self( 2025-12-04T08:51:06.5921676Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.5921750Z return func(*args, **kwargs) 2025-12-04T08:51:06.5922041Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 239, in forward 2025-12-04T08:51:06.5922133Z query_layer = self.query(hidden_states) 2025-12-04T08:51:06.5922138Z 2025-12-04T08:51:06.5922259Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:06.5922471Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:06.5922540Z res = mod(**inputs) 2025-12-04T08:51:06.5922837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T08:51:06.5922911Z outputs = self.electra( 2025-12-04T08:51:06.5923207Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T08:51:06.5923285Z hidden_states = self.encoder( 2025-12-04T08:51:06.5923569Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T08:51:06.5923656Z layer_outputs = layer_module( 2025-12-04T08:51:06.5923897Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:06.5923980Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:06.5924248Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.5924322Z return func(*args, **kwargs) 2025-12-04T08:51:06.5924624Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-12-04T08:51:06.5924715Z self_attention_outputs = self.attention( 2025-12-04T08:51:06.5925047Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.5925130Z return func(*args, **kwargs) 2025-12-04T08:51:06.5925421Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-12-04T08:51:06.5925537Z self_outputs = self.self( 2025-12-04T08:51:06.5925806Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.5925879Z return func(*args, **kwargs) 2025-12-04T08:51:06.5926182Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 263, in forward 2025-12-04T08:51:06.5926268Z key_layer = self.key(current_states) 2025-12-04T08:51:06.5926272Z 2025-12-04T08:51:06.5926421Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:06.5926648Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:06.5926718Z res = mod(**inputs) 2025-12-04T08:51:06.5927025Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T08:51:06.5927100Z outputs = self.electra( 2025-12-04T08:51:06.5927393Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T08:51:06.5927477Z hidden_states = self.encoder( 2025-12-04T08:51:06.5927774Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T08:51:06.5927849Z layer_outputs = layer_module( 2025-12-04T08:51:06.5928097Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:06.5928182Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:06.5928466Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.5928535Z return func(*args, **kwargs) 2025-12-04T08:51:06.5928793Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-12-04T08:51:06.5928883Z self_attention_outputs = self.attention( 2025-12-04T08:51:06.5929126Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.5929196Z return func(*args, **kwargs) 2025-12-04T08:51:06.5929469Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-12-04T08:51:06.5929539Z self_outputs = self.self( 2025-12-04T08:51:06.5929794Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.5929870Z return func(*args, **kwargs) 2025-12-04T08:51:06.5930137Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 267, in forward 2025-12-04T08:51:06.5930229Z value_layer = self.value(current_states) 2025-12-04T08:51:06.5930234Z 2025-12-04T08:51:06.5930320Z cudagraph partition due to non gpu ops 2025-12-04T08:51:06.5930422Z cudagraph partition due to non gpu ops 2025-12-04T08:51:06.5930524Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:06.5930723Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:06.5930794Z res = mod(**inputs) 2025-12-04T08:51:06.5931059Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T08:51:06.5931127Z outputs = self.electra( 2025-12-04T08:51:06.5931426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T08:51:06.5931498Z hidden_states = self.encoder( 2025-12-04T08:51:06.5931770Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T08:51:06.5931862Z layer_outputs = layer_module( 2025-12-04T08:51:06.5932092Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:06.5932182Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:06.5932433Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.5932503Z return func(*args, **kwargs) 2025-12-04T08:51:06.5932778Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-12-04T08:51:06.5932884Z self_attention_outputs = self.attention( 2025-12-04T08:51:06.5933143Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.5933212Z return func(*args, **kwargs) 2025-12-04T08:51:06.5933483Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 402, in forward 2025-12-04T08:51:06.5933626Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T08:51:06.5933894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 341, in forward 2025-12-04T08:51:06.5933986Z hidden_states = self.dense(hidden_states) 2025-12-04T08:51:06.5933990Z 2025-12-04T08:51:06.5934095Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:06.5934299Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:06.5934375Z res = mod(**inputs) 2025-12-04T08:51:06.5934652Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T08:51:06.5934720Z outputs = self.electra( 2025-12-04T08:51:06.5934994Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T08:51:06.5935067Z hidden_states = self.encoder( 2025-12-04T08:51:06.5935344Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T08:51:06.5935415Z layer_outputs = layer_module( 2025-12-04T08:51:06.5935645Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:06.5935733Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:06.5935992Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.5936075Z return func(*args, **kwargs) 2025-12-04T08:51:06.5955346Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-12-04T08:51:06.5955561Z layer_output = apply_chunking_to_forward( 2025-12-04T08:51:06.5955925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:51:06.5956021Z return forward_fn(*input_tensors) 2025-12-04T08:51:06.5956375Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 504, in feed_forward_chunk 2025-12-04T08:51:06.5956515Z intermediate_output = self.intermediate(attention_output) 2025-12-04T08:51:06.5956818Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 418, in forward 2025-12-04T08:51:06.5956925Z hidden_states = self.dense(hidden_states) 2025-12-04T08:51:06.5956932Z 2025-12-04T08:51:06.5957206Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:06.5957457Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:06.5957575Z res = mod(**inputs) 2025-12-04T08:51:06.5957881Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T08:51:06.5957977Z outputs = self.electra( 2025-12-04T08:51:06.5958366Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T08:51:06.5958463Z hidden_states = self.encoder( 2025-12-04T08:51:06.5958751Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T08:51:06.5958869Z layer_outputs = layer_module( 2025-12-04T08:51:06.5959135Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:06.5959229Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:06.5959504Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.5959601Z return func(*args, **kwargs) 2025-12-04T08:51:06.5959959Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-12-04T08:51:06.5960071Z layer_output = apply_chunking_to_forward( 2025-12-04T08:51:06.5960341Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:51:06.5960424Z return forward_fn(*input_tensors) 2025-12-04T08:51:06.5960740Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 504, in feed_forward_chunk 2025-12-04T08:51:06.5960872Z intermediate_output = self.intermediate(attention_output) 2025-12-04T08:51:06.5961150Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 419, in forward 2025-12-04T08:51:06.5961272Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:51:06.5961499Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T08:51:06.5961585Z return self.act(input) 2025-12-04T08:51:06.5961590Z 2025-12-04T08:51:06.5961704Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:06.5961919Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:06.5961999Z res = mod(**inputs) 2025-12-04T08:51:06.5962279Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T08:51:06.5962365Z outputs = self.electra( 2025-12-04T08:51:06.5962639Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T08:51:06.5962715Z hidden_states = self.encoder( 2025-12-04T08:51:06.5962995Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T08:51:06.5963072Z layer_outputs = layer_module( 2025-12-04T08:51:06.5963310Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:06.5963404Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:06.5963656Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.5963739Z return func(*args, **kwargs) 2025-12-04T08:51:06.5964016Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-12-04T08:51:06.5964157Z layer_output = apply_chunking_to_forward( 2025-12-04T08:51:06.5964437Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:51:06.5964530Z return forward_fn(*input_tensors) 2025-12-04T08:51:06.5964843Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 505, in feed_forward_chunk 2025-12-04T08:51:06.5964987Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T08:51:06.5965261Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 432, in forward 2025-12-04T08:51:06.5965358Z hidden_states = self.dense(hidden_states) 2025-12-04T08:51:06.5965362Z 2025-12-04T08:51:06.5965473Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:06.5965728Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:06.5965796Z res = mod(**inputs) 2025-12-04T08:51:06.5966069Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T08:51:06.5966178Z outputs = self.electra( 2025-12-04T08:51:06.5966445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T08:51:06.5966520Z hidden_states = self.encoder( 2025-12-04T08:51:06.5966796Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T08:51:06.5966870Z layer_outputs = layer_module( 2025-12-04T08:51:06.5967107Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:06.5967191Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:06.5967446Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.5967529Z return func(*args, **kwargs) 2025-12-04T08:51:06.5967814Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-12-04T08:51:06.5967909Z self_attention_outputs = self.attention( 2025-12-04T08:51:06.5968184Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.5968261Z return func(*args, **kwargs) 2025-12-04T08:51:06.5968567Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-12-04T08:51:06.5968646Z self_outputs = self.self( 2025-12-04T08:51:06.5968913Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.5969000Z return func(*args, **kwargs) 2025-12-04T08:51:06.5969284Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 239, in forward 2025-12-04T08:51:06.5969381Z query_layer = self.query(hidden_states) 2025-12-04T08:51:06.5969387Z 2025-12-04T08:51:06.5969501Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:06.5969719Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:06.5969799Z res = mod(**inputs) 2025-12-04T08:51:06.5970089Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T08:51:06.5970165Z outputs = self.electra( 2025-12-04T08:51:06.5970457Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T08:51:06.5970538Z hidden_states = self.encoder( 2025-12-04T08:51:06.5970872Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T08:51:06.5970949Z layer_outputs = layer_module( 2025-12-04T08:51:06.5971205Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:06.5971317Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:06.5971581Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.5971658Z return func(*args, **kwargs) 2025-12-04T08:51:06.5971951Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-12-04T08:51:06.5972041Z self_attention_outputs = self.attention( 2025-12-04T08:51:06.5972313Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.5972412Z return func(*args, **kwargs) 2025-12-04T08:51:06.5972697Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-12-04T08:51:06.5972784Z self_outputs = self.self( 2025-12-04T08:51:06.5973059Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.5973140Z return func(*args, **kwargs) 2025-12-04T08:51:06.5973409Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 263, in forward 2025-12-04T08:51:06.5973490Z key_layer = self.key(current_states) 2025-12-04T08:51:06.5973494Z 2025-12-04T08:51:06.5973609Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:06.5973812Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:06.5973882Z res = mod(**inputs) 2025-12-04T08:51:06.5974165Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T08:51:06.5974238Z outputs = self.electra( 2025-12-04T08:51:06.5974526Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T08:51:06.5974600Z hidden_states = self.encoder( 2025-12-04T08:51:06.5974870Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T08:51:06.5974953Z layer_outputs = layer_module( 2025-12-04T08:51:06.5975181Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:06.5975261Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:06.5975519Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.5975594Z return func(*args, **kwargs) 2025-12-04T08:51:06.5975872Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-12-04T08:51:06.5975958Z self_attention_outputs = self.attention( 2025-12-04T08:51:06.5976205Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.5976284Z return func(*args, **kwargs) 2025-12-04T08:51:06.5976550Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-12-04T08:51:06.5976633Z self_outputs = self.self( 2025-12-04T08:51:06.5976884Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.5976957Z return func(*args, **kwargs) 2025-12-04T08:51:06.5977270Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 267, in forward 2025-12-04T08:51:06.5977354Z value_layer = self.value(current_states) 2025-12-04T08:51:06.5977358Z 2025-12-04T08:51:06.5977443Z cudagraph partition due to non gpu ops 2025-12-04T08:51:06.5977554Z cudagraph partition due to non gpu ops 2025-12-04T08:51:06.5977662Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:06.5977879Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:06.5977947Z res = mod(**inputs) 2025-12-04T08:51:06.5978239Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T08:51:06.5978323Z outputs = self.electra( 2025-12-04T08:51:06.5978610Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T08:51:06.5978709Z hidden_states = self.encoder( 2025-12-04T08:51:06.5979011Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T08:51:06.5979089Z layer_outputs = layer_module( 2025-12-04T08:51:06.5979345Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:06.5979431Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:06.5979696Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.5979781Z return func(*args, **kwargs) 2025-12-04T08:51:06.5980070Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-12-04T08:51:06.5980162Z self_attention_outputs = self.attention( 2025-12-04T08:51:06.5980419Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.5980493Z return func(*args, **kwargs) 2025-12-04T08:51:06.5980770Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 402, in forward 2025-12-04T08:51:06.5980909Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T08:51:06.5981179Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 341, in forward 2025-12-04T08:51:06.5981276Z hidden_states = self.dense(hidden_states) 2025-12-04T08:51:06.5981279Z 2025-12-04T08:51:06.5981384Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:06.5981593Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:06.5981662Z res = mod(**inputs) 2025-12-04T08:51:06.5981952Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T08:51:06.5982046Z outputs = self.electra( 2025-12-04T08:51:06.5982332Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T08:51:06.5982422Z hidden_states = self.encoder( 2025-12-04T08:51:06.5982706Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T08:51:06.5982784Z layer_outputs = layer_module( 2025-12-04T08:51:06.5983039Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:06.5983125Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:06.5983389Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.5983474Z return func(*args, **kwargs) 2025-12-04T08:51:06.5983800Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-12-04T08:51:06.5983903Z layer_output = apply_chunking_to_forward( 2025-12-04T08:51:06.5984189Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:51:06.5984290Z return forward_fn(*input_tensors) 2025-12-04T08:51:06.5984622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 504, in feed_forward_chunk 2025-12-04T08:51:06.5984754Z intermediate_output = self.intermediate(attention_output) 2025-12-04T08:51:06.5985055Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 418, in forward 2025-12-04T08:51:06.5985157Z hidden_states = self.dense(hidden_states) 2025-12-04T08:51:06.5985179Z 2025-12-04T08:51:06.5985290Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:06.5985517Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:06.5985599Z res = mod(**inputs) 2025-12-04T08:51:06.5985874Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T08:51:06.5985957Z outputs = self.electra( 2025-12-04T08:51:06.5986226Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T08:51:06.5986309Z hidden_states = self.encoder( 2025-12-04T08:51:06.5986582Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T08:51:06.5986655Z layer_outputs = layer_module( 2025-12-04T08:51:06.5986902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:06.5986989Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:06.5987257Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.5987342Z return func(*args, **kwargs) 2025-12-04T08:51:06.5987624Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-12-04T08:51:06.5987719Z layer_output = apply_chunking_to_forward( 2025-12-04T08:51:06.5987991Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:51:06.5988069Z return forward_fn(*input_tensors) 2025-12-04T08:51:06.5988383Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 504, in feed_forward_chunk 2025-12-04T08:51:06.5988506Z intermediate_output = self.intermediate(attention_output) 2025-12-04T08:51:06.5988798Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 419, in forward 2025-12-04T08:51:06.5988921Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:51:06.5989161Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T08:51:06.5989253Z return self.act(input) 2025-12-04T08:51:06.5989257Z 2025-12-04T08:51:06.5989368Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:06.5989583Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:06.5989659Z res = mod(**inputs) 2025-12-04T08:51:06.5989947Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T08:51:06.5990021Z outputs = self.electra( 2025-12-04T08:51:06.5990314Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T08:51:06.5990441Z hidden_states = self.encoder( 2025-12-04T08:51:06.5990740Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T08:51:06.5990831Z layer_outputs = layer_module( 2025-12-04T08:51:06.5991059Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:06.5991146Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:06.5991396Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.5991467Z return func(*args, **kwargs) 2025-12-04T08:51:06.5991741Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-12-04T08:51:06.5991847Z layer_output = apply_chunking_to_forward( 2025-12-04T08:51:06.5992125Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:51:06.5992202Z return forward_fn(*input_tensors) 2025-12-04T08:51:06.5992502Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 505, in feed_forward_chunk 2025-12-04T08:51:06.5992652Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T08:51:06.5992919Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 432, in forward 2025-12-04T08:51:06.5993010Z hidden_states = self.dense(hidden_states) 2025-12-04T08:51:06.5993014Z 2025-12-04T08:51:06.5993118Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:06.5993323Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:06.5993401Z res = mod(**inputs) 2025-12-04T08:51:06.5993677Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T08:51:06.5993748Z outputs = self.electra( 2025-12-04T08:51:06.5994020Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T08:51:06.5994094Z hidden_states = self.encoder( 2025-12-04T08:51:06.5994371Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T08:51:06.5994442Z layer_outputs = layer_module( 2025-12-04T08:51:06.5994670Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:06.5994759Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:06.5995021Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.5995110Z return func(*args, **kwargs) 2025-12-04T08:51:06.5995397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-12-04T08:51:06.5995487Z self_attention_outputs = self.attention( 2025-12-04T08:51:06.5995757Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.5995833Z return func(*args, **kwargs) 2025-12-04T08:51:06.5996119Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-12-04T08:51:06.5996205Z self_outputs = self.self( 2025-12-04T08:51:06.5996464Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.5996546Z return func(*args, **kwargs) 2025-12-04T08:51:06.5996868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 239, in forward 2025-12-04T08:51:06.5996958Z query_layer = self.query(hidden_states) 2025-12-04T08:51:06.5996962Z 2025-12-04T08:51:06.5997082Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:06.5997361Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:06.5997438Z res = mod(**inputs) 2025-12-04T08:51:06.5997738Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T08:51:06.5997811Z outputs = self.electra( 2025-12-04T08:51:06.5998107Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T08:51:06.5998183Z hidden_states = self.encoder( 2025-12-04T08:51:06.5998556Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T08:51:06.5998681Z layer_outputs = layer_module( 2025-12-04T08:51:06.5998932Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:06.5999030Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:06.5999305Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.5999383Z return func(*args, **kwargs) 2025-12-04T08:51:06.5999686Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-12-04T08:51:06.5999778Z self_attention_outputs = self.attention( 2025-12-04T08:51:06.6000049Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.6000147Z return func(*args, **kwargs) 2025-12-04T08:51:06.6000443Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-12-04T08:51:06.6000528Z self_outputs = self.self( 2025-12-04T08:51:06.6000791Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.6000867Z return func(*args, **kwargs) 2025-12-04T08:51:06.6001162Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 263, in forward 2025-12-04T08:51:06.6001247Z key_layer = self.key(current_states) 2025-12-04T08:51:06.6001252Z 2025-12-04T08:51:06.6001373Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:06.6001589Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:06.6001659Z res = mod(**inputs) 2025-12-04T08:51:06.6001957Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T08:51:06.6002034Z outputs = self.electra( 2025-12-04T08:51:06.6002318Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T08:51:06.6002407Z hidden_states = self.encoder( 2025-12-04T08:51:06.6002693Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T08:51:06.6002776Z layer_outputs = layer_module( 2025-12-04T08:51:06.6003019Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:06.6003103Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:06.6003375Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.6003450Z return func(*args, **kwargs) 2025-12-04T08:51:06.6003772Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-12-04T08:51:06.6003870Z self_attention_outputs = self.attention( 2025-12-04T08:51:06.6004132Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.6004231Z return func(*args, **kwargs) 2025-12-04T08:51:06.6004519Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-12-04T08:51:06.6004595Z self_outputs = self.self( 2025-12-04T08:51:06.6004875Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.6004949Z return func(*args, **kwargs) 2025-12-04T08:51:06.6005242Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 267, in forward 2025-12-04T08:51:06.6005412Z value_layer = self.value(current_states) 2025-12-04T08:51:06.6005419Z 2025-12-04T08:51:06.6005506Z cudagraph partition due to non gpu ops 2025-12-04T08:51:06.6005598Z cudagraph partition due to non gpu ops 2025-12-04T08:51:06.6005713Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:06.6005929Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:06.6006009Z res = mod(**inputs) 2025-12-04T08:51:06.6006303Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T08:51:06.6006384Z outputs = self.electra( 2025-12-04T08:51:06.6006665Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T08:51:06.6006741Z hidden_states = self.encoder( 2025-12-04T08:51:06.6007037Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T08:51:06.6007113Z layer_outputs = layer_module( 2025-12-04T08:51:06.6007357Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:06.6007454Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:06.6007720Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.6007803Z return func(*args, **kwargs) 2025-12-04T08:51:06.6008086Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-12-04T08:51:06.6008175Z self_attention_outputs = self.attention( 2025-12-04T08:51:06.6008448Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.6008524Z return func(*args, **kwargs) 2025-12-04T08:51:06.6008817Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 402, in forward 2025-12-04T08:51:06.6008957Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T08:51:06.6009239Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 341, in forward 2025-12-04T08:51:06.6009339Z hidden_states = self.dense(hidden_states) 2025-12-04T08:51:06.6009342Z 2025-12-04T08:51:06.6009453Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:06.6009674Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:06.6009750Z res = mod(**inputs) 2025-12-04T08:51:06.6010041Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T08:51:06.6010123Z outputs = self.electra( 2025-12-04T08:51:06.6010440Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T08:51:06.6010517Z hidden_states = self.encoder( 2025-12-04T08:51:06.6010809Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T08:51:06.6010906Z layer_outputs = layer_module( 2025-12-04T08:51:06.6011157Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:06.6011242Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:06.6011505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.6011589Z return func(*args, **kwargs) 2025-12-04T08:51:06.6011872Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-12-04T08:51:06.6011990Z layer_output = apply_chunking_to_forward( 2025-12-04T08:51:06.6012281Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:51:06.6012363Z return forward_fn(*input_tensors) 2025-12-04T08:51:06.6012701Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 504, in feed_forward_chunk 2025-12-04T08:51:06.6012824Z intermediate_output = self.intermediate(attention_output) 2025-12-04T08:51:06.6013092Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 418, in forward 2025-12-04T08:51:06.6013184Z hidden_states = self.dense(hidden_states) 2025-12-04T08:51:06.6013188Z 2025-12-04T08:51:06.6013293Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:06.6013502Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:06.6013571Z res = mod(**inputs) 2025-12-04T08:51:06.6013843Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T08:51:06.6013921Z outputs = self.electra( 2025-12-04T08:51:06.6014188Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T08:51:06.6014260Z hidden_states = self.encoder( 2025-12-04T08:51:06.6014534Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T08:51:06.6014606Z layer_outputs = layer_module( 2025-12-04T08:51:06.6014840Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:06.6014918Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:06.6015171Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.6015249Z return func(*args, **kwargs) 2025-12-04T08:51:06.6015518Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-12-04T08:51:06.6015604Z layer_output = apply_chunking_to_forward( 2025-12-04T08:51:06.6015876Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:51:06.6015953Z return forward_fn(*input_tensors) 2025-12-04T08:51:06.6016261Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 504, in feed_forward_chunk 2025-12-04T08:51:06.6016387Z intermediate_output = self.intermediate(attention_output) 2025-12-04T08:51:06.6016655Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 419, in forward 2025-12-04T08:51:06.6016817Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:51:06.6017049Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T08:51:06.6017129Z return self.act(input) 2025-12-04T08:51:06.6017148Z 2025-12-04T08:51:06.6017255Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:06.6017460Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:06.6017535Z res = mod(**inputs) 2025-12-04T08:51:06.6017813Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T08:51:06.6017884Z outputs = self.electra( 2025-12-04T08:51:06.6018161Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T08:51:06.6018260Z hidden_states = self.encoder( 2025-12-04T08:51:06.6018542Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T08:51:06.6018614Z layer_outputs = layer_module( 2025-12-04T08:51:06.6018842Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:06.6018934Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:06.6019187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.6019267Z return func(*args, **kwargs) 2025-12-04T08:51:06.6019535Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-12-04T08:51:06.6019620Z layer_output = apply_chunking_to_forward( 2025-12-04T08:51:06.6019893Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:51:06.6019973Z return forward_fn(*input_tensors) 2025-12-04T08:51:06.6020276Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 505, in feed_forward_chunk 2025-12-04T08:51:06.6020424Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T08:51:06.6020834Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 432, in forward 2025-12-04T08:51:06.6020938Z hidden_states = self.dense(hidden_states) 2025-12-04T08:51:06.6020942Z 2025-12-04T08:51:06.6021049Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:06.6021253Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:06.6021328Z res = mod(**inputs) 2025-12-04T08:51:06.6021600Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T08:51:06.6021685Z outputs = self.electra( 2025-12-04T08:51:06.6021955Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T08:51:06.6022025Z hidden_states = self.encoder( 2025-12-04T08:51:06.6022300Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T08:51:06.6022372Z layer_outputs = layer_module( 2025-12-04T08:51:06.6022599Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:06.6022689Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:06.6022937Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.6023014Z return func(*args, **kwargs) 2025-12-04T08:51:06.6023386Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-12-04T08:51:06.6023474Z self_attention_outputs = self.attention( 2025-12-04T08:51:06.6023735Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.6023836Z return func(*args, **kwargs) 2025-12-04T08:51:06.6024101Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-12-04T08:51:06.6024182Z self_outputs = self.self( 2025-12-04T08:51:06.6024428Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.6024517Z return func(*args, **kwargs) 2025-12-04T08:51:06.6024782Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 239, in forward 2025-12-04T08:51:06.6024887Z query_layer = self.query(hidden_states) 2025-12-04T08:51:06.6024894Z 2025-12-04T08:51:06.6025009Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:06.6025212Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:06.6025286Z res = mod(**inputs) 2025-12-04T08:51:06.6025559Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T08:51:06.6025629Z outputs = self.electra( 2025-12-04T08:51:06.6025903Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T08:51:06.6025972Z hidden_states = self.encoder( 2025-12-04T08:51:06.6026238Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T08:51:06.6026318Z layer_outputs = layer_module( 2025-12-04T08:51:06.6026554Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:06.6026638Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:06.6026887Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.6026958Z return func(*args, **kwargs) 2025-12-04T08:51:06.6027233Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-12-04T08:51:06.6027315Z self_attention_outputs = self.attention( 2025-12-04T08:51:06.6027571Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.6027640Z return func(*args, **kwargs) 2025-12-04T08:51:06.6027906Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-12-04T08:51:06.6027986Z self_outputs = self.self( 2025-12-04T08:51:06.6028238Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.6028306Z return func(*args, **kwargs) 2025-12-04T08:51:06.6028585Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 263, in forward 2025-12-04T08:51:06.6028664Z key_layer = self.key(current_states) 2025-12-04T08:51:06.6028668Z 2025-12-04T08:51:06.6028777Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:06.6028979Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:06.6029042Z res = mod(**inputs) 2025-12-04T08:51:06.6029318Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T08:51:06.6029389Z outputs = self.electra( 2025-12-04T08:51:06.6029694Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T08:51:06.6029773Z hidden_states = self.encoder( 2025-12-04T08:51:06.6030033Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T08:51:06.6030129Z layer_outputs = layer_module( 2025-12-04T08:51:06.6030350Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:06.6030427Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:06.6030682Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.6030752Z return func(*args, **kwargs) 2025-12-04T08:51:06.6031027Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-12-04T08:51:06.6031131Z self_attention_outputs = self.attention( 2025-12-04T08:51:06.6031383Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.6031458Z return func(*args, **kwargs) 2025-12-04T08:51:06.6031727Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-12-04T08:51:06.6031803Z self_outputs = self.self( 2025-12-04T08:51:06.6032061Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.6032127Z return func(*args, **kwargs) 2025-12-04T08:51:06.6032400Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 267, in forward 2025-12-04T08:51:06.6032480Z value_layer = self.value(current_states) 2025-12-04T08:51:06.6032485Z 2025-12-04T08:51:06.6032564Z cudagraph partition due to non gpu ops 2025-12-04T08:51:06.6032653Z cudagraph partition due to non gpu ops 2025-12-04T08:51:06.6032756Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:06.6032952Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:06.6033024Z res = mod(**inputs) 2025-12-04T08:51:06.6033292Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T08:51:06.6033366Z outputs = self.electra( 2025-12-04T08:51:06.6033631Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T08:51:06.6033703Z hidden_states = self.encoder( 2025-12-04T08:51:06.6033979Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T08:51:06.6034053Z layer_outputs = layer_module( 2025-12-04T08:51:06.6034292Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:06.6034372Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:06.6034622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.6034699Z return func(*args, **kwargs) 2025-12-04T08:51:06.6034970Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-12-04T08:51:06.6035052Z self_attention_outputs = self.attention( 2025-12-04T08:51:06.6035310Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.6035380Z return func(*args, **kwargs) 2025-12-04T08:51:06.6035658Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 402, in forward 2025-12-04T08:51:06.6035825Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T08:51:06.6036116Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 341, in forward 2025-12-04T08:51:06.6036252Z hidden_states = self.dense(hidden_states) 2025-12-04T08:51:06.6036256Z 2025-12-04T08:51:06.6036375Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:06.6036597Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:06.6036672Z res = mod(**inputs) 2025-12-04T08:51:06.6036960Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T08:51:06.6037034Z outputs = self.electra( 2025-12-04T08:51:06.6037333Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T08:51:06.6037427Z hidden_states = self.encoder( 2025-12-04T08:51:06.6037721Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T08:51:06.6037805Z layer_outputs = layer_module( 2025-12-04T08:51:06.6038049Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:06.6038141Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:06.6038483Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.6038566Z return func(*args, **kwargs) 2025-12-04T08:51:06.6038870Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-12-04T08:51:06.6038964Z layer_output = apply_chunking_to_forward( 2025-12-04T08:51:06.6039280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:51:06.6039363Z return forward_fn(*input_tensors) 2025-12-04T08:51:06.6039681Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 504, in feed_forward_chunk 2025-12-04T08:51:06.6039822Z intermediate_output = self.intermediate(attention_output) 2025-12-04T08:51:06.6040117Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 418, in forward 2025-12-04T08:51:06.6040207Z hidden_states = self.dense(hidden_states) 2025-12-04T08:51:06.6040220Z 2025-12-04T08:51:06.6040333Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:06.6040546Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:06.6040624Z res = mod(**inputs) 2025-12-04T08:51:06.6040915Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T08:51:06.6040985Z outputs = self.electra( 2025-12-04T08:51:06.6041259Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T08:51:06.6041334Z hidden_states = self.encoder( 2025-12-04T08:51:06.6041612Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T08:51:06.6041685Z layer_outputs = layer_module( 2025-12-04T08:51:06.6041919Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:06.6042013Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:06.6042287Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.6042364Z return func(*args, **kwargs) 2025-12-04T08:51:06.6042709Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-12-04T08:51:06.6042801Z layer_output = apply_chunking_to_forward( 2025-12-04T08:51:06.6043091Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:51:06.6043191Z return forward_fn(*input_tensors) 2025-12-04T08:51:06.6043510Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 504, in feed_forward_chunk 2025-12-04T08:51:06.6043646Z intermediate_output = self.intermediate(attention_output) 2025-12-04T08:51:06.6043945Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 419, in forward 2025-12-04T08:51:06.6044073Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:51:06.6044331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T08:51:06.6044409Z return self.act(input) 2025-12-04T08:51:06.6044413Z 2025-12-04T08:51:06.6044533Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:06.6044758Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:06.6044828Z res = mod(**inputs) 2025-12-04T08:51:06.6045144Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T08:51:06.6045219Z outputs = self.electra( 2025-12-04T08:51:06.6045547Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T08:51:06.6045626Z hidden_states = self.encoder( 2025-12-04T08:51:06.6045951Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T08:51:06.6046043Z layer_outputs = layer_module( 2025-12-04T08:51:06.6046293Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:06.6046380Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:06.6046660Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.6046739Z return func(*args, **kwargs) 2025-12-04T08:51:06.6047036Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-12-04T08:51:06.6047127Z layer_output = apply_chunking_to_forward( 2025-12-04T08:51:06.6047416Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:51:06.6047509Z return forward_fn(*input_tensors) 2025-12-04T08:51:06.6047854Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 505, in feed_forward_chunk 2025-12-04T08:51:06.6048012Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T08:51:06.6048305Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 432, in forward 2025-12-04T08:51:06.6048398Z hidden_states = self.dense(hidden_states) 2025-12-04T08:51:06.6048402Z 2025-12-04T08:51:06.6048523Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:06.6048743Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:06.6048814Z res = mod(**inputs) 2025-12-04T08:51:06.6049119Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T08:51:06.6049196Z outputs = self.electra( 2025-12-04T08:51:06.6049529Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T08:51:06.6049611Z hidden_states = self.encoder( 2025-12-04T08:51:06.6049901Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T08:51:06.6050005Z layer_outputs = layer_module( 2025-12-04T08:51:06.6050260Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:06.6050353Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:06.6050615Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.6050689Z return func(*args, **kwargs) 2025-12-04T08:51:06.6050989Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-12-04T08:51:06.6051098Z self_attention_outputs = self.attention( 2025-12-04T08:51:06.6051362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.6051444Z return func(*args, **kwargs) 2025-12-04T08:51:06.6051727Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-12-04T08:51:06.6051808Z self_outputs = self.self( 2025-12-04T08:51:06.6052070Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.6052143Z return func(*args, **kwargs) 2025-12-04T08:51:06.6052433Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 239, in forward 2025-12-04T08:51:06.6052521Z query_layer = self.query(hidden_states) 2025-12-04T08:51:06.6052527Z 2025-12-04T08:51:06.6052643Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:06.6052861Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:06.6052930Z res = mod(**inputs) 2025-12-04T08:51:06.6053223Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T08:51:06.6053299Z outputs = self.electra( 2025-12-04T08:51:06.6053598Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T08:51:06.6053682Z hidden_states = self.encoder( 2025-12-04T08:51:06.6053981Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T08:51:06.6054065Z layer_outputs = layer_module( 2025-12-04T08:51:06.6054303Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:06.6054388Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:06.6054660Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.6054734Z return func(*args, **kwargs) 2025-12-04T08:51:06.6055016Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-12-04T08:51:06.6055114Z self_attention_outputs = self.attention( 2025-12-04T08:51:06.6055375Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.6055456Z return func(*args, **kwargs) 2025-12-04T08:51:06.6055756Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-12-04T08:51:06.6055834Z self_outputs = self.self( 2025-12-04T08:51:06.6056150Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.6056226Z return func(*args, **kwargs) 2025-12-04T08:51:06.6056518Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 263, in forward 2025-12-04T08:51:06.6056620Z key_layer = self.key(current_states) 2025-12-04T08:51:06.6056624Z 2025-12-04T08:51:06.6056734Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:06.6056955Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:06.6057024Z res = mod(**inputs) 2025-12-04T08:51:06.6057309Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T08:51:06.6057392Z outputs = self.electra( 2025-12-04T08:51:06.6057672Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T08:51:06.6057775Z hidden_states = self.encoder( 2025-12-04T08:51:06.6058055Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T08:51:06.6058131Z layer_outputs = layer_module( 2025-12-04T08:51:06.6058381Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:06.6058465Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:06.6058726Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.6058806Z return func(*args, **kwargs) 2025-12-04T08:51:06.6059090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-12-04T08:51:06.6059186Z self_attention_outputs = self.attention( 2025-12-04T08:51:06.6059454Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.6059528Z return func(*args, **kwargs) 2025-12-04T08:51:06.6059818Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 393, in forward 2025-12-04T08:51:06.6059897Z self_outputs = self.self( 2025-12-04T08:51:06.6060170Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.6060244Z return func(*args, **kwargs) 2025-12-04T08:51:06.6060526Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 267, in forward 2025-12-04T08:51:06.6060623Z value_layer = self.value(current_states) 2025-12-04T08:51:06.6060626Z 2025-12-04T08:51:06.6060714Z cudagraph partition due to non gpu ops 2025-12-04T08:51:06.6060803Z cudagraph partition due to non gpu ops 2025-12-04T08:51:06.6060923Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:06.6061139Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:06.6061216Z res = mod(**inputs) 2025-12-04T08:51:06.6061504Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T08:51:06.6061579Z outputs = self.electra( 2025-12-04T08:51:06.6061867Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T08:51:06.6061944Z hidden_states = self.encoder( 2025-12-04T08:51:06.6062235Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T08:51:06.6062320Z layer_outputs = layer_module( 2025-12-04T08:51:06.6062561Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:06.6062704Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:06.6062969Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.6063044Z return func(*args, **kwargs) 2025-12-04T08:51:06.6063353Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 466, in forward 2025-12-04T08:51:06.6063441Z self_attention_outputs = self.attention( 2025-12-04T08:51:06.6063713Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.6063785Z return func(*args, **kwargs) 2025-12-04T08:51:06.6064067Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 402, in forward 2025-12-04T08:51:06.6064233Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T08:51:06.6064519Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 341, in forward 2025-12-04T08:51:06.6064610Z hidden_states = self.dense(hidden_states) 2025-12-04T08:51:06.6064614Z 2025-12-04T08:51:06.6064733Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:06.6064948Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:06.6065025Z res = mod(**inputs) 2025-12-04T08:51:06.6065315Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T08:51:06.6065390Z outputs = self.electra( 2025-12-04T08:51:06.6065692Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T08:51:06.6065767Z hidden_states = self.encoder( 2025-12-04T08:51:06.6066069Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T08:51:06.6066144Z layer_outputs = layer_module( 2025-12-04T08:51:06.6066394Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:06.6066487Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:06.6066752Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.6066825Z return func(*args, **kwargs) 2025-12-04T08:51:06.6067117Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-12-04T08:51:06.6067209Z layer_output = apply_chunking_to_forward( 2025-12-04T08:51:06.6067498Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:51:06.6067583Z return forward_fn(*input_tensors) 2025-12-04T08:51:06.6067902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 504, in feed_forward_chunk 2025-12-04T08:51:06.6068039Z intermediate_output = self.intermediate(attention_output) 2025-12-04T08:51:06.6068328Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 418, in forward 2025-12-04T08:51:06.6068414Z hidden_states = self.dense(hidden_states) 2025-12-04T08:51:06.6068425Z 2025-12-04T08:51:06.6068529Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:06.6068730Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:06.6068810Z res = mod(**inputs) 2025-12-04T08:51:06.6069095Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T08:51:06.6069179Z outputs = self.electra( 2025-12-04T08:51:06.6069498Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T08:51:06.6069574Z hidden_states = self.encoder( 2025-12-04T08:51:06.6069863Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T08:51:06.6069960Z layer_outputs = layer_module( 2025-12-04T08:51:06.6070209Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:06.6070294Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:06.6070554Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.6070636Z return func(*args, **kwargs) 2025-12-04T08:51:06.6070919Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-12-04T08:51:06.6071034Z layer_output = apply_chunking_to_forward( 2025-12-04T08:51:06.6071334Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:51:06.6071417Z return forward_fn(*input_tensors) 2025-12-04T08:51:06.6071743Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 504, in feed_forward_chunk 2025-12-04T08:51:06.6071870Z intermediate_output = self.intermediate(attention_output) 2025-12-04T08:51:06.6072162Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 419, in forward 2025-12-04T08:51:06.6072288Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:51:06.6072509Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T08:51:06.6072590Z return self.act(input) 2025-12-04T08:51:06.6072594Z 2025-12-04T08:51:06.6072700Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:06.6072902Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:06.6072978Z res = mod(**inputs) 2025-12-04T08:51:06.6073250Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1533, in forward 2025-12-04T08:51:06.6073321Z outputs = self.electra( 2025-12-04T08:51:06.6073594Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 789, in forward 2025-12-04T08:51:06.6073665Z hidden_states = self.encoder( 2025-12-04T08:51:06.6073938Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 559, in forward 2025-12-04T08:51:06.6074011Z layer_outputs = layer_module( 2025-12-04T08:51:06.6074239Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:06.6074326Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:06.6074574Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:06.6074646Z return func(*args, **kwargs) 2025-12-04T08:51:06.6074919Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 496, in forward 2025-12-04T08:51:06.6075004Z layer_output = apply_chunking_to_forward( 2025-12-04T08:51:06.6075289Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:51:06.6075369Z return forward_fn(*input_tensors) 2025-12-04T08:51:06.6075684Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 505, in feed_forward_chunk 2025-12-04T08:51:06.6075876Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T08:51:06.6076160Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 432, in forward 2025-12-04T08:51:06.6076279Z hidden_states = self.dense(hidden_states) 2025-12-04T08:51:06.6076283Z 2025-12-04T08:51:06.6076397Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:06.6076614Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:06.6076692Z res = mod(**inputs) 2025-12-04T08:51:06.6076981Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1550, in forward 2025-12-04T08:51:06.6077195Z prediction_scores = self.generator_lm_head(self.generator_predictions(sequence_output)) 2025-12-04T08:51:06.6077501Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 630, in forward 2025-12-04T08:51:06.6077614Z hidden_states = self.dense(generator_hidden_states) 2025-12-04T08:51:06.6077618Z 2025-12-04T08:51:06.6077738Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:06.6077952Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:06.6078023Z res = mod(**inputs) 2025-12-04T08:51:06.6078538Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1550, in forward 2025-12-04T08:51:06.6078739Z prediction_scores = self.generator_lm_head(self.generator_predictions(sequence_output)) 2025-12-04T08:51:06.6078743Z 2025-12-04T08:51:06.6078862Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:06.6079080Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:06.6079157Z res = mod(**inputs) 2025-12-04T08:51:06.6079471Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1554, in forward 2025-12-04T08:51:06.6079551Z lm_loss = self.loss_function( 2025-12-04T08:51:06.6079835Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/loss/loss_utils.py", line 67, in ForCausalLMLoss 2025-12-04T08:51:06.6080038Z loss = fixed_cross_entropy(logits, shift_labels, num_items_in_batch, ignore_index, **kwargs) 2025-12-04T08:51:06.6080334Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/loss/loss_utils.py", line 36, in fixed_cross_entropy 2025-12-04T08:51:06.6080558Z loss = nn.functional.cross_entropy(source, target, ignore_index=ignore_index, reduction=reduction) 2025-12-04T08:51:06.6080562Z 2025-12-04T08:51:17.6245666Z Compilation time (from dynamo_timed): 18.633943116 2025-12-04T08:51:17.6360611Z pass 2025-12-04T08:51:17.6361076Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T08:51:17.6362007Z TIMING: _recursive_pre_grad_passes:0.00854 _recursive_joint_graph_passes:0.49538 _recursive_post_grad_passes:0.07028 async_compile.wait:0.92553 code_gen:10.5495 inductor_compile:11.87937 backend_compile:15.48975 gc:0.00035 entire_frame_compile:18.63394 total_wall_time:18.63394 2025-12-04T08:51:17.6363067Z STATS: call_* op count: 377 | FakeTensorMode.__torch_dispatch__:8666 | FakeTensor.__torch_dispatch__:4346 | ProxyTorchDispatchMode.__torch_dispatch__:2555 2025-12-04T08:51:17.6363640Z Dynamo produced 1 graphs covering 377 ops with 0 graph breaks (0 unique) 2025-12-04T08:51:20.0769134Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/__init__.py:65: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-12-04T08:51:20.0770130Z import pynvml # type: ignore[import] 2025-12-04T08:51:23.4794704Z 2025-12-04T08:51:25.2203703Z loading model: 0it [00:00, ?it/s] 2025-12-04T08:51:25.2204050Z loading model: 0it [00:01, ?it/s] 2025-12-04T08:51:25.2212683Z cpu eval GPT2ForSequenceClassification 2025-12-04T08:51:27.7692836Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T08:51:28.9242969Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T08:51:30.0389934Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T08:51:36.8366768Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:36.8367226Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:36.8367582Z res = mod(**inputs) 2025-12-04T08:51:36.8367987Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T08:51:36.8368839Z transformer_outputs = self.transformer( 2025-12-04T08:51:36.8369316Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 873, in forward 2025-12-04T08:51:36.8369750Z causal_mask = create_causal_mask( 2025-12-04T08:51:36.8370156Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/masking_utils.py", line 788, in create_causal_mask 2025-12-04T08:51:36.8370689Z early_exit, attention_mask, packed_sequence_mask, kv_length, kv_offset = _preprocess_mask_arguments( 2025-12-04T08:51:36.8371241Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/masking_utils.py", line 740, in _preprocess_mask_arguments 2025-12-04T08:51:36.8371725Z packed_sequence_mask = find_packed_sequence_indices(position_ids) 2025-12-04T08:51:36.8372199Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/masking_utils.py", line 655, in find_packed_sequence_indices 2025-12-04T08:51:36.8372734Z first_dummy_value = position_ids[:, :1] - 1 # We just need the diff on this first value to be 1 2025-12-04T08:51:36.8373010Z 2025-12-04T08:51:36.8373100Z cudagraph partition due to non gpu ops 2025-12-04T08:51:36.8373361Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:36.8373743Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:36.8374126Z res = mod(**inputs) 2025-12-04T08:51:36.8374510Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1394, in forward 2025-12-04T08:51:36.8374971Z last_non_pad_token = (token_indices * non_pad_mask).argmax(-1) 2025-12-04T08:51:36.8375153Z 2025-12-04T08:51:36.8375264Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:36.8375850Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:36.8376526Z res = mod(**inputs) 2025-12-04T08:51:36.8377009Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T08:51:36.8377442Z transformer_outputs = self.transformer( 2025-12-04T08:51:36.8377899Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T08:51:36.8378345Z outputs = block( 2025-12-04T08:51:36.8378740Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:36.8379167Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:36.8379597Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:36.8380105Z return func(*args, **kwargs) 2025-12-04T08:51:36.8380543Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T08:51:36.8381099Z attn_output, self_attn_weights = self.attn( 2025-12-04T08:51:36.8381814Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:36.8382228Z return func(*args, **kwargs) 2025-12-04T08:51:36.8382819Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 313, in forward 2025-12-04T08:51:36.8383365Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-12-04T08:51:36.8383844Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T08:51:36.8384263Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T08:51:36.8384454Z 2025-12-04T08:51:36.8384569Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:36.8384998Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:36.8385337Z res = mod(**inputs) 2025-12-04T08:51:36.8385706Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T08:51:36.8386124Z transformer_outputs = self.transformer( 2025-12-04T08:51:36.8386535Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 873, in forward 2025-12-04T08:51:36.8386939Z causal_mask = create_causal_mask( 2025-12-04T08:51:36.8387345Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/masking_utils.py", line 788, in create_causal_mask 2025-12-04T08:51:36.8387875Z early_exit, attention_mask, packed_sequence_mask, kv_length, kv_offset = _preprocess_mask_arguments( 2025-12-04T08:51:36.8388424Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/masking_utils.py", line 740, in _preprocess_mask_arguments 2025-12-04T08:51:36.8388899Z packed_sequence_mask = find_packed_sequence_indices(position_ids) 2025-12-04T08:51:36.8389394Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/masking_utils.py", line 656, in find_packed_sequence_indices 2025-12-04T08:51:36.8389881Z position_diff = torch.diff(position_ids, prepend=first_dummy_value, dim=-1) 2025-12-04T08:51:36.8390094Z 2025-12-04T08:51:36.8390186Z cudagraph partition due to non gpu ops 2025-12-04T08:51:36.8390436Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:36.8390819Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:36.8391158Z res = mod(**inputs) 2025-12-04T08:51:36.8391546Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T08:51:36.8391955Z transformer_outputs = self.transformer( 2025-12-04T08:51:36.8392377Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 873, in forward 2025-12-04T08:51:36.8392790Z causal_mask = create_causal_mask( 2025-12-04T08:51:36.8393181Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/masking_utils.py", line 788, in create_causal_mask 2025-12-04T08:51:36.8393704Z early_exit, attention_mask, packed_sequence_mask, kv_length, kv_offset = _preprocess_mask_arguments( 2025-12-04T08:51:36.8394250Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/masking_utils.py", line 740, in _preprocess_mask_arguments 2025-12-04T08:51:36.8394723Z packed_sequence_mask = find_packed_sequence_indices(position_ids) 2025-12-04T08:51:36.8395193Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/masking_utils.py", line 657, in find_packed_sequence_indices 2025-12-04T08:51:36.8395672Z packed_sequence_mask = (position_diff != 1).cumsum(-1) 2025-12-04T08:51:36.8395864Z 2025-12-04T08:51:36.8395987Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:36.8396434Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:36.8396831Z res = mod(**inputs) 2025-12-04T08:51:36.8397232Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T08:51:36.8397741Z transformer_outputs = self.transformer( 2025-12-04T08:51:36.8398162Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 873, in forward 2025-12-04T08:51:36.8398748Z causal_mask = create_causal_mask( 2025-12-04T08:51:36.8399168Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/masking_utils.py", line 788, in create_causal_mask 2025-12-04T08:51:36.8399735Z early_exit, attention_mask, packed_sequence_mask, kv_length, kv_offset = _preprocess_mask_arguments( 2025-12-04T08:51:36.8400344Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/masking_utils.py", line 740, in _preprocess_mask_arguments 2025-12-04T08:51:36.8400883Z packed_sequence_mask = find_packed_sequence_indices(position_ids) 2025-12-04T08:51:36.8401380Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/masking_utils.py", line 657, in find_packed_sequence_indices 2025-12-04T08:51:36.8401880Z packed_sequence_mask = (position_diff != 1).cumsum(-1) 2025-12-04T08:51:36.8402058Z 2025-12-04T08:51:36.8402147Z cudagraph partition due to non gpu ops 2025-12-04T08:51:36.8402413Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:36.8402809Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:36.8403164Z res = mod(**inputs) 2025-12-04T08:51:36.8403551Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T08:51:36.8403983Z transformer_outputs = self.transformer( 2025-12-04T08:51:36.8404428Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T08:51:36.8404853Z outputs = block( 2025-12-04T08:51:36.8405229Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:36.8405631Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:36.8406048Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:36.8406458Z return func(*args, **kwargs) 2025-12-04T08:51:36.8406863Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T08:51:36.8407310Z attn_output, self_attn_weights = self.attn( 2025-12-04T08:51:36.8407743Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:36.8408141Z return func(*args, **kwargs) 2025-12-04T08:51:36.8408530Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 345, in forward 2025-12-04T08:51:36.8408963Z attn_output, attn_weights = attention_interface( 2025-12-04T08:51:36.8409429Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T08:51:36.8409937Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T08:51:36.8410145Z 2025-12-04T08:51:36.8410254Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:36.8410633Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:36.8410976Z res = mod(**inputs) 2025-12-04T08:51:36.8411354Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T08:51:36.8411773Z transformer_outputs = self.transformer( 2025-12-04T08:51:36.8412210Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T08:51:36.8412607Z outputs = block( 2025-12-04T08:51:36.8412941Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:36.8413357Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:36.8413749Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:36.8414143Z return func(*args, **kwargs) 2025-12-04T08:51:36.8414525Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T08:51:36.8414939Z attn_output, self_attn_weights = self.attn( 2025-12-04T08:51:36.8415331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:36.8415737Z return func(*args, **kwargs) 2025-12-04T08:51:36.8416122Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 358, in forward 2025-12-04T08:51:36.8416517Z attn_output = self.c_proj(attn_output) 2025-12-04T08:51:36.8417160Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T08:51:36.8417613Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T08:51:36.8417795Z 2025-12-04T08:51:36.8417913Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:36.8418285Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:36.8418626Z res = mod(**inputs) 2025-12-04T08:51:36.8418997Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T08:51:36.8419409Z transformer_outputs = self.transformer( 2025-12-04T08:51:36.8419807Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T08:51:36.8420189Z outputs = block( 2025-12-04T08:51:36.8420525Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:36.8421090Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:36.8421489Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:36.8421877Z return func(*args, **kwargs) 2025-12-04T08:51:36.8422256Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T08:51:36.8422680Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T08:51:36.8423109Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 374, in forward 2025-12-04T08:51:36.8423516Z hidden_states = self.c_fc(hidden_states) 2025-12-04T08:51:36.8423879Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T08:51:36.8424291Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T08:51:36.8424476Z 2025-12-04T08:51:36.8424584Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:36.8424956Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:36.8425285Z res = mod(**inputs) 2025-12-04T08:51:36.8425650Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T08:51:36.8426056Z transformer_outputs = self.transformer( 2025-12-04T08:51:36.8426450Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T08:51:36.8426902Z outputs = block( 2025-12-04T08:51:36.8427244Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:36.8427620Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:36.8428032Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:36.8428421Z return func(*args, **kwargs) 2025-12-04T08:51:36.8428804Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T08:51:36.8429229Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T08:51:36.8429639Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 375, in forward 2025-12-04T08:51:36.8430043Z hidden_states = self.act(hidden_states) 2025-12-04T08:51:36.8430483Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 62, in forward 2025-12-04T08:51:36.8430955Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-12-04T08:51:36.8431202Z 2025-12-04T08:51:36.8431310Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:36.8431688Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:36.8432020Z res = mod(**inputs) 2025-12-04T08:51:36.8432382Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T08:51:36.8432792Z transformer_outputs = self.transformer( 2025-12-04T08:51:36.8433193Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T08:51:36.8433588Z outputs = block( 2025-12-04T08:51:36.8433919Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:36.8434293Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:36.8434684Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:36.8435074Z return func(*args, **kwargs) 2025-12-04T08:51:36.8435477Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T08:51:36.8435923Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T08:51:36.8436362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 376, in forward 2025-12-04T08:51:36.8436786Z hidden_states = self.c_proj(hidden_states) 2025-12-04T08:51:36.8437180Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T08:51:36.8437620Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T08:51:36.8437809Z 2025-12-04T08:51:36.8437927Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:36.8438387Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:36.8438767Z res = mod(**inputs) 2025-12-04T08:51:36.8439171Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T08:51:36.8439598Z transformer_outputs = self.transformer( 2025-12-04T08:51:36.8440011Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T08:51:36.8440388Z outputs = block( 2025-12-04T08:51:36.8440716Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:36.8441080Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:36.8441497Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:36.8441880Z return func(*args, **kwargs) 2025-12-04T08:51:36.8442248Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T08:51:36.8442672Z attn_output, self_attn_weights = self.attn( 2025-12-04T08:51:36.8443077Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:36.8443459Z return func(*args, **kwargs) 2025-12-04T08:51:36.8443835Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 313, in forward 2025-12-04T08:51:36.8444349Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-12-04T08:51:36.8444847Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T08:51:36.8445270Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T08:51:36.8445443Z 2025-12-04T08:51:36.8445526Z cudagraph partition due to non gpu ops 2025-12-04T08:51:36.8445766Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:36.8446132Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:36.8446451Z res = mod(**inputs) 2025-12-04T08:51:36.8446815Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T08:51:36.8447220Z transformer_outputs = self.transformer( 2025-12-04T08:51:36.8447609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T08:51:36.8447975Z outputs = block( 2025-12-04T08:51:36.8448307Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:36.8448679Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:36.8449054Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:36.8449430Z return func(*args, **kwargs) 2025-12-04T08:51:36.8449800Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T08:51:36.8450197Z attn_output, self_attn_weights = self.attn( 2025-12-04T08:51:36.8450585Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:36.8450970Z return func(*args, **kwargs) 2025-12-04T08:51:36.8451347Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 345, in forward 2025-12-04T08:51:36.8451767Z attn_output, attn_weights = attention_interface( 2025-12-04T08:51:36.8452223Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T08:51:36.8452723Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T08:51:36.8452912Z 2025-12-04T08:51:36.8453027Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:36.8453393Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:36.8453727Z res = mod(**inputs) 2025-12-04T08:51:36.8454096Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T08:51:36.8454503Z transformer_outputs = self.transformer( 2025-12-04T08:51:36.8454895Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T08:51:36.8455280Z outputs = block( 2025-12-04T08:51:36.8455655Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:36.8456028Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:36.8456425Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:36.8456830Z return func(*args, **kwargs) 2025-12-04T08:51:36.8457218Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T08:51:36.8457649Z attn_output, self_attn_weights = self.attn( 2025-12-04T08:51:36.8458073Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:36.8458482Z return func(*args, **kwargs) 2025-12-04T08:51:36.8458881Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 358, in forward 2025-12-04T08:51:36.8459300Z attn_output = self.c_proj(attn_output) 2025-12-04T08:51:36.8459675Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T08:51:36.8460093Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T08:51:36.8460270Z 2025-12-04T08:51:36.8460378Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:36.8460750Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:36.8461082Z res = mod(**inputs) 2025-12-04T08:51:36.8461449Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T08:51:36.8461849Z transformer_outputs = self.transformer( 2025-12-04T08:51:36.8462253Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T08:51:36.8462640Z outputs = block( 2025-12-04T08:51:36.8462967Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:36.8463344Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:36.8463755Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:36.8464164Z return func(*args, **kwargs) 2025-12-04T08:51:36.8464557Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T08:51:36.8465010Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T08:51:36.8465502Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 374, in forward 2025-12-04T08:51:36.8465908Z hidden_states = self.c_fc(hidden_states) 2025-12-04T08:51:36.8466272Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T08:51:36.8466693Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T08:51:36.8466882Z 2025-12-04T08:51:36.8467003Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:36.8467391Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:36.8467742Z res = mod(**inputs) 2025-12-04T08:51:36.8468134Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T08:51:36.8468613Z transformer_outputs = self.transformer( 2025-12-04T08:51:36.8469006Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T08:51:36.8469416Z outputs = block( 2025-12-04T08:51:36.8469785Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:36.8470155Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:36.8470592Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:36.8471062Z return func(*args, **kwargs) 2025-12-04T08:51:36.8471468Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T08:51:36.8471936Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T08:51:36.8472391Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 375, in forward 2025-12-04T08:51:36.8472817Z hidden_states = self.act(hidden_states) 2025-12-04T08:51:36.8473206Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 62, in forward 2025-12-04T08:51:36.8473701Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-12-04T08:51:36.8473990Z 2025-12-04T08:51:36.8474117Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:36.8474516Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:36.8474869Z res = mod(**inputs) 2025-12-04T08:51:36.8475264Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T08:51:36.8475706Z transformer_outputs = self.transformer( 2025-12-04T08:51:36.8476142Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T08:51:36.8476550Z outputs = block( 2025-12-04T08:51:36.8476909Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:36.8477317Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:36.8477739Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:36.8478152Z return func(*args, **kwargs) 2025-12-04T08:51:36.8478652Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T08:51:36.8479154Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T08:51:36.8479618Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 376, in forward 2025-12-04T08:51:36.8480073Z hidden_states = self.c_proj(hidden_states) 2025-12-04T08:51:36.8480472Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T08:51:36.8480912Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T08:51:36.8481104Z 2025-12-04T08:51:36.8481221Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:36.8481634Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:36.8481992Z res = mod(**inputs) 2025-12-04T08:51:36.8482383Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T08:51:36.8482811Z transformer_outputs = self.transformer( 2025-12-04T08:51:36.8483236Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T08:51:36.8483712Z outputs = block( 2025-12-04T08:51:36.8484054Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:36.8484453Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:36.8484876Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:36.8485285Z return func(*args, **kwargs) 2025-12-04T08:51:36.8485729Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T08:51:36.8486143Z attn_output, self_attn_weights = self.attn( 2025-12-04T08:51:36.8486543Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:36.8486942Z return func(*args, **kwargs) 2025-12-04T08:51:36.8487331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 313, in forward 2025-12-04T08:51:36.8487848Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-12-04T08:51:36.8488335Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T08:51:36.8488744Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T08:51:36.8488952Z 2025-12-04T08:51:36.8489040Z cudagraph partition due to non gpu ops 2025-12-04T08:51:36.8489305Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:36.8489689Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:36.8490044Z res = mod(**inputs) 2025-12-04T08:51:36.8490454Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T08:51:36.8490895Z transformer_outputs = self.transformer( 2025-12-04T08:51:36.8491320Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T08:51:36.8491743Z outputs = block( 2025-12-04T08:51:36.8492083Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:36.8492464Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:36.8492856Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:36.8493253Z return func(*args, **kwargs) 2025-12-04T08:51:36.8493638Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T08:51:36.8494048Z attn_output, self_attn_weights = self.attn( 2025-12-04T08:51:36.8494461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:36.8494848Z return func(*args, **kwargs) 2025-12-04T08:51:36.8495230Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 345, in forward 2025-12-04T08:51:36.8495655Z attn_output, attn_weights = attention_interface( 2025-12-04T08:51:36.8496128Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T08:51:36.8496630Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T08:51:36.8496823Z 2025-12-04T08:51:36.8496949Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:36.8497345Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:36.8497707Z res = mod(**inputs) 2025-12-04T08:51:36.8498114Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T08:51:36.8498550Z transformer_outputs = self.transformer( 2025-12-04T08:51:36.8498979Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T08:51:36.8499376Z outputs = block( 2025-12-04T08:51:36.8499712Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:36.8500089Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:36.8500528Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:36.8500912Z return func(*args, **kwargs) 2025-12-04T08:51:36.8501290Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T08:51:36.8501720Z attn_output, self_attn_weights = self.attn( 2025-12-04T08:51:36.8502124Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:36.8502540Z return func(*args, **kwargs) 2025-12-04T08:51:36.8502935Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 358, in forward 2025-12-04T08:51:36.8503365Z attn_output = self.c_proj(attn_output) 2025-12-04T08:51:36.8503756Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T08:51:36.8504226Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T08:51:36.8504428Z 2025-12-04T08:51:36.8504547Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:36.8504951Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:36.8505314Z res = mod(**inputs) 2025-12-04T08:51:36.8505706Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T08:51:36.8506144Z transformer_outputs = self.transformer( 2025-12-04T08:51:36.8506573Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T08:51:36.8506991Z outputs = block( 2025-12-04T08:51:36.8507348Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:36.8507756Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:36.8508187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:36.8508592Z return func(*args, **kwargs) 2025-12-04T08:51:36.8509007Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T08:51:36.8509475Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T08:51:36.8509937Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 374, in forward 2025-12-04T08:51:36.8510368Z hidden_states = self.c_fc(hidden_states) 2025-12-04T08:51:36.8510786Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T08:51:36.8511238Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T08:51:36.8511434Z 2025-12-04T08:51:36.8511566Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:36.8511974Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:36.8512344Z res = mod(**inputs) 2025-12-04T08:51:36.8512745Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T08:51:36.8513180Z transformer_outputs = self.transformer( 2025-12-04T08:51:36.8513613Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T08:51:36.8514029Z outputs = block( 2025-12-04T08:51:36.8514388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:36.8514786Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:36.8515212Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:36.8515634Z return func(*args, **kwargs) 2025-12-04T08:51:36.8516094Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T08:51:36.8516550Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T08:51:36.8516995Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 375, in forward 2025-12-04T08:51:36.8517444Z hidden_states = self.act(hidden_states) 2025-12-04T08:51:36.8517848Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 62, in forward 2025-12-04T08:51:36.8518443Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-12-04T08:51:36.8518730Z 2025-12-04T08:51:36.8518853Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:36.8519288Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:36.8519662Z res = mod(**inputs) 2025-12-04T08:51:36.8520058Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T08:51:36.8520492Z transformer_outputs = self.transformer( 2025-12-04T08:51:36.8521088Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T08:51:36.8521516Z outputs = block( 2025-12-04T08:51:36.8521879Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:36.8522283Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:36.8522698Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:36.8523123Z return func(*args, **kwargs) 2025-12-04T08:51:36.8523530Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T08:51:36.8523991Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T08:51:36.8524454Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 376, in forward 2025-12-04T08:51:36.8524899Z hidden_states = self.c_proj(hidden_states) 2025-12-04T08:51:36.8525307Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T08:51:36.8525750Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T08:51:36.8525945Z 2025-12-04T08:51:36.8526058Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:36.8526455Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:36.8526809Z res = mod(**inputs) 2025-12-04T08:51:36.8527197Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T08:51:36.8527638Z transformer_outputs = self.transformer( 2025-12-04T08:51:36.8528066Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T08:51:36.8528475Z outputs = block( 2025-12-04T08:51:36.8528825Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:36.8529228Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:36.8529653Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:36.8530057Z return func(*args, **kwargs) 2025-12-04T08:51:36.8530464Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 451, in forward 2025-12-04T08:51:36.8530924Z hidden_states = residual + feed_forward_hidden_states 2025-12-04T08:51:36.8531102Z 2025-12-04T08:51:36.8531230Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:36.8531680Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:36.8532030Z res = mod(**inputs) 2025-12-04T08:51:36.8532419Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T08:51:36.8532867Z transformer_outputs = self.transformer( 2025-12-04T08:51:36.8533283Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T08:51:36.8533691Z outputs = block( 2025-12-04T08:51:36.8534034Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:36.8534411Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:36.8534813Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:36.8535271Z return func(*args, **kwargs) 2025-12-04T08:51:36.8535657Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T08:51:36.8536081Z attn_output, self_attn_weights = self.attn( 2025-12-04T08:51:36.8536498Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:36.8536898Z return func(*args, **kwargs) 2025-12-04T08:51:36.8537283Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 313, in forward 2025-12-04T08:51:36.8537826Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-12-04T08:51:36.8538335Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T08:51:36.8538782Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T08:51:36.8538969Z 2025-12-04T08:51:36.8539062Z cudagraph partition due to non gpu ops 2025-12-04T08:51:36.8539328Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:36.8539702Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:36.8540037Z res = mod(**inputs) 2025-12-04T08:51:36.8540410Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T08:51:36.8540820Z transformer_outputs = self.transformer( 2025-12-04T08:51:36.8541218Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T08:51:36.8541599Z outputs = block( 2025-12-04T08:51:36.8541932Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:36.8542314Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:36.8542701Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:36.8543087Z return func(*args, **kwargs) 2025-12-04T08:51:36.8543466Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T08:51:36.8543878Z attn_output, self_attn_weights = self.attn( 2025-12-04T08:51:36.8544270Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:36.8544655Z return func(*args, **kwargs) 2025-12-04T08:51:36.8545034Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 345, in forward 2025-12-04T08:51:36.8545457Z attn_output, attn_weights = attention_interface( 2025-12-04T08:51:36.8545917Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T08:51:36.8546453Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T08:51:36.8546642Z 2025-12-04T08:51:36.8546756Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:36.8547128Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:36.8547484Z res = mod(**inputs) 2025-12-04T08:51:36.8547857Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T08:51:36.8548256Z transformer_outputs = self.transformer( 2025-12-04T08:51:36.8548644Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T08:51:36.8549020Z outputs = block( 2025-12-04T08:51:36.8549345Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:36.8549725Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:36.8550100Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:36.8550486Z return func(*args, **kwargs) 2025-12-04T08:51:36.8550866Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T08:51:36.8551267Z attn_output, self_attn_weights = self.attn( 2025-12-04T08:51:36.8551668Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:36.8552049Z return func(*args, **kwargs) 2025-12-04T08:51:36.8552429Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 358, in forward 2025-12-04T08:51:36.8552843Z attn_output = self.c_proj(attn_output) 2025-12-04T08:51:36.8553241Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T08:51:36.8553673Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T08:51:36.8553858Z 2025-12-04T08:51:36.8553978Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:36.8554362Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:36.8554714Z res = mod(**inputs) 2025-12-04T08:51:36.8555102Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T08:51:36.8555522Z transformer_outputs = self.transformer( 2025-12-04T08:51:36.8555958Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T08:51:36.8556362Z outputs = block( 2025-12-04T08:51:36.8556711Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:36.8557112Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:36.8557521Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:36.8557924Z return func(*args, **kwargs) 2025-12-04T08:51:36.8558384Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T08:51:36.8558862Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T08:51:36.8559337Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 374, in forward 2025-12-04T08:51:36.8559766Z hidden_states = self.c_fc(hidden_states) 2025-12-04T08:51:36.8560156Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T08:51:36.8560612Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T08:51:36.8560801Z 2025-12-04T08:51:36.8560961Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:36.8561368Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:36.8561726Z res = mod(**inputs) 2025-12-04T08:51:36.8562140Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T08:51:36.8562569Z transformer_outputs = self.transformer( 2025-12-04T08:51:36.8562985Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T08:51:36.8563399Z outputs = block( 2025-12-04T08:51:36.8563751Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:36.8564144Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:36.8564582Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:36.8564992Z return func(*args, **kwargs) 2025-12-04T08:51:36.8565392Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T08:51:36.8565836Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T08:51:36.8566283Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 375, in forward 2025-12-04T08:51:36.8566701Z hidden_states = self.act(hidden_states) 2025-12-04T08:51:36.8567083Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 62, in forward 2025-12-04T08:51:36.8567573Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-12-04T08:51:36.8567833Z 2025-12-04T08:51:36.8567946Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:36.8568347Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:36.8568705Z res = mod(**inputs) 2025-12-04T08:51:36.8569092Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T08:51:36.8569522Z transformer_outputs = self.transformer( 2025-12-04T08:51:36.8569942Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T08:51:36.8570344Z outputs = block( 2025-12-04T08:51:36.8570694Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:36.8571095Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:36.8571507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:36.8571907Z return func(*args, **kwargs) 2025-12-04T08:51:36.8572307Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T08:51:36.8572754Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T08:51:36.8573191Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 376, in forward 2025-12-04T08:51:36.8573622Z hidden_states = self.c_proj(hidden_states) 2025-12-04T08:51:36.8574018Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T08:51:36.8574454Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T08:51:36.8574638Z 2025-12-04T08:51:36.8574752Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:36.8575144Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:36.8575494Z res = mod(**inputs) 2025-12-04T08:51:36.8575922Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T08:51:36.8576353Z transformer_outputs = self.transformer( 2025-12-04T08:51:36.8576772Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T08:51:36.8577207Z outputs = block( 2025-12-04T08:51:36.8577558Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:36.8577963Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:36.8578383Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:36.8578795Z return func(*args, **kwargs) 2025-12-04T08:51:36.8579198Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T08:51:36.8579666Z attn_output, self_attn_weights = self.attn( 2025-12-04T08:51:36.8580096Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:36.8580491Z return func(*args, **kwargs) 2025-12-04T08:51:36.8580872Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 313, in forward 2025-12-04T08:51:36.8581385Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-12-04T08:51:36.8581867Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T08:51:36.8582274Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T08:51:36.8582501Z 2025-12-04T08:51:36.8582590Z cudagraph partition due to non gpu ops 2025-12-04T08:51:36.8582856Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:36.8583252Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:36.8583601Z res = mod(**inputs) 2025-12-04T08:51:36.8583997Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T08:51:36.8584411Z transformer_outputs = self.transformer( 2025-12-04T08:51:36.8584803Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T08:51:36.8585195Z outputs = block( 2025-12-04T08:51:36.8585524Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:36.8585903Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:36.8586296Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:36.8586705Z return func(*args, **kwargs) 2025-12-04T08:51:36.8587112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T08:51:36.8587540Z attn_output, self_attn_weights = self.attn( 2025-12-04T08:51:36.8587961Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:36.8588370Z return func(*args, **kwargs) 2025-12-04T08:51:36.8588772Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 345, in forward 2025-12-04T08:51:36.8589207Z attn_output, attn_weights = attention_interface( 2025-12-04T08:51:36.8589694Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T08:51:36.8590217Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T08:51:36.8590416Z 2025-12-04T08:51:36.8590536Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:36.8590960Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:36.8591323Z res = mod(**inputs) 2025-12-04T08:51:36.8591720Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T08:51:36.8592166Z transformer_outputs = self.transformer( 2025-12-04T08:51:36.8592574Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T08:51:36.8592962Z outputs = block( 2025-12-04T08:51:36.8593295Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:36.8593666Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:36.8594058Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:36.8594465Z return func(*args, **kwargs) 2025-12-04T08:51:36.8594846Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T08:51:36.8595248Z attn_output, self_attn_weights = self.attn( 2025-12-04T08:51:36.8595651Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:36.8596038Z return func(*args, **kwargs) 2025-12-04T08:51:36.8596429Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 358, in forward 2025-12-04T08:51:36.8596855Z attn_output = self.c_proj(attn_output) 2025-12-04T08:51:36.8597245Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T08:51:36.8597675Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T08:51:36.8597863Z 2025-12-04T08:51:36.8597976Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:36.8598448Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:36.8598830Z res = mod(**inputs) 2025-12-04T08:51:36.8599233Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T08:51:36.8599687Z transformer_outputs = self.transformer( 2025-12-04T08:51:36.8600131Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T08:51:36.8600542Z outputs = block( 2025-12-04T08:51:36.8600892Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:36.8601293Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:36.8601714Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:36.8602130Z return func(*args, **kwargs) 2025-12-04T08:51:36.8602527Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T08:51:36.8602981Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T08:51:36.8603444Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 374, in forward 2025-12-04T08:51:36.8603881Z hidden_states = self.c_fc(hidden_states) 2025-12-04T08:51:36.8604286Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T08:51:36.8604723Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T08:51:36.8604909Z 2025-12-04T08:51:36.8605029Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:36.8605418Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:36.8605770Z res = mod(**inputs) 2025-12-04T08:51:36.8606208Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T08:51:36.8606643Z transformer_outputs = self.transformer( 2025-12-04T08:51:36.8607073Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T08:51:36.8607509Z outputs = block( 2025-12-04T08:51:36.8607864Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:36.8608254Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:36.8608670Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:36.8609056Z return func(*args, **kwargs) 2025-12-04T08:51:36.8609436Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T08:51:36.8609878Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T08:51:36.8610296Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 375, in forward 2025-12-04T08:51:36.8610699Z hidden_states = self.act(hidden_states) 2025-12-04T08:51:36.8611049Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 62, in forward 2025-12-04T08:51:36.8611521Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-12-04T08:51:36.8611769Z 2025-12-04T08:51:36.8611876Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:36.8612247Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:36.8612568Z res = mod(**inputs) 2025-12-04T08:51:36.8612940Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T08:51:36.8613350Z transformer_outputs = self.transformer( 2025-12-04T08:51:36.8613750Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T08:51:36.8614125Z outputs = block( 2025-12-04T08:51:36.8614458Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:36.8614829Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:36.8615210Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:36.8615594Z return func(*args, **kwargs) 2025-12-04T08:51:36.8615972Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T08:51:36.8616395Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T08:51:36.8616808Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 376, in forward 2025-12-04T08:51:36.8617214Z hidden_states = self.c_proj(hidden_states) 2025-12-04T08:51:36.8617589Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T08:51:36.8617991Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T08:51:36.8618171Z 2025-12-04T08:51:36.8618277Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:36.8618648Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:36.8618977Z res = mod(**inputs) 2025-12-04T08:51:36.8619336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T08:51:36.8619741Z transformer_outputs = self.transformer( 2025-12-04T08:51:36.8620184Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T08:51:36.8620571Z outputs = block( 2025-12-04T08:51:36.8621090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:36.8621555Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:36.8621984Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:36.8622401Z return func(*args, **kwargs) 2025-12-04T08:51:36.8622817Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 451, in forward 2025-12-04T08:51:36.8623258Z hidden_states = residual + feed_forward_hidden_states 2025-12-04T08:51:36.8623429Z 2025-12-04T08:51:36.8623544Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:36.8623945Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:36.8624309Z res = mod(**inputs) 2025-12-04T08:51:36.8624710Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T08:51:36.8625156Z transformer_outputs = self.transformer( 2025-12-04T08:51:36.8625601Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T08:51:36.8626018Z outputs = block( 2025-12-04T08:51:36.8626378Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:36.8626777Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:36.8627202Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:36.8627622Z return func(*args, **kwargs) 2025-12-04T08:51:36.8628039Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T08:51:36.8628480Z attn_output, self_attn_weights = self.attn( 2025-12-04T08:51:36.8628925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:36.8629359Z return func(*args, **kwargs) 2025-12-04T08:51:36.8629794Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 313, in forward 2025-12-04T08:51:36.8630373Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-12-04T08:51:36.8630887Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T08:51:36.8631333Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T08:51:36.8631520Z 2025-12-04T08:51:36.8631616Z cudagraph partition due to non gpu ops 2025-12-04T08:51:36.8631888Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:36.8632296Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:36.8632649Z res = mod(**inputs) 2025-12-04T08:51:36.8633048Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T08:51:36.8633488Z transformer_outputs = self.transformer( 2025-12-04T08:51:36.8633921Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T08:51:36.8634329Z outputs = block( 2025-12-04T08:51:36.8634698Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:36.8635104Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:36.8635525Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:36.8635993Z return func(*args, **kwargs) 2025-12-04T08:51:36.8636403Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T08:51:36.8636837Z attn_output, self_attn_weights = self.attn( 2025-12-04T08:51:36.8637273Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:36.8637681Z return func(*args, **kwargs) 2025-12-04T08:51:36.8638082Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 345, in forward 2025-12-04T08:51:36.8638644Z attn_output, attn_weights = attention_interface( 2025-12-04T08:51:36.8639170Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T08:51:36.8639740Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T08:51:36.8639939Z 2025-12-04T08:51:36.8640064Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:36.8640468Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:36.8640822Z res = mod(**inputs) 2025-12-04T08:51:36.8641224Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T08:51:36.8641664Z transformer_outputs = self.transformer( 2025-12-04T08:51:36.8642082Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T08:51:36.8642491Z outputs = block( 2025-12-04T08:51:36.8642842Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:36.8643241Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:36.8643659Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:36.8644040Z return func(*args, **kwargs) 2025-12-04T08:51:36.8644420Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T08:51:36.8644826Z attn_output, self_attn_weights = self.attn( 2025-12-04T08:51:36.8645230Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:36.8645616Z return func(*args, **kwargs) 2025-12-04T08:51:36.8645997Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 358, in forward 2025-12-04T08:51:36.8646400Z attn_output = self.c_proj(attn_output) 2025-12-04T08:51:36.8646771Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T08:51:36.8647186Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T08:51:36.8647360Z 2025-12-04T08:51:36.8647485Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:36.8647839Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:36.8648167Z res = mod(**inputs) 2025-12-04T08:51:36.8648527Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T08:51:36.8648921Z transformer_outputs = self.transformer( 2025-12-04T08:51:36.8649312Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T08:51:36.8649694Z outputs = block( 2025-12-04T08:51:36.8650016Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:36.8650373Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:36.8650800Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:36.8651183Z return func(*args, **kwargs) 2025-12-04T08:51:36.8651550Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T08:51:36.8652004Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T08:51:36.8652412Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 374, in forward 2025-12-04T08:51:36.8652804Z hidden_states = self.c_fc(hidden_states) 2025-12-04T08:51:36.8653157Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T08:51:36.8653556Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T08:51:36.8653728Z 2025-12-04T08:51:36.8653839Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:36.8654225Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:36.8654546Z res = mod(**inputs) 2025-12-04T08:51:36.8654905Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T08:51:36.8655328Z transformer_outputs = self.transformer( 2025-12-04T08:51:36.8655709Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T08:51:36.8656087Z outputs = block( 2025-12-04T08:51:36.8656415Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:36.8656777Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:36.8657150Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:36.8657525Z return func(*args, **kwargs) 2025-12-04T08:51:36.8657897Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T08:51:36.8658309Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T08:51:36.8658716Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 375, in forward 2025-12-04T08:51:36.8659113Z hidden_states = self.act(hidden_states) 2025-12-04T08:51:36.8659470Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 62, in forward 2025-12-04T08:51:36.8659919Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-12-04T08:51:36.8660163Z 2025-12-04T08:51:36.8660269Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:36.8660632Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:36.8660975Z res = mod(**inputs) 2025-12-04T08:51:36.8661349Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T08:51:36.8661745Z transformer_outputs = self.transformer( 2025-12-04T08:51:36.8662140Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T08:51:36.8662522Z outputs = block( 2025-12-04T08:51:36.8662856Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:36.8663261Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:36.8663653Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:36.8664034Z return func(*args, **kwargs) 2025-12-04T08:51:36.8664415Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T08:51:36.8664877Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T08:51:36.8665302Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 376, in forward 2025-12-04T08:51:36.8665716Z hidden_states = self.c_proj(hidden_states) 2025-12-04T08:51:36.8666150Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T08:51:36.8666564Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T08:51:36.8666743Z 2025-12-04T08:51:36.8666851Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:36.8667220Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:36.8667555Z res = mod(**inputs) 2025-12-04T08:51:36.8667922Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T08:51:36.8668345Z transformer_outputs = self.transformer( 2025-12-04T08:51:36.8668762Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T08:51:36.8669156Z outputs = block( 2025-12-04T08:51:36.8669493Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:36.8669871Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:36.8670270Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:36.8670664Z return func(*args, **kwargs) 2025-12-04T08:51:36.8671063Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T08:51:36.8671506Z attn_output, self_attn_weights = self.attn( 2025-12-04T08:51:36.8671939Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:36.8672347Z return func(*args, **kwargs) 2025-12-04T08:51:36.8672726Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 313, in forward 2025-12-04T08:51:36.8673240Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-12-04T08:51:36.8673720Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T08:51:36.8674130Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T08:51:36.8674317Z 2025-12-04T08:51:36.8674407Z cudagraph partition due to non gpu ops 2025-12-04T08:51:36.8674663Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:36.8675043Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:36.8675378Z res = mod(**inputs) 2025-12-04T08:51:36.8675761Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T08:51:36.8676180Z transformer_outputs = self.transformer( 2025-12-04T08:51:36.8676580Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T08:51:36.8676971Z outputs = block( 2025-12-04T08:51:36.8677317Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:36.8677718Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:36.8678131Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:36.8678618Z return func(*args, **kwargs) 2025-12-04T08:51:36.8679033Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T08:51:36.8679475Z attn_output, self_attn_weights = self.attn( 2025-12-04T08:51:36.8679955Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:36.8680370Z return func(*args, **kwargs) 2025-12-04T08:51:36.8680796Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 345, in forward 2025-12-04T08:51:36.8681235Z attn_output, attn_weights = attention_interface( 2025-12-04T08:51:36.8681727Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T08:51:36.8682257Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T08:51:36.8682454Z 2025-12-04T08:51:36.8682576Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:36.8682965Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:36.8683339Z res = mod(**inputs) 2025-12-04T08:51:36.8683738Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T08:51:36.8684178Z transformer_outputs = self.transformer( 2025-12-04T08:51:36.8684601Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T08:51:36.8685011Z outputs = block( 2025-12-04T08:51:36.8685364Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:36.8685755Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:36.8686175Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:36.8686585Z return func(*args, **kwargs) 2025-12-04T08:51:36.8686993Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T08:51:36.8687427Z attn_output, self_attn_weights = self.attn( 2025-12-04T08:51:36.8687849Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:36.8688257Z return func(*args, **kwargs) 2025-12-04T08:51:36.8688656Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 358, in forward 2025-12-04T08:51:36.8689082Z attn_output = self.c_proj(attn_output) 2025-12-04T08:51:36.8689475Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T08:51:36.8689911Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T08:51:36.8690107Z 2025-12-04T08:51:36.8690215Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:36.8690593Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:36.8690932Z res = mod(**inputs) 2025-12-04T08:51:36.8691293Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T08:51:36.8691702Z transformer_outputs = self.transformer( 2025-12-04T08:51:36.8692106Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T08:51:36.8692499Z outputs = block( 2025-12-04T08:51:36.8692825Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:36.8693205Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:36.8693600Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:36.8693992Z return func(*args, **kwargs) 2025-12-04T08:51:36.8694399Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T08:51:36.8694831Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T08:51:36.8695256Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 374, in forward 2025-12-04T08:51:36.8695670Z hidden_states = self.c_fc(hidden_states) 2025-12-04T08:51:36.8696042Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T08:51:36.8696473Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T08:51:36.8696659Z 2025-12-04T08:51:36.8696783Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:36.8697175Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:36.8697530Z res = mod(**inputs) 2025-12-04T08:51:36.8697926Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T08:51:36.8698333Z transformer_outputs = self.transformer( 2025-12-04T08:51:36.8698723Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T08:51:36.8699109Z outputs = block( 2025-12-04T08:51:36.8699438Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:36.8699812Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:36.8700198Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:36.8700584Z return func(*args, **kwargs) 2025-12-04T08:51:36.8700964Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T08:51:36.8701381Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T08:51:36.8701803Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 375, in forward 2025-12-04T08:51:36.8702207Z hidden_states = self.act(hidden_states) 2025-12-04T08:51:36.8702557Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 62, in forward 2025-12-04T08:51:36.8703030Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-12-04T08:51:36.8703276Z 2025-12-04T08:51:36.8703382Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:36.8703772Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:36.8704121Z res = mod(**inputs) 2025-12-04T08:51:36.8704502Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T08:51:36.8704909Z transformer_outputs = self.transformer( 2025-12-04T08:51:36.8705309Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T08:51:36.8705685Z outputs = block( 2025-12-04T08:51:36.8706020Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:36.8706393Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:36.8706775Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:36.8707157Z return func(*args, **kwargs) 2025-12-04T08:51:36.8707535Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T08:51:36.8707957Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T08:51:36.8708366Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 376, in forward 2025-12-04T08:51:36.8708814Z hidden_states = self.c_proj(hidden_states) 2025-12-04T08:51:36.8709193Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T08:51:36.8709601Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T08:51:36.8709793Z 2025-12-04T08:51:36.8709900Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:36.8710272Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:36.8710644Z res = mod(**inputs) 2025-12-04T08:51:36.8710995Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T08:51:36.8711428Z transformer_outputs = self.transformer( 2025-12-04T08:51:36.8711825Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T08:51:36.8712228Z outputs = block( 2025-12-04T08:51:36.8712565Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:36.8712930Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:36.8713312Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:36.8713680Z return func(*args, **kwargs) 2025-12-04T08:51:36.8714051Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 451, in forward 2025-12-04T08:51:36.8714466Z hidden_states = residual + feed_forward_hidden_states 2025-12-04T08:51:36.8714627Z 2025-12-04T08:51:36.8714740Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:36.8715096Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:36.8715423Z res = mod(**inputs) 2025-12-04T08:51:36.8715784Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T08:51:36.8716179Z transformer_outputs = self.transformer( 2025-12-04T08:51:36.8716564Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T08:51:36.8716948Z outputs = block( 2025-12-04T08:51:36.8717277Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:36.8717640Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:36.8718031Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:36.8718501Z return func(*args, **kwargs) 2025-12-04T08:51:36.8718924Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T08:51:36.8719376Z attn_output, self_attn_weights = self.attn( 2025-12-04T08:51:36.8719806Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:36.8720206Z return func(*args, **kwargs) 2025-12-04T08:51:36.8720572Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 313, in forward 2025-12-04T08:51:36.8721240Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-12-04T08:51:36.8721726Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T08:51:36.8722143Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T08:51:36.8722323Z 2025-12-04T08:51:36.8722414Z cudagraph partition due to non gpu ops 2025-12-04T08:51:36.8722680Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:36.8723088Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:36.8723494Z res = mod(**inputs) 2025-12-04T08:51:36.8723875Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T08:51:36.8724334Z transformer_outputs = self.transformer( 2025-12-04T08:51:36.8724760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T08:51:36.8725154Z outputs = block( 2025-12-04T08:51:36.8725488Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:36.8725866Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:36.8726256Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:36.8726660Z return func(*args, **kwargs) 2025-12-04T08:51:36.8727036Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T08:51:36.8727457Z attn_output, self_attn_weights = self.attn( 2025-12-04T08:51:36.8727870Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:36.8728276Z return func(*args, **kwargs) 2025-12-04T08:51:36.8728672Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 345, in forward 2025-12-04T08:51:36.8729112Z attn_output, attn_weights = attention_interface( 2025-12-04T08:51:36.8729590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T08:51:36.8730113Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T08:51:36.8730309Z 2025-12-04T08:51:36.8730429Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:36.8730825Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:36.8731171Z res = mod(**inputs) 2025-12-04T08:51:36.8731561Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T08:51:36.8731994Z transformer_outputs = self.transformer( 2025-12-04T08:51:36.8732408Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T08:51:36.8732814Z outputs = block( 2025-12-04T08:51:36.8733163Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:36.8733560Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:36.8733968Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:36.8734365Z return func(*args, **kwargs) 2025-12-04T08:51:36.8734745Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T08:51:36.8735148Z attn_output, self_attn_weights = self.attn( 2025-12-04T08:51:36.8735547Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:36.8735938Z return func(*args, **kwargs) 2025-12-04T08:51:36.8736336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 358, in forward 2025-12-04T08:51:36.8736755Z attn_output = self.c_proj(attn_output) 2025-12-04T08:51:36.8737144Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T08:51:36.8737578Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T08:51:36.8737769Z 2025-12-04T08:51:36.8737889Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:36.8738330Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:36.8738687Z res = mod(**inputs) 2025-12-04T08:51:36.8739079Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T08:51:36.8739534Z transformer_outputs = self.transformer( 2025-12-04T08:51:36.8739940Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T08:51:36.8740336Z outputs = block( 2025-12-04T08:51:36.8740673Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:36.8741051Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:36.8741450Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:36.8741856Z return func(*args, **kwargs) 2025-12-04T08:51:36.8742233Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T08:51:36.8742662Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T08:51:36.8743084Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 374, in forward 2025-12-04T08:51:36.8743507Z hidden_states = self.c_fc(hidden_states) 2025-12-04T08:51:36.8743891Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T08:51:36.8744322Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T08:51:36.8744514Z 2025-12-04T08:51:36.8744628Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:36.8745033Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:36.8745379Z res = mod(**inputs) 2025-12-04T08:51:36.8745771Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T08:51:36.8746208Z transformer_outputs = self.transformer( 2025-12-04T08:51:36.8746615Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T08:51:36.8747011Z outputs = block( 2025-12-04T08:51:36.8747360Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:36.8747757Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:36.8748163Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:36.8748569Z return func(*args, **kwargs) 2025-12-04T08:51:36.8748972Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T08:51:36.8749440Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T08:51:36.8749890Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 375, in forward 2025-12-04T08:51:36.8750315Z hidden_states = self.act(hidden_states) 2025-12-04T08:51:36.8750700Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 62, in forward 2025-12-04T08:51:36.8751192Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-12-04T08:51:36.8751456Z 2025-12-04T08:51:36.8751569Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:36.8751964Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:36.8752318Z res = mod(**inputs) 2025-12-04T08:51:36.8752697Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T08:51:36.8753172Z transformer_outputs = self.transformer( 2025-12-04T08:51:36.8753594Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T08:51:36.8754012Z outputs = block( 2025-12-04T08:51:36.8754362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:36.8754813Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:36.8755225Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:36.8755629Z return func(*args, **kwargs) 2025-12-04T08:51:36.8756035Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T08:51:36.8756457Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T08:51:36.8756902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 376, in forward 2025-12-04T08:51:36.8757325Z hidden_states = self.c_proj(hidden_states) 2025-12-04T08:51:36.8757729Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T08:51:36.8758173Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T08:51:36.8758430Z 2025-12-04T08:51:36.8758553Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:36.8758951Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:36.8759313Z res = mod(**inputs) 2025-12-04T08:51:36.8759726Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T08:51:36.8760176Z transformer_outputs = self.transformer( 2025-12-04T08:51:36.8760627Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T08:51:36.8761078Z outputs = block( 2025-12-04T08:51:36.8761403Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:36.8761781Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:36.8762175Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:36.8762561Z return func(*args, **kwargs) 2025-12-04T08:51:36.8762939Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T08:51:36.8763354Z attn_output, self_attn_weights = self.attn( 2025-12-04T08:51:36.8763779Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:36.8764189Z return func(*args, **kwargs) 2025-12-04T08:51:36.8764645Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 313, in forward 2025-12-04T08:51:36.8765153Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-12-04T08:51:36.8765634Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T08:51:36.8766040Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T08:51:36.8766221Z 2025-12-04T08:51:36.8766307Z cudagraph partition due to non gpu ops 2025-12-04T08:51:36.8766557Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:36.8766926Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:36.8767269Z res = mod(**inputs) 2025-12-04T08:51:36.8767660Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T08:51:36.8768137Z transformer_outputs = self.transformer( 2025-12-04T08:51:36.8768558Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T08:51:36.8768939Z outputs = block( 2025-12-04T08:51:36.8769291Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:36.8769683Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:36.8770090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:36.8770500Z return func(*args, **kwargs) 2025-12-04T08:51:36.8770908Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T08:51:36.8771319Z attn_output, self_attn_weights = self.attn( 2025-12-04T08:51:36.8771760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:36.8772163Z return func(*args, **kwargs) 2025-12-04T08:51:36.8772563Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 345, in forward 2025-12-04T08:51:36.8773000Z attn_output, attn_weights = attention_interface( 2025-12-04T08:51:36.8773487Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T08:51:36.8774014Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T08:51:36.8774211Z 2025-12-04T08:51:36.8774340Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:36.8774729Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:36.8775082Z res = mod(**inputs) 2025-12-04T08:51:36.8775478Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T08:51:36.8775911Z transformer_outputs = self.transformer( 2025-12-04T08:51:36.8776328Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T08:51:36.8776734Z outputs = block( 2025-12-04T08:51:36.8777081Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:36.8777469Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:36.8777883Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:36.8778289Z return func(*args, **kwargs) 2025-12-04T08:51:36.8778689Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T08:51:36.8779116Z attn_output, self_attn_weights = self.attn( 2025-12-04T08:51:36.8779539Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:36.8779944Z return func(*args, **kwargs) 2025-12-04T08:51:36.8780334Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 358, in forward 2025-12-04T08:51:36.8780435Z attn_output = self.c_proj(attn_output) 2025-12-04T08:51:36.8780671Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T08:51:36.8780806Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T08:51:36.8780810Z 2025-12-04T08:51:36.8780924Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:36.8781155Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:36.8781229Z res = mod(**inputs) 2025-12-04T08:51:36.8781517Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T08:51:36.8781609Z transformer_outputs = self.transformer( 2025-12-04T08:51:36.8781858Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T08:51:36.8781942Z outputs = block( 2025-12-04T08:51:36.8782176Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:36.8782258Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:36.8782506Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:36.8782583Z return func(*args, **kwargs) 2025-12-04T08:51:36.8782837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T08:51:36.8782985Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T08:51:36.8783243Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 374, in forward 2025-12-04T08:51:36.8783330Z hidden_states = self.c_fc(hidden_states) 2025-12-04T08:51:36.8783566Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T08:51:36.8783694Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T08:51:36.8783697Z 2025-12-04T08:51:36.8783816Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:36.8784024Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:36.8784094Z res = mod(**inputs) 2025-12-04T08:51:36.8784362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T08:51:36.8784452Z transformer_outputs = self.transformer( 2025-12-04T08:51:36.8784710Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T08:51:36.8784788Z outputs = block( 2025-12-04T08:51:36.8785019Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:36.8785111Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:36.8785365Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:36.8785443Z return func(*args, **kwargs) 2025-12-04T08:51:36.8785722Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T08:51:36.8785837Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T08:51:36.8786111Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 375, in forward 2025-12-04T08:51:36.8786214Z hidden_states = self.act(hidden_states) 2025-12-04T08:51:36.8786451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 62, in forward 2025-12-04T08:51:36.8786659Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-12-04T08:51:36.8786665Z 2025-12-04T08:51:36.8786782Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:36.8787003Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:36.8787086Z res = mod(**inputs) 2025-12-04T08:51:36.8787365Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T08:51:36.8787467Z transformer_outputs = self.transformer( 2025-12-04T08:51:36.8787739Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T08:51:36.8787814Z outputs = block( 2025-12-04T08:51:36.8788103Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:36.8788189Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:36.8788475Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:36.8788558Z return func(*args, **kwargs) 2025-12-04T08:51:36.8788840Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T08:51:36.8788959Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T08:51:36.8789228Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 376, in forward 2025-12-04T08:51:36.8789321Z hidden_states = self.c_proj(hidden_states) 2025-12-04T08:51:36.8789589Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T08:51:36.8789715Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T08:51:36.8789719Z 2025-12-04T08:51:36.8789840Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:36.8790056Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:36.8790128Z res = mod(**inputs) 2025-12-04T08:51:36.8790408Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T08:51:36.8790496Z transformer_outputs = self.transformer( 2025-12-04T08:51:36.8790767Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T08:51:36.8790843Z outputs = block( 2025-12-04T08:51:36.8791090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:36.8791184Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:36.8791446Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:36.8791520Z return func(*args, **kwargs) 2025-12-04T08:51:36.8791796Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 451, in forward 2025-12-04T08:51:36.8791912Z hidden_states = residual + feed_forward_hidden_states 2025-12-04T08:51:36.8791916Z 2025-12-04T08:51:36.8792032Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:36.8792245Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:36.8792317Z res = mod(**inputs) 2025-12-04T08:51:36.8792598Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T08:51:36.8792690Z transformer_outputs = self.transformer( 2025-12-04T08:51:36.8792963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T08:51:36.8793038Z outputs = block( 2025-12-04T08:51:36.8793277Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:36.8793369Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:36.8793632Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:36.8793706Z return func(*args, **kwargs) 2025-12-04T08:51:36.8793979Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T08:51:36.8794075Z attn_output, self_attn_weights = self.attn( 2025-12-04T08:51:36.8794337Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:36.8794454Z return func(*args, **kwargs) 2025-12-04T08:51:36.8794725Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 313, in forward 2025-12-04T08:51:36.8794952Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-12-04T08:51:36.8795194Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T08:51:36.8795317Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T08:51:36.8795321Z 2025-12-04T08:51:36.8795419Z cudagraph partition due to non gpu ops 2025-12-04T08:51:36.8795530Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:36.8795753Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:36.8795840Z res = mod(**inputs) 2025-12-04T08:51:36.8796122Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T08:51:36.8796220Z transformer_outputs = self.transformer( 2025-12-04T08:51:36.8796492Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T08:51:36.8796563Z outputs = block( 2025-12-04T08:51:36.8796812Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:36.8796896Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:36.8797170Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:36.8797243Z return func(*args, **kwargs) 2025-12-04T08:51:36.8797508Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T08:51:36.8797612Z attn_output, self_attn_weights = self.attn( 2025-12-04T08:51:36.8797879Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:36.8797961Z return func(*args, **kwargs) 2025-12-04T08:51:36.8798307Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 345, in forward 2025-12-04T08:51:36.8798423Z attn_output, attn_weights = attention_interface( 2025-12-04T08:51:36.8798775Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T08:51:36.8798918Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T08:51:36.8798923Z 2025-12-04T08:51:36.8799037Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:36.8799265Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:36.8799342Z res = mod(**inputs) 2025-12-04T08:51:36.8799633Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T08:51:36.8799726Z transformer_outputs = self.transformer( 2025-12-04T08:51:36.8800012Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T08:51:36.8800089Z outputs = block( 2025-12-04T08:51:36.8800331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:36.8800417Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:36.8800690Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:36.8800766Z return func(*args, **kwargs) 2025-12-04T08:51:36.8801045Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T08:51:36.8801182Z attn_output, self_attn_weights = self.attn( 2025-12-04T08:51:36.8801445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:36.8801546Z return func(*args, **kwargs) 2025-12-04T08:51:36.8801815Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 358, in forward 2025-12-04T08:51:36.8801911Z attn_output = self.c_proj(attn_output) 2025-12-04T08:51:36.8802149Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T08:51:36.8802275Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T08:51:36.8802279Z 2025-12-04T08:51:36.8802398Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:36.8802613Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:36.8802702Z res = mod(**inputs) 2025-12-04T08:51:36.8802983Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T08:51:36.8803074Z transformer_outputs = self.transformer( 2025-12-04T08:51:36.8803354Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T08:51:36.8803423Z outputs = block( 2025-12-04T08:51:36.8803665Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:36.8803759Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:36.8804026Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:36.8804108Z return func(*args, **kwargs) 2025-12-04T08:51:36.8804381Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T08:51:36.8804498Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T08:51:36.8804775Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 374, in forward 2025-12-04T08:51:36.8804864Z hidden_states = self.c_fc(hidden_states) 2025-12-04T08:51:36.8805101Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T08:51:36.8805234Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T08:51:36.8805238Z 2025-12-04T08:51:36.8805349Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:36.8805569Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:36.8805642Z res = mod(**inputs) 2025-12-04T08:51:36.8805916Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T08:51:36.8806017Z transformer_outputs = self.transformer( 2025-12-04T08:51:36.8806287Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T08:51:36.8806364Z outputs = block( 2025-12-04T08:51:36.8806612Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:36.8806695Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:36.8806977Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:36.8807050Z return func(*args, **kwargs) 2025-12-04T08:51:36.8807317Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T08:51:36.8807435Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T08:51:36.8807736Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 375, in forward 2025-12-04T08:51:36.8807831Z hidden_states = self.act(hidden_states) 2025-12-04T08:51:36.8808061Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 62, in forward 2025-12-04T08:51:36.8808282Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-12-04T08:51:36.8808286Z 2025-12-04T08:51:36.8808405Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:36.8808622Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:36.8808699Z res = mod(**inputs) 2025-12-04T08:51:36.8808982Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T08:51:36.8809088Z transformer_outputs = self.transformer( 2025-12-04T08:51:36.8809417Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T08:51:36.8809483Z outputs = block( 2025-12-04T08:51:36.8809709Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:36.8809800Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:36.8810048Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:36.8810127Z return func(*args, **kwargs) 2025-12-04T08:51:36.8810385Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T08:51:36.8810489Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T08:51:36.8810751Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 376, in forward 2025-12-04T08:51:36.8810842Z hidden_states = self.c_proj(hidden_states) 2025-12-04T08:51:36.8811067Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T08:51:36.8811193Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T08:51:36.8811198Z 2025-12-04T08:51:36.8811301Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:36.8811510Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:36.8811579Z res = mod(**inputs) 2025-12-04T08:51:36.8811838Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T08:51:36.8811930Z transformer_outputs = self.transformer( 2025-12-04T08:51:36.8812185Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T08:51:36.8812257Z outputs = block( 2025-12-04T08:51:36.8812486Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:36.8812567Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:36.8812822Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:36.8812893Z return func(*args, **kwargs) 2025-12-04T08:51:36.8813148Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T08:51:36.8813246Z attn_output, self_attn_weights = self.attn( 2025-12-04T08:51:36.8813495Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:36.8813571Z return func(*args, **kwargs) 2025-12-04T08:51:36.8813820Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 313, in forward 2025-12-04T08:51:36.8814047Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-12-04T08:51:36.8814280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T08:51:36.8814417Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T08:51:36.8814421Z 2025-12-04T08:51:36.8814512Z cudagraph partition due to non gpu ops 2025-12-04T08:51:36.8814618Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:36.8814824Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:36.8814897Z res = mod(**inputs) 2025-12-04T08:51:36.8815157Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T08:51:36.8815240Z transformer_outputs = self.transformer( 2025-12-04T08:51:36.8815536Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T08:51:36.8815602Z outputs = block( 2025-12-04T08:51:36.8815837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:36.8815918Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:36.8816169Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:36.8816246Z return func(*args, **kwargs) 2025-12-04T08:51:36.8816501Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T08:51:36.8816589Z attn_output, self_attn_weights = self.attn( 2025-12-04T08:51:36.8816853Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:36.8816926Z return func(*args, **kwargs) 2025-12-04T08:51:36.8817195Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 345, in forward 2025-12-04T08:51:36.8817296Z attn_output, attn_weights = attention_interface( 2025-12-04T08:51:36.8817604Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T08:51:36.8817746Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T08:51:36.8817749Z 2025-12-04T08:51:36.8817855Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:36.8818065Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:36.8818133Z res = mod(**inputs) 2025-12-04T08:51:36.8818391Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T08:51:36.8818487Z transformer_outputs = self.transformer( 2025-12-04T08:51:36.8818749Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T08:51:36.8818816Z outputs = block( 2025-12-04T08:51:36.8819052Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:36.8819133Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:36.8819391Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:36.8819463Z return func(*args, **kwargs) 2025-12-04T08:51:36.8819718Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T08:51:36.8819815Z attn_output, self_attn_weights = self.attn( 2025-12-04T08:51:36.8820067Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:36.8820180Z return func(*args, **kwargs) 2025-12-04T08:51:36.8820438Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 358, in forward 2025-12-04T08:51:36.8820523Z attn_output = self.c_proj(attn_output) 2025-12-04T08:51:36.8820922Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T08:51:36.8821046Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T08:51:36.8821050Z 2025-12-04T08:51:36.8821157Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:36.8821369Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:36.8821437Z res = mod(**inputs) 2025-12-04T08:51:36.8821702Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T08:51:36.8821834Z transformer_outputs = self.transformer( 2025-12-04T08:51:36.8822088Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T08:51:36.8822164Z outputs = block( 2025-12-04T08:51:36.8822394Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:36.8822473Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:36.8822729Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:36.8822799Z return func(*args, **kwargs) 2025-12-04T08:51:36.8823058Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T08:51:36.8823162Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T08:51:36.8823416Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 374, in forward 2025-12-04T08:51:36.8823508Z hidden_states = self.c_fc(hidden_states) 2025-12-04T08:51:36.8823730Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T08:51:36.8823854Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T08:51:36.8823858Z 2025-12-04T08:51:36.8823963Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:36.8824163Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:36.8824238Z res = mod(**inputs) 2025-12-04T08:51:36.8824496Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T08:51:36.8824580Z transformer_outputs = self.transformer( 2025-12-04T08:51:36.8824841Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T08:51:36.8824908Z outputs = block( 2025-12-04T08:51:36.8825145Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:36.8825224Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:36.8825481Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:36.8825559Z return func(*args, **kwargs) 2025-12-04T08:51:36.8825821Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T08:51:36.8825933Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T08:51:36.8826184Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 375, in forward 2025-12-04T08:51:36.8826267Z hidden_states = self.act(hidden_states) 2025-12-04T08:51:36.8826582Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 62, in forward 2025-12-04T08:51:36.8826768Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-12-04T08:51:36.8826772Z 2025-12-04T08:51:36.8826878Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:36.8827116Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:36.8827182Z res = mod(**inputs) 2025-12-04T08:51:36.8827446Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T08:51:36.8827532Z transformer_outputs = self.transformer( 2025-12-04T08:51:36.8827802Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T08:51:36.8827877Z outputs = block( 2025-12-04T08:51:36.8828139Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:36.8828232Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:36.8828495Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:36.8828573Z return func(*args, **kwargs) 2025-12-04T08:51:36.8828853Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T08:51:36.8828963Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T08:51:36.8829232Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 376, in forward 2025-12-04T08:51:36.8829332Z hidden_states = self.c_proj(hidden_states) 2025-12-04T08:51:36.8829569Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T08:51:36.8829702Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T08:51:36.8829709Z 2025-12-04T08:51:36.8829817Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:36.8830029Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:36.8830108Z res = mod(**inputs) 2025-12-04T08:51:36.8830386Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T08:51:36.8830482Z transformer_outputs = self.transformer( 2025-12-04T08:51:36.8830749Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T08:51:36.8830816Z outputs = block( 2025-12-04T08:51:36.8831063Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:36.8831147Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:36.8831411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:36.8831494Z return func(*args, **kwargs) 2025-12-04T08:51:36.8831760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 451, in forward 2025-12-04T08:51:36.8831882Z hidden_states = residual + feed_forward_hidden_states 2025-12-04T08:51:36.8831886Z 2025-12-04T08:51:36.8831995Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:36.8832208Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:36.8832287Z res = mod(**inputs) 2025-12-04T08:51:36.8832559Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T08:51:36.8832647Z transformer_outputs = self.transformer( 2025-12-04T08:51:36.8833005Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T08:51:36.8833076Z outputs = block( 2025-12-04T08:51:36.8833323Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:36.8833424Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:36.8833690Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:36.8833773Z return func(*args, **kwargs) 2025-12-04T08:51:36.8834040Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T08:51:36.8834144Z attn_output, self_attn_weights = self.attn( 2025-12-04T08:51:36.8834405Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:36.8834498Z return func(*args, **kwargs) 2025-12-04T08:51:36.8834778Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 313, in forward 2025-12-04T08:51:36.8834980Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-12-04T08:51:36.8835219Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T08:51:36.8835355Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T08:51:36.8835359Z 2025-12-04T08:51:36.8835447Z cudagraph partition due to non gpu ops 2025-12-04T08:51:36.8835566Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:36.8835780Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:36.8835851Z res = mod(**inputs) 2025-12-04T08:51:36.8836132Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T08:51:36.8836226Z transformer_outputs = self.transformer( 2025-12-04T08:51:36.8836505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T08:51:36.8836575Z outputs = block( 2025-12-04T08:51:36.8836817Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:36.8836909Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:36.8837172Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:36.8837247Z return func(*args, **kwargs) 2025-12-04T08:51:36.8837524Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T08:51:36.8837617Z attn_output, self_attn_weights = self.attn( 2025-12-04T08:51:36.8837896Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:36.8837972Z return func(*args, **kwargs) 2025-12-04T08:51:36.8838288Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 345, in forward 2025-12-04T08:51:36.8838416Z attn_output, attn_weights = attention_interface( 2025-12-04T08:51:36.8838745Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T08:51:36.8838895Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T08:51:36.8838908Z 2025-12-04T08:51:36.8839022Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:36.8839243Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:36.8839332Z res = mod(**inputs) 2025-12-04T08:51:36.8839621Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T08:51:36.8839750Z transformer_outputs = self.transformer( 2025-12-04T08:51:36.8840044Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T08:51:36.8840131Z outputs = block( 2025-12-04T08:51:36.8840381Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:36.8840466Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:36.8840727Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:36.8840808Z return func(*args, **kwargs) 2025-12-04T08:51:36.8841078Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 413, in forward 2025-12-04T08:51:36.8841189Z attn_output, self_attn_weights = self.attn( 2025-12-04T08:51:36.8841460Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:36.8841532Z return func(*args, **kwargs) 2025-12-04T08:51:36.8841805Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 358, in forward 2025-12-04T08:51:36.8841898Z attn_output = self.c_proj(attn_output) 2025-12-04T08:51:36.8842133Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T08:51:36.8842266Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T08:51:36.8842270Z 2025-12-04T08:51:36.8842387Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:36.8842594Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:36.8842661Z res = mod(**inputs) 2025-12-04T08:51:36.8842919Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T08:51:36.8843009Z transformer_outputs = self.transformer( 2025-12-04T08:51:36.8843275Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T08:51:36.8843344Z outputs = block( 2025-12-04T08:51:36.8843589Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:36.8843673Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:36.8843940Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:36.8844014Z return func(*args, **kwargs) 2025-12-04T08:51:36.8844280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T08:51:36.8844401Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T08:51:36.8844668Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 374, in forward 2025-12-04T08:51:36.8844756Z hidden_states = self.c_fc(hidden_states) 2025-12-04T08:51:36.8845000Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T08:51:36.8845117Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T08:51:36.8845121Z 2025-12-04T08:51:36.8845229Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:36.8845428Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:36.8845495Z res = mod(**inputs) 2025-12-04T08:51:36.8845758Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T08:51:36.8845843Z transformer_outputs = self.transformer( 2025-12-04T08:51:36.8846136Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T08:51:36.8846205Z outputs = block( 2025-12-04T08:51:36.8846447Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:36.8846555Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:36.8846817Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:36.8846891Z return func(*args, **kwargs) 2025-12-04T08:51:36.8847165Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T08:51:36.8847274Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T08:51:36.8847546Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 375, in forward 2025-12-04T08:51:36.8847648Z hidden_states = self.act(hidden_states) 2025-12-04T08:51:36.8847882Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 62, in forward 2025-12-04T08:51:36.8848085Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-12-04T08:51:36.8848091Z 2025-12-04T08:51:36.8848200Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:36.8848420Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:36.8848489Z res = mod(**inputs) 2025-12-04T08:51:36.8848757Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1365, in forward 2025-12-04T08:51:36.8848852Z transformer_outputs = self.transformer( 2025-12-04T08:51:36.8849117Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 925, in forward 2025-12-04T08:51:36.8849188Z outputs = block( 2025-12-04T08:51:36.8849436Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:51:36.8849519Z return super().__call__(*args, **kwargs) 2025-12-04T08:51:36.8849803Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:51:36.8849875Z return func(*args, **kwargs) 2025-12-04T08:51:36.8850127Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 449, in forward 2025-12-04T08:51:36.8850238Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-12-04T08:51:36.8850495Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 376, in forward 2025-12-04T08:51:36.8850595Z hidden_states = self.c_proj(hidden_states) 2025-12-04T08:51:36.8850837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 122, in forward 2025-12-04T08:51:36.8850960Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-12-04T08:51:36.8850963Z 2025-12-04T08:51:36.8851080Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:36.8851297Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:36.8851367Z res = mod(**inputs) 2025-12-04T08:51:36.8851657Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1379, in forward 2025-12-04T08:51:36.8851743Z logits = self.score(hidden_states) 2025-12-04T08:51:36.8851747Z 2025-12-04T08:51:36.8851864Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:36.8852078Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:36.8852151Z res = mod(**inputs) 2025-12-04T08:51:36.8852465Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1422, in forward 2025-12-04T08:51:36.8852628Z loss = loss_fct(pooled_logits.view(-1, self.num_labels), labels.view(-1)) 2025-12-04T08:51:36.8852632Z 2025-12-04T08:51:36.8852752Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:51:36.8852978Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:51:36.8853047Z res = mod(**inputs) 2025-12-04T08:51:36.8853324Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1422, in forward 2025-12-04T08:51:36.8853476Z loss = loss_fct(pooled_logits.view(-1, self.num_labels), labels.view(-1)) 2025-12-04T08:51:36.8853481Z 2025-12-04T08:51:51.0914929Z Compilation time (from dynamo_timed): 19.558905777 2025-12-04T08:51:51.0915309Z pass 2025-12-04T08:51:51.0916329Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T08:51:51.0917258Z TIMING: _recursive_pre_grad_passes:0.01326 _recursive_joint_graph_passes:0.59846 _recursive_post_grad_passes:0.06666 async_compile.wait:1.51104 code_gen:10.54507 inductor_compile:11.71358 backend_compile:14.85839 gc:0.00096 entire_frame_compile:19.55891 total_wall_time:19.55891 2025-12-04T08:51:51.0918475Z STATS: call_* op count: 1126 | FakeTensorMode.__torch_dispatch__:7443 | FakeTensor.__torch_dispatch__:4239 | ProxyTorchDispatchMode.__torch_dispatch__:1681 2025-12-04T08:51:51.0919054Z Dynamo produced 2 graphs covering 1126 ops with 0 graph breaks (0 unique) 2025-12-04T08:51:53.6166959Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/__init__.py:65: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-12-04T08:51:53.6167959Z import pynvml # type: ignore[import] 2025-12-04T08:51:57.1075734Z 2025-12-04T08:51:58.2736266Z loading model: 0it [00:00, ?it/s] 2025-12-04T08:51:58.2736569Z loading model: 0it [00:01, ?it/s] 2025-12-04T08:51:58.2744090Z cpu eval GoogleFnet 2025-12-04T08:51:59.3805602Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T08:51:59.8397858Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T08:52:00.3049907Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T08:52:05.7067548Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:05.7068032Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:05.7068430Z res = mod(**inputs) 2025-12-04T08:52:05.7068865Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T08:52:05.7069325Z outputs = self.fnet( 2025-12-04T08:52:05.7069742Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T08:52:05.7070233Z encoder_outputs = self.encoder( 2025-12-04T08:52:05.7070670Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T08:52:05.7071112Z layer_outputs = layer_module(hidden_states) 2025-12-04T08:52:05.7071533Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:05.7071935Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:05.7072363Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T08:52:05.7072816Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T08:52:05.7073740Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T08:52:05.7074194Z self_outputs = self.self(hidden_states) 2025-12-04T08:52:05.7074657Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T08:52:05.7075205Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T08:52:05.7075391Z 2025-12-04T08:52:05.7075518Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:05.7075947Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:05.7076331Z res = mod(**inputs) 2025-12-04T08:52:05.7076745Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T08:52:05.7077178Z outputs = self.fnet( 2025-12-04T08:52:05.7077597Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T08:52:05.7078109Z encoder_outputs = self.encoder( 2025-12-04T08:52:05.7078780Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T08:52:05.7079246Z layer_outputs = layer_module(hidden_states) 2025-12-04T08:52:05.7079677Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:05.7080078Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:05.7080499Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T08:52:05.7080947Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T08:52:05.7081391Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T08:52:05.7081829Z self_outputs = self.self(hidden_states) 2025-12-04T08:52:05.7082251Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T08:52:05.7082702Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T08:52:05.7082871Z 2025-12-04T08:52:05.7082996Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:05.7083383Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:05.7083737Z res = mod(**inputs) 2025-12-04T08:52:05.7084122Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T08:52:05.7084529Z outputs = self.fnet( 2025-12-04T08:52:05.7084915Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T08:52:05.7085335Z encoder_outputs = self.encoder( 2025-12-04T08:52:05.7085749Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T08:52:05.7086174Z layer_outputs = layer_module(hidden_states) 2025-12-04T08:52:05.7086580Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:05.7086981Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:05.7087403Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T08:52:05.7087842Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T08:52:05.7088278Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T08:52:05.7088705Z self_outputs = self.self(hidden_states) 2025-12-04T08:52:05.7089126Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T08:52:05.7089737Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T08:52:05.7089957Z 2025-12-04T08:52:05.7090075Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:05.7090475Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:05.7090852Z res = mod(**inputs) 2025-12-04T08:52:05.7091247Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T08:52:05.7091664Z outputs = self.fnet( 2025-12-04T08:52:05.7092056Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T08:52:05.7092472Z encoder_outputs = self.encoder( 2025-12-04T08:52:05.7092890Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T08:52:05.7093346Z layer_outputs = layer_module(hidden_states) 2025-12-04T08:52:05.7093753Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:05.7094147Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:05.7094568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T08:52:05.7095015Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T08:52:05.7095447Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T08:52:05.7095885Z self_outputs = self.self(hidden_states) 2025-12-04T08:52:05.7096282Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T08:52:05.7096708Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T08:52:05.7096867Z 2025-12-04T08:52:05.7097002Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:05.7097370Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:05.7097709Z res = mod(**inputs) 2025-12-04T08:52:05.7098069Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T08:52:05.7098454Z outputs = self.fnet( 2025-12-04T08:52:05.7098818Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T08:52:05.7099213Z encoder_outputs = self.encoder( 2025-12-04T08:52:05.7099601Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T08:52:05.7100003Z layer_outputs = layer_module(hidden_states) 2025-12-04T08:52:05.7100399Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:05.7100776Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:05.7101170Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T08:52:05.7101602Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T08:52:05.7102021Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T08:52:05.7102438Z self_outputs = self.self(hidden_states) 2025-12-04T08:52:05.7102836Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T08:52:05.7103258Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T08:52:05.7103422Z 2025-12-04T08:52:05.7103529Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:05.7103912Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:05.7104255Z res = mod(**inputs) 2025-12-04T08:52:05.7104670Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T08:52:05.7105069Z outputs = self.fnet( 2025-12-04T08:52:05.7105427Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T08:52:05.7105864Z encoder_outputs = self.encoder( 2025-12-04T08:52:05.7106277Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T08:52:05.7106697Z layer_outputs = layer_module(hidden_states) 2025-12-04T08:52:05.7107079Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:05.7107462Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:05.7107875Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T08:52:05.7108340Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T08:52:05.7108762Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T08:52:05.7109180Z self_outputs = self.self(hidden_states) 2025-12-04T08:52:05.7109601Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T08:52:05.7110047Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T08:52:05.7110223Z 2025-12-04T08:52:05.7110338Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:05.7110732Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:05.7111097Z res = mod(**inputs) 2025-12-04T08:52:05.7111466Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T08:52:05.7111868Z outputs = self.fnet( 2025-12-04T08:52:05.7112249Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T08:52:05.7112670Z encoder_outputs = self.encoder( 2025-12-04T08:52:05.7113086Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T08:52:05.7113516Z layer_outputs = layer_module(hidden_states) 2025-12-04T08:52:05.7113916Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:05.7114305Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:05.7114727Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T08:52:05.7115172Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T08:52:05.7115618Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T08:52:05.7116041Z self_outputs = self.self(hidden_states) 2025-12-04T08:52:05.7116464Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T08:52:05.7116913Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T08:52:05.7117081Z 2025-12-04T08:52:05.7117202Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:05.7117586Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:05.7117939Z res = mod(**inputs) 2025-12-04T08:52:05.7118490Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T08:52:05.7118936Z outputs = self.fnet( 2025-12-04T08:52:05.7119347Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T08:52:05.7119830Z encoder_outputs = self.encoder( 2025-12-04T08:52:05.7120245Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T08:52:05.7120859Z layer_outputs = layer_module(hidden_states) 2025-12-04T08:52:05.7121278Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:05.7121685Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:05.7122101Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T08:52:05.7122524Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T08:52:05.7122998Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T08:52:05.7123488Z self_outputs = self.self(hidden_states) 2025-12-04T08:52:05.7123906Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T08:52:05.7124357Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T08:52:05.7124523Z 2025-12-04T08:52:05.7124648Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:05.7125046Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:05.7125373Z res = mod(**inputs) 2025-12-04T08:52:05.7125749Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T08:52:05.7126162Z outputs = self.fnet( 2025-12-04T08:52:05.7126544Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T08:52:05.7126973Z encoder_outputs = self.encoder( 2025-12-04T08:52:05.7127388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T08:52:05.7127818Z layer_outputs = layer_module(hidden_states) 2025-12-04T08:52:05.7128210Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:05.7128606Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:05.7129025Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T08:52:05.7129441Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T08:52:05.7129879Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T08:52:05.7130307Z self_outputs = self.self(hidden_states) 2025-12-04T08:52:05.7130725Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T08:52:05.7131168Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T08:52:05.7131343Z 2025-12-04T08:52:05.7131456Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:05.7131861Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:05.7132215Z res = mod(**inputs) 2025-12-04T08:52:05.7132587Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T08:52:05.7132995Z outputs = self.fnet( 2025-12-04T08:52:05.7133375Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T08:52:05.7133786Z encoder_outputs = self.encoder( 2025-12-04T08:52:05.7134197Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T08:52:05.7134628Z layer_outputs = layer_module(hidden_states) 2025-12-04T08:52:05.7135104Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:05.7135499Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:05.7135923Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T08:52:05.7136394Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T08:52:05.7136828Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T08:52:05.7137253Z self_outputs = self.self(hidden_states) 2025-12-04T08:52:05.7137673Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T08:52:05.7138123Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T08:52:05.7138306Z 2025-12-04T08:52:05.7138417Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:05.7138812Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:05.7139165Z res = mod(**inputs) 2025-12-04T08:52:05.7139548Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T08:52:05.7139956Z outputs = self.fnet( 2025-12-04T08:52:05.7140344Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T08:52:05.7140761Z encoder_outputs = self.encoder( 2025-12-04T08:52:05.7141163Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T08:52:05.7141593Z layer_outputs = layer_module(hidden_states) 2025-12-04T08:52:05.7141991Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:05.7142387Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:05.7142798Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T08:52:05.7143216Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T08:52:05.7143628Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T08:52:05.7144027Z self_outputs = self.self(hidden_states) 2025-12-04T08:52:05.7144415Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T08:52:05.7144837Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T08:52:05.7144996Z 2025-12-04T08:52:05.7145108Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:05.7145472Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:05.7145807Z res = mod(**inputs) 2025-12-04T08:52:05.7146182Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T08:52:05.7146570Z outputs = self.fnet( 2025-12-04T08:52:05.7146925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T08:52:05.7147322Z encoder_outputs = self.encoder( 2025-12-04T08:52:05.7147710Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T08:52:05.7148109Z layer_outputs = layer_module(hidden_states) 2025-12-04T08:52:05.7148487Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:05.7148862Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:05.7149261Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T08:52:05.7149721Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T08:52:05.7150137Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T08:52:05.7150559Z self_outputs = self.self(hidden_states) 2025-12-04T08:52:05.7150958Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T08:52:05.7151375Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T08:52:05.7151539Z 2025-12-04T08:52:05.7151644Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:05.7152012Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:05.7152338Z res = mod(**inputs) 2025-12-04T08:52:05.7152703Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T08:52:05.7153112Z outputs = self.fnet( 2025-12-04T08:52:05.7153476Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 511, in forward 2025-12-04T08:52:05.7153871Z embedding_output = self.embeddings( 2025-12-04T08:52:05.7154270Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 141, in forward 2025-12-04T08:52:05.7154690Z embeddings = self.projection(embeddings) 2025-12-04T08:52:05.7154836Z 2025-12-04T08:52:05.7154927Z cudagraph partition due to non gpu ops 2025-12-04T08:52:05.7155188Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:05.7155575Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:05.7155930Z res = mod(**inputs) 2025-12-04T08:52:05.7156307Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T08:52:05.7156724Z outputs = self.fnet( 2025-12-04T08:52:05.7157109Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T08:52:05.7157516Z encoder_outputs = self.encoder( 2025-12-04T08:52:05.7157926Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T08:52:05.7158483Z layer_outputs = layer_module(hidden_states) 2025-12-04T08:52:05.7158903Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:05.7159316Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:05.7159752Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T08:52:05.7160217Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T08:52:05.7160681Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T08:52:05.7161102Z self_outputs = self.self(hidden_states) 2025-12-04T08:52:05.7161527Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T08:52:05.7161979Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T08:52:05.7162147Z 2025-12-04T08:52:05.7162262Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:05.7162647Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:05.7162999Z res = mod(**inputs) 2025-12-04T08:52:05.7163387Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T08:52:05.7163794Z outputs = self.fnet( 2025-12-04T08:52:05.7164184Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T08:52:05.7164704Z encoder_outputs = self.encoder( 2025-12-04T08:52:05.7165112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T08:52:05.7165569Z layer_outputs = layer_module(hidden_states) 2025-12-04T08:52:05.7165974Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:05.7166372Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:05.7166785Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T08:52:05.7167245Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T08:52:05.7167683Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T08:52:05.7168188Z self_outputs = self.self(hidden_states) 2025-12-04T08:52:05.7168610Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T08:52:05.7169178Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T08:52:05.7169352Z 2025-12-04T08:52:05.7169476Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:05.7169879Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:05.7170232Z res = mod(**inputs) 2025-12-04T08:52:05.7170623Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T08:52:05.7171035Z outputs = self.fnet( 2025-12-04T08:52:05.7171415Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T08:52:05.7171835Z encoder_outputs = self.encoder( 2025-12-04T08:52:05.7172249Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T08:52:05.7172680Z layer_outputs = layer_module(hidden_states) 2025-12-04T08:52:05.7173074Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:05.7173473Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:05.7173890Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T08:52:05.7174323Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T08:52:05.7174820Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T08:52:05.7175241Z self_outputs = self.self(hidden_states) 2025-12-04T08:52:05.7175660Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T08:52:05.7176104Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T08:52:05.7176279Z 2025-12-04T08:52:05.7176392Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:05.7176780Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:05.7177131Z res = mod(**inputs) 2025-12-04T08:52:05.7177506Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T08:52:05.7177914Z outputs = self.fnet( 2025-12-04T08:52:05.7178295Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T08:52:05.7178704Z encoder_outputs = self.encoder( 2025-12-04T08:52:05.7179111Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T08:52:05.7179543Z layer_outputs = layer_module(hidden_states) 2025-12-04T08:52:05.7179973Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:05.7180359Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:05.7180776Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T08:52:05.7181242Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T08:52:05.7181670Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T08:52:05.7182097Z self_outputs = self.self(hidden_states) 2025-12-04T08:52:05.7182518Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T08:52:05.7182965Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T08:52:05.7183164Z 2025-12-04T08:52:05.7183276Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:05.7183671Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:05.7184024Z res = mod(**inputs) 2025-12-04T08:52:05.7184411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T08:52:05.7184816Z outputs = self.fnet( 2025-12-04T08:52:05.7185204Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T08:52:05.7185620Z encoder_outputs = self.encoder( 2025-12-04T08:52:05.7186019Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T08:52:05.7186448Z layer_outputs = layer_module(hidden_states) 2025-12-04T08:52:05.7186851Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:05.7187252Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:05.7187663Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 251, in forward 2025-12-04T08:52:05.7188092Z layer_output = apply_chunking_to_forward( 2025-12-04T08:52:05.7188540Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:52:05.7188970Z return forward_fn(*input_tensors) 2025-12-04T08:52:05.7189420Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 260, in feed_forward_chunk 2025-12-04T08:52:05.7189914Z intermediate_output = self.intermediate(fourier_output) 2025-12-04T08:52:05.7190372Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 218, in forward 2025-12-04T08:52:05.7190800Z hidden_states = self.dense(hidden_states) 2025-12-04T08:52:05.7190960Z 2025-12-04T08:52:05.7191077Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:05.7191479Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:05.7191841Z res = mod(**inputs) 2025-12-04T08:52:05.7192229Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T08:52:05.7192653Z outputs = self.fnet( 2025-12-04T08:52:05.7193037Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T08:52:05.7193444Z encoder_outputs = self.encoder( 2025-12-04T08:52:05.7193857Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T08:52:05.7194297Z layer_outputs = layer_module(hidden_states) 2025-12-04T08:52:05.7194708Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:05.7195142Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:05.7195574Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 251, in forward 2025-12-04T08:52:05.7196034Z layer_output = apply_chunking_to_forward( 2025-12-04T08:52:05.7196481Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:52:05.7196924Z return forward_fn(*input_tensors) 2025-12-04T08:52:05.7197382Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 260, in feed_forward_chunk 2025-12-04T08:52:05.7197892Z intermediate_output = self.intermediate(fourier_output) 2025-12-04T08:52:05.7198416Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 219, in forward 2025-12-04T08:52:05.7198925Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:52:05.7199362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 62, in forward 2025-12-04T08:52:05.7199880Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-12-04T08:52:05.7200145Z 2025-12-04T08:52:05.7200265Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:05.7200673Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:05.7201039Z res = mod(**inputs) 2025-12-04T08:52:05.7201441Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T08:52:05.7201858Z outputs = self.fnet( 2025-12-04T08:52:05.7202259Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T08:52:05.7202692Z encoder_outputs = self.encoder( 2025-12-04T08:52:05.7203116Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T08:52:05.7203561Z layer_outputs = layer_module(hidden_states) 2025-12-04T08:52:05.7203975Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:05.7204391Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:05.7204840Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 251, in forward 2025-12-04T08:52:05.7205281Z layer_output = apply_chunking_to_forward( 2025-12-04T08:52:05.7205746Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:52:05.7206170Z return forward_fn(*input_tensors) 2025-12-04T08:52:05.7206621Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-12-04T08:52:05.7207130Z layer_output = self.output(intermediate_output, fourier_output) 2025-12-04T08:52:05.7207599Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 232, in forward 2025-12-04T08:52:05.7208023Z hidden_states = self.dense(hidden_states) 2025-12-04T08:52:05.7208181Z 2025-12-04T08:52:05.7208273Z cudagraph partition due to non gpu ops 2025-12-04T08:52:05.7208537Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:05.7208927Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:05.7209271Z res = mod(**inputs) 2025-12-04T08:52:05.7209656Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T08:52:05.7210070Z outputs = self.fnet( 2025-12-04T08:52:05.7210485Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T08:52:05.7210901Z encoder_outputs = self.encoder( 2025-12-04T08:52:05.7211309Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T08:52:05.7211760Z layer_outputs = layer_module(hidden_states) 2025-12-04T08:52:05.7212161Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:05.7212558Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:05.7212979Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T08:52:05.7213435Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T08:52:05.7213877Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T08:52:05.7214328Z self_outputs = self.self(hidden_states) 2025-12-04T08:52:05.7214755Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T08:52:05.7215201Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T08:52:05.7215379Z 2025-12-04T08:52:05.7215492Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:05.7216045Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:05.7216404Z res = mod(**inputs) 2025-12-04T08:52:05.7216792Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T08:52:05.7217207Z outputs = self.fnet( 2025-12-04T08:52:05.7217595Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T08:52:05.7218014Z encoder_outputs = self.encoder( 2025-12-04T08:52:05.7218433Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T08:52:05.7218869Z layer_outputs = layer_module(hidden_states) 2025-12-04T08:52:05.7219276Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:05.7219668Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:05.7220086Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T08:52:05.7220531Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T08:52:05.7221074Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T08:52:05.7221501Z self_outputs = self.self(hidden_states) 2025-12-04T08:52:05.7221931Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T08:52:05.7222383Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T08:52:05.7222551Z 2025-12-04T08:52:05.7222664Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:05.7223058Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:05.7223410Z res = mod(**inputs) 2025-12-04T08:52:05.7223805Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T08:52:05.7224190Z outputs = self.fnet( 2025-12-04T08:52:05.7224561Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T08:52:05.7224963Z encoder_outputs = self.encoder( 2025-12-04T08:52:05.7225343Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T08:52:05.7225826Z layer_outputs = layer_module(hidden_states) 2025-12-04T08:52:05.7226210Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:05.7226587Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:05.7227005Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T08:52:05.7227428Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T08:52:05.7227848Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T08:52:05.7228257Z self_outputs = self.self(hidden_states) 2025-12-04T08:52:05.7228652Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T08:52:05.7229104Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T08:52:05.7229261Z 2025-12-04T08:52:05.7229378Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:05.7229740Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:05.7230077Z res = mod(**inputs) 2025-12-04T08:52:05.7230442Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T08:52:05.7230831Z outputs = self.fnet( 2025-12-04T08:52:05.7231189Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T08:52:05.7231583Z encoder_outputs = self.encoder( 2025-12-04T08:52:05.7231970Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T08:52:05.7232369Z layer_outputs = layer_module(hidden_states) 2025-12-04T08:52:05.7232770Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:05.7233164Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:05.7233584Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T08:52:05.7234020Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T08:52:05.7234457Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T08:52:05.7234878Z self_outputs = self.self(hidden_states) 2025-12-04T08:52:05.7235299Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T08:52:05.7235734Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T08:52:05.7235906Z 2025-12-04T08:52:05.7236017Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:05.7236406Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:05.7236751Z res = mod(**inputs) 2025-12-04T08:52:05.7237136Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T08:52:05.7237546Z outputs = self.fnet( 2025-12-04T08:52:05.7237933Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T08:52:05.7238401Z encoder_outputs = self.encoder( 2025-12-04T08:52:05.7238812Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T08:52:05.7239246Z layer_outputs = layer_module(hidden_states) 2025-12-04T08:52:05.7239646Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:05.7240055Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:05.7240531Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 251, in forward 2025-12-04T08:52:05.7240976Z layer_output = apply_chunking_to_forward( 2025-12-04T08:52:05.7241412Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:52:05.7241915Z return forward_fn(*input_tensors) 2025-12-04T08:52:05.7242368Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 260, in feed_forward_chunk 2025-12-04T08:52:05.7242865Z intermediate_output = self.intermediate(fourier_output) 2025-12-04T08:52:05.7243322Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 218, in forward 2025-12-04T08:52:05.7243753Z hidden_states = self.dense(hidden_states) 2025-12-04T08:52:05.7243925Z 2025-12-04T08:52:05.7244048Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:05.7244442Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:05.7244803Z res = mod(**inputs) 2025-12-04T08:52:05.7245196Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T08:52:05.7245617Z outputs = self.fnet( 2025-12-04T08:52:05.7246000Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T08:52:05.7246425Z encoder_outputs = self.encoder( 2025-12-04T08:52:05.7246848Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T08:52:05.7247285Z layer_outputs = layer_module(hidden_states) 2025-12-04T08:52:05.7247682Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:05.7248083Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:05.7248507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 251, in forward 2025-12-04T08:52:05.7248935Z layer_output = apply_chunking_to_forward( 2025-12-04T08:52:05.7249382Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:52:05.7249821Z return forward_fn(*input_tensors) 2025-12-04T08:52:05.7250275Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 260, in feed_forward_chunk 2025-12-04T08:52:05.7250769Z intermediate_output = self.intermediate(fourier_output) 2025-12-04T08:52:05.7251234Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 219, in forward 2025-12-04T08:52:05.7251701Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:52:05.7252130Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 62, in forward 2025-12-04T08:52:05.7252627Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-12-04T08:52:05.7252889Z 2025-12-04T08:52:05.7253004Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:05.7253399Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:05.7253749Z res = mod(**inputs) 2025-12-04T08:52:05.7254138Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T08:52:05.7254553Z outputs = self.fnet( 2025-12-04T08:52:05.7254946Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T08:52:05.7255359Z encoder_outputs = self.encoder( 2025-12-04T08:52:05.7255818Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T08:52:05.7256250Z layer_outputs = layer_module(hidden_states) 2025-12-04T08:52:05.7256651Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:05.7257071Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:05.7257492Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 251, in forward 2025-12-04T08:52:05.7257920Z layer_output = apply_chunking_to_forward( 2025-12-04T08:52:05.7258355Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:52:05.7258787Z return forward_fn(*input_tensors) 2025-12-04T08:52:05.7259233Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-12-04T08:52:05.7259768Z layer_output = self.output(intermediate_output, fourier_output) 2025-12-04T08:52:05.7260242Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 232, in forward 2025-12-04T08:52:05.7260681Z hidden_states = self.dense(hidden_states) 2025-12-04T08:52:05.7260837Z 2025-12-04T08:52:05.7260939Z cudagraph partition due to non gpu ops 2025-12-04T08:52:05.7261204Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:05.7261602Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:05.7261965Z res = mod(**inputs) 2025-12-04T08:52:05.7262361Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T08:52:05.7262771Z outputs = self.fnet( 2025-12-04T08:52:05.7263165Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T08:52:05.7263594Z encoder_outputs = self.encoder( 2025-12-04T08:52:05.7264009Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T08:52:05.7264443Z layer_outputs = layer_module(hidden_states) 2025-12-04T08:52:05.7264862Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:05.7265268Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:05.7265686Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T08:52:05.7266135Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T08:52:05.7266577Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T08:52:05.7266983Z self_outputs = self.self(hidden_states) 2025-12-04T08:52:05.7267383Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T08:52:05.7267815Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T08:52:05.7267977Z 2025-12-04T08:52:05.7268098Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:05.7268491Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:05.7268843Z res = mod(**inputs) 2025-12-04T08:52:05.7269237Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T08:52:05.7269653Z outputs = self.fnet( 2025-12-04T08:52:05.7270023Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T08:52:05.7270450Z encoder_outputs = self.encoder( 2025-12-04T08:52:05.7270912Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T08:52:05.7271354Z layer_outputs = layer_module(hidden_states) 2025-12-04T08:52:05.7271726Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:05.7272123Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:05.7272520Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T08:52:05.7272933Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T08:52:05.7273348Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T08:52:05.7273749Z self_outputs = self.self(hidden_states) 2025-12-04T08:52:05.7274149Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T08:52:05.7274583Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T08:52:05.7274751Z 2025-12-04T08:52:05.7274857Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:05.7275244Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:05.7275597Z res = mod(**inputs) 2025-12-04T08:52:05.7275969Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T08:52:05.7276377Z outputs = self.fnet( 2025-12-04T08:52:05.7276763Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T08:52:05.7277168Z encoder_outputs = self.encoder( 2025-12-04T08:52:05.7277574Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T08:52:05.7278008Z layer_outputs = layer_module(hidden_states) 2025-12-04T08:52:05.7278488Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:05.7278885Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:05.7279307Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T08:52:05.7279757Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T08:52:05.7280189Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T08:52:05.7280615Z self_outputs = self.self(hidden_states) 2025-12-04T08:52:05.7281037Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T08:52:05.7281488Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T08:52:05.7281659Z 2025-12-04T08:52:05.7281774Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:05.7282170Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:05.7282532Z res = mod(**inputs) 2025-12-04T08:52:05.7282897Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T08:52:05.7283297Z outputs = self.fnet( 2025-12-04T08:52:05.7283680Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T08:52:05.7284100Z encoder_outputs = self.encoder( 2025-12-04T08:52:05.7284496Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T08:52:05.7284929Z layer_outputs = layer_module(hidden_states) 2025-12-04T08:52:05.7285330Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:05.7285722Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:05.7286175Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T08:52:05.7286618Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T08:52:05.7287082Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T08:52:05.7287503Z self_outputs = self.self(hidden_states) 2025-12-04T08:52:05.7287932Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T08:52:05.7288385Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T08:52:05.7288555Z 2025-12-04T08:52:05.7288677Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:05.7289062Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:05.7289436Z res = mod(**inputs) 2025-12-04T08:52:05.7289825Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T08:52:05.7290239Z outputs = self.fnet( 2025-12-04T08:52:05.7290621Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T08:52:05.7291047Z encoder_outputs = self.encoder( 2025-12-04T08:52:05.7291456Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T08:52:05.7291881Z layer_outputs = layer_module(hidden_states) 2025-12-04T08:52:05.7292276Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:05.7292650Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:05.7293044Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 251, in forward 2025-12-04T08:52:05.7293448Z layer_output = apply_chunking_to_forward( 2025-12-04T08:52:05.7293871Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:52:05.7294283Z return forward_fn(*input_tensors) 2025-12-04T08:52:05.7294699Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 260, in feed_forward_chunk 2025-12-04T08:52:05.7295164Z intermediate_output = self.intermediate(fourier_output) 2025-12-04T08:52:05.7295600Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 218, in forward 2025-12-04T08:52:05.7296004Z hidden_states = self.dense(hidden_states) 2025-12-04T08:52:05.7296144Z 2025-12-04T08:52:05.7296252Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:05.7296624Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:05.7296959Z res = mod(**inputs) 2025-12-04T08:52:05.7297321Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T08:52:05.7297702Z outputs = self.fnet( 2025-12-04T08:52:05.7298072Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T08:52:05.7298469Z encoder_outputs = self.encoder( 2025-12-04T08:52:05.7298850Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T08:52:05.7299260Z layer_outputs = layer_module(hidden_states) 2025-12-04T08:52:05.7299638Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:05.7300013Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:05.7300454Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 251, in forward 2025-12-04T08:52:05.7300865Z layer_output = apply_chunking_to_forward( 2025-12-04T08:52:05.7301284Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:52:05.7301714Z return forward_fn(*input_tensors) 2025-12-04T08:52:05.7302133Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 260, in feed_forward_chunk 2025-12-04T08:52:05.7302607Z intermediate_output = self.intermediate(fourier_output) 2025-12-04T08:52:05.7303046Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 219, in forward 2025-12-04T08:52:05.7303474Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:52:05.7303873Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 62, in forward 2025-12-04T08:52:05.7304381Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-12-04T08:52:05.7304626Z 2025-12-04T08:52:05.7304743Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:05.7305111Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:05.7305448Z res = mod(**inputs) 2025-12-04T08:52:05.7305814Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T08:52:05.7306198Z outputs = self.fnet( 2025-12-04T08:52:05.7306565Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T08:52:05.7306964Z encoder_outputs = self.encoder( 2025-12-04T08:52:05.7307352Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T08:52:05.7307759Z layer_outputs = layer_module(hidden_states) 2025-12-04T08:52:05.7308140Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:05.7308514Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:05.7308917Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 251, in forward 2025-12-04T08:52:05.7309316Z layer_output = apply_chunking_to_forward( 2025-12-04T08:52:05.7309734Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:52:05.7310144Z return forward_fn(*input_tensors) 2025-12-04T08:52:05.7310559Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-12-04T08:52:05.7311038Z layer_output = self.output(intermediate_output, fourier_output) 2025-12-04T08:52:05.7311488Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 232, in forward 2025-12-04T08:52:05.7311894Z hidden_states = self.dense(hidden_states) 2025-12-04T08:52:05.7312035Z 2025-12-04T08:52:05.7312123Z cudagraph partition due to non gpu ops 2025-12-04T08:52:05.7312372Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:05.7312753Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:05.7313100Z res = mod(**inputs) 2025-12-04T08:52:05.7313457Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T08:52:05.7313890Z outputs = self.fnet( 2025-12-04T08:52:05.7314255Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T08:52:05.7314645Z encoder_outputs = self.encoder( 2025-12-04T08:52:05.7315072Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T08:52:05.7315486Z layer_outputs = layer_module(hidden_states) 2025-12-04T08:52:05.7315907Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:05.7316325Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:05.7316744Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T08:52:05.7317194Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T08:52:05.7317638Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T08:52:05.7318061Z self_outputs = self.self(hidden_states) 2025-12-04T08:52:05.7318594Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T08:52:05.7319080Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T08:52:05.7319258Z 2025-12-04T08:52:05.7319376Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:05.7319786Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:05.7320155Z res = mod(**inputs) 2025-12-04T08:52:05.7320540Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T08:52:05.7321097Z outputs = self.fnet( 2025-12-04T08:52:05.7321492Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T08:52:05.7321915Z encoder_outputs = self.encoder( 2025-12-04T08:52:05.7322319Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T08:52:05.7322769Z layer_outputs = layer_module(hidden_states) 2025-12-04T08:52:05.7323177Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:05.7323582Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:05.7323998Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T08:52:05.7324445Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T08:52:05.7324859Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T08:52:05.7325277Z self_outputs = self.self(hidden_states) 2025-12-04T08:52:05.7325690Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T08:52:05.7326135Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T08:52:05.7326308Z 2025-12-04T08:52:05.7326434Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:05.7326819Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:05.7327166Z res = mod(**inputs) 2025-12-04T08:52:05.7327543Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T08:52:05.7327955Z outputs = self.fnet( 2025-12-04T08:52:05.7328336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T08:52:05.7328754Z encoder_outputs = self.encoder( 2025-12-04T08:52:05.7329165Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T08:52:05.7329591Z layer_outputs = layer_module(hidden_states) 2025-12-04T08:52:05.7329992Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:05.7330473Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:05.7330897Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T08:52:05.7331378Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T08:52:05.7331824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T08:52:05.7332258Z self_outputs = self.self(hidden_states) 2025-12-04T08:52:05.7332687Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T08:52:05.7333144Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T08:52:05.7333320Z 2025-12-04T08:52:05.7333433Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:05.7333850Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:05.7334189Z res = mod(**inputs) 2025-12-04T08:52:05.7334582Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T08:52:05.7335001Z outputs = self.fnet( 2025-12-04T08:52:05.7335383Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T08:52:05.7335798Z encoder_outputs = self.encoder( 2025-12-04T08:52:05.7336215Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T08:52:05.7336702Z layer_outputs = layer_module(hidden_states) 2025-12-04T08:52:05.7337097Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:05.7337500Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:05.7337931Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T08:52:05.7338382Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T08:52:05.7338819Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T08:52:05.7339254Z self_outputs = self.self(hidden_states) 2025-12-04T08:52:05.7339681Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T08:52:05.7340147Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T08:52:05.7340315Z 2025-12-04T08:52:05.7340427Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:05.7340824Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:05.7341178Z res = mod(**inputs) 2025-12-04T08:52:05.7341566Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T08:52:05.7341984Z outputs = self.fnet( 2025-12-04T08:52:05.7342400Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T08:52:05.7342810Z encoder_outputs = self.encoder( 2025-12-04T08:52:05.7343204Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T08:52:05.7343647Z layer_outputs = layer_module(hidden_states) 2025-12-04T08:52:05.7344048Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:05.7344447Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:05.7344867Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 251, in forward 2025-12-04T08:52:05.7345304Z layer_output = apply_chunking_to_forward( 2025-12-04T08:52:05.7345790Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:52:05.7346223Z return forward_fn(*input_tensors) 2025-12-04T08:52:05.7346673Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 260, in feed_forward_chunk 2025-12-04T08:52:05.7347189Z intermediate_output = self.intermediate(fourier_output) 2025-12-04T08:52:05.7347651Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 218, in forward 2025-12-04T08:52:05.7348079Z hidden_states = self.dense(hidden_states) 2025-12-04T08:52:05.7348240Z 2025-12-04T08:52:05.7348354Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:05.7348750Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:05.7349130Z res = mod(**inputs) 2025-12-04T08:52:05.7349521Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T08:52:05.7349942Z outputs = self.fnet( 2025-12-04T08:52:05.7350343Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T08:52:05.7350777Z encoder_outputs = self.encoder( 2025-12-04T08:52:05.7351195Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T08:52:05.7351640Z layer_outputs = layer_module(hidden_states) 2025-12-04T08:52:05.7352054Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:05.7352465Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:05.7352895Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 251, in forward 2025-12-04T08:52:05.7369695Z layer_output = apply_chunking_to_forward( 2025-12-04T08:52:05.7370346Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:52:05.7370847Z return forward_fn(*input_tensors) 2025-12-04T08:52:05.7371308Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 260, in feed_forward_chunk 2025-12-04T08:52:05.7371799Z intermediate_output = self.intermediate(fourier_output) 2025-12-04T08:52:05.7372258Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 219, in forward 2025-12-04T08:52:05.7372699Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:52:05.7373110Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 62, in forward 2025-12-04T08:52:05.7373597Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-12-04T08:52:05.7373849Z 2025-12-04T08:52:05.7373974Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:05.7374360Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:05.7374714Z res = mod(**inputs) 2025-12-04T08:52:05.7375101Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T08:52:05.7375507Z outputs = self.fnet( 2025-12-04T08:52:05.7375880Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T08:52:05.7376293Z encoder_outputs = self.encoder( 2025-12-04T08:52:05.7376698Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T08:52:05.7377114Z layer_outputs = layer_module(hidden_states) 2025-12-04T08:52:05.7377637Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:05.7378024Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:05.7378419Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 251, in forward 2025-12-04T08:52:05.7378882Z layer_output = apply_chunking_to_forward( 2025-12-04T08:52:05.7379311Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:52:05.7379734Z return forward_fn(*input_tensors) 2025-12-04T08:52:05.7380147Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-12-04T08:52:05.7380631Z layer_output = self.output(intermediate_output, fourier_output) 2025-12-04T08:52:05.7381124Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 232, in forward 2025-12-04T08:52:05.7381540Z hidden_states = self.dense(hidden_states) 2025-12-04T08:52:05.7381688Z 2025-12-04T08:52:05.7381776Z cudagraph partition due to non gpu ops 2025-12-04T08:52:05.7382032Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:05.7382417Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:05.7382759Z res = mod(**inputs) 2025-12-04T08:52:05.7383123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T08:52:05.7383521Z outputs = self.fnet( 2025-12-04T08:52:05.7383911Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T08:52:05.7384401Z encoder_outputs = self.encoder( 2025-12-04T08:52:05.7384795Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T08:52:05.7385220Z layer_outputs = layer_module(hidden_states) 2025-12-04T08:52:05.7385608Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:05.7385983Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:05.7386383Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T08:52:05.7386811Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T08:52:05.7387250Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T08:52:05.7387683Z self_outputs = self.self(hidden_states) 2025-12-04T08:52:05.7388109Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T08:52:05.7388567Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T08:52:05.7388742Z 2025-12-04T08:52:05.7388856Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:05.7389253Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:05.7389613Z res = mod(**inputs) 2025-12-04T08:52:05.7390001Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T08:52:05.7390411Z outputs = self.fnet( 2025-12-04T08:52:05.7390798Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T08:52:05.7391199Z encoder_outputs = self.encoder( 2025-12-04T08:52:05.7391581Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T08:52:05.7392014Z layer_outputs = layer_module(hidden_states) 2025-12-04T08:52:05.7392459Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:05.7392867Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:05.7393285Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T08:52:05.7393756Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T08:52:05.7394201Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T08:52:05.7394628Z self_outputs = self.self(hidden_states) 2025-12-04T08:52:05.7395061Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T08:52:05.7395517Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T08:52:05.7395689Z 2025-12-04T08:52:05.7395884Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:05.7396276Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:05.7396633Z res = mod(**inputs) 2025-12-04T08:52:05.7397027Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T08:52:05.7397444Z outputs = self.fnet( 2025-12-04T08:52:05.7397826Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T08:52:05.7398341Z encoder_outputs = self.encoder( 2025-12-04T08:52:05.7398769Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T08:52:05.7399208Z layer_outputs = layer_module(hidden_states) 2025-12-04T08:52:05.7399622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:05.7400043Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:05.7400476Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T08:52:05.7400912Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T08:52:05.7401352Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T08:52:05.7401784Z self_outputs = self.self(hidden_states) 2025-12-04T08:52:05.7402208Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T08:52:05.7402650Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T08:52:05.7402826Z 2025-12-04T08:52:05.7402944Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:05.7403341Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:05.7403687Z res = mod(**inputs) 2025-12-04T08:52:05.7404077Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T08:52:05.7404469Z outputs = self.fnet( 2025-12-04T08:52:05.7404825Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T08:52:05.7405206Z encoder_outputs = self.encoder( 2025-12-04T08:52:05.7405583Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T08:52:05.7405985Z layer_outputs = layer_module(hidden_states) 2025-12-04T08:52:05.7406347Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:05.7406715Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:05.7407103Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T08:52:05.7407551Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T08:52:05.7407953Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T08:52:05.7408344Z self_outputs = self.self(hidden_states) 2025-12-04T08:52:05.7408778Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T08:52:05.7409188Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T08:52:05.7409346Z 2025-12-04T08:52:05.7409451Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:05.7409811Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:05.7410137Z res = mod(**inputs) 2025-12-04T08:52:05.7410486Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T08:52:05.7410883Z outputs = self.fnet( 2025-12-04T08:52:05.7411244Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T08:52:05.7411622Z encoder_outputs = self.encoder( 2025-12-04T08:52:05.7412005Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T08:52:05.7412406Z layer_outputs = layer_module(hidden_states) 2025-12-04T08:52:05.7412779Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:05.7413142Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:05.7413528Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 251, in forward 2025-12-04T08:52:05.7413942Z layer_output = apply_chunking_to_forward( 2025-12-04T08:52:05.7414347Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:52:05.7414753Z return forward_fn(*input_tensors) 2025-12-04T08:52:05.7415167Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 260, in feed_forward_chunk 2025-12-04T08:52:05.7415625Z intermediate_output = self.intermediate(fourier_output) 2025-12-04T08:52:05.7416043Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 218, in forward 2025-12-04T08:52:05.7416441Z hidden_states = self.dense(hidden_states) 2025-12-04T08:52:05.7416579Z 2025-12-04T08:52:05.7416692Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:05.7417052Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:05.7417370Z res = mod(**inputs) 2025-12-04T08:52:05.7417729Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T08:52:05.7418179Z outputs = self.fnet( 2025-12-04T08:52:05.7418526Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T08:52:05.7418912Z encoder_outputs = self.encoder( 2025-12-04T08:52:05.7419290Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T08:52:05.7419694Z layer_outputs = layer_module(hidden_states) 2025-12-04T08:52:05.7420058Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:05.7420429Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:05.7420985Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 251, in forward 2025-12-04T08:52:05.7421389Z layer_output = apply_chunking_to_forward( 2025-12-04T08:52:05.7421912Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:52:05.7422327Z return forward_fn(*input_tensors) 2025-12-04T08:52:05.7422750Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 260, in feed_forward_chunk 2025-12-04T08:52:05.7423248Z intermediate_output = self.intermediate(fourier_output) 2025-12-04T08:52:05.7423697Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 219, in forward 2025-12-04T08:52:05.7424130Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:52:05.7424517Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 62, in forward 2025-12-04T08:52:05.7424979Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-12-04T08:52:05.7425250Z 2025-12-04T08:52:05.7425358Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:05.7425723Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:05.7426049Z res = mod(**inputs) 2025-12-04T08:52:05.7426402Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T08:52:05.7426780Z outputs = self.fnet( 2025-12-04T08:52:05.7427138Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T08:52:05.7427517Z encoder_outputs = self.encoder( 2025-12-04T08:52:05.7427894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T08:52:05.7428294Z layer_outputs = layer_module(hidden_states) 2025-12-04T08:52:05.7428666Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:05.7429033Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:05.7429425Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 251, in forward 2025-12-04T08:52:05.7429821Z layer_output = apply_chunking_to_forward( 2025-12-04T08:52:05.7430219Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:52:05.7430620Z return forward_fn(*input_tensors) 2025-12-04T08:52:05.7431043Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-12-04T08:52:05.7431498Z layer_output = self.output(intermediate_output, fourier_output) 2025-12-04T08:52:05.7431917Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 232, in forward 2025-12-04T08:52:05.7432303Z hidden_states = self.dense(hidden_states) 2025-12-04T08:52:05.7432445Z 2025-12-04T08:52:05.7432526Z cudagraph partition due to non gpu ops 2025-12-04T08:52:05.7432764Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:05.7433113Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:05.7433434Z res = mod(**inputs) 2025-12-04T08:52:05.7433780Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T08:52:05.7434148Z outputs = self.fnet( 2025-12-04T08:52:05.7434508Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T08:52:05.7434894Z encoder_outputs = self.encoder( 2025-12-04T08:52:05.7435271Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T08:52:05.7435669Z layer_outputs = layer_module(hidden_states) 2025-12-04T08:52:05.7436076Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:05.7436447Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:05.7436836Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T08:52:05.7437267Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T08:52:05.7437678Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T08:52:05.7438084Z self_outputs = self.self(hidden_states) 2025-12-04T08:52:05.7438541Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T08:52:05.7438974Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T08:52:05.7439186Z 2025-12-04T08:52:05.7439305Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:05.7439725Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:05.7440071Z res = mod(**inputs) 2025-12-04T08:52:05.7440459Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T08:52:05.7440877Z outputs = self.fnet( 2025-12-04T08:52:05.7441262Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T08:52:05.7441652Z encoder_outputs = self.encoder( 2025-12-04T08:52:05.7442029Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T08:52:05.7442427Z layer_outputs = layer_module(hidden_states) 2025-12-04T08:52:05.7442789Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:05.7443158Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:05.7443544Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T08:52:05.7443945Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T08:52:05.7444355Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T08:52:05.7444764Z self_outputs = self.self(hidden_states) 2025-12-04T08:52:05.7445167Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T08:52:05.7445593Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T08:52:05.7445755Z 2025-12-04T08:52:05.7445860Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:05.7446221Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:05.7446556Z res = mod(**inputs) 2025-12-04T08:52:05.7446896Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T08:52:05.7447269Z outputs = self.fnet( 2025-12-04T08:52:05.7447618Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T08:52:05.7447997Z encoder_outputs = self.encoder( 2025-12-04T08:52:05.7448375Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T08:52:05.7448777Z layer_outputs = layer_module(hidden_states) 2025-12-04T08:52:05.7449151Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:05.7449503Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:05.7449915Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T08:52:05.7450319Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T08:52:05.7450711Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T08:52:05.7451104Z self_outputs = self.self(hidden_states) 2025-12-04T08:52:05.7451482Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T08:52:05.7451887Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T08:52:05.7452036Z 2025-12-04T08:52:05.7452138Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:05.7452497Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:05.7452814Z res = mod(**inputs) 2025-12-04T08:52:05.7453175Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T08:52:05.7453545Z outputs = self.fnet( 2025-12-04T08:52:05.7453891Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T08:52:05.7454268Z encoder_outputs = self.encoder( 2025-12-04T08:52:05.7454629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T08:52:05.7455023Z layer_outputs = layer_module(hidden_states) 2025-12-04T08:52:05.7455382Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:05.7455741Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:05.7456114Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T08:52:05.7456537Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T08:52:05.7456956Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T08:52:05.7457348Z self_outputs = self.self(hidden_states) 2025-12-04T08:52:05.7457728Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T08:52:05.7458151Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T08:52:05.7458298Z 2025-12-04T08:52:05.7458408Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:05.7458757Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:05.7459083Z res = mod(**inputs) 2025-12-04T08:52:05.7459438Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T08:52:05.7459816Z outputs = self.fnet( 2025-12-04T08:52:05.7460166Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T08:52:05.7460551Z encoder_outputs = self.encoder( 2025-12-04T08:52:05.7460923Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T08:52:05.7461316Z layer_outputs = layer_module(hidden_states) 2025-12-04T08:52:05.7461683Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:05.7462047Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:05.7462434Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 251, in forward 2025-12-04T08:52:05.7462823Z layer_output = apply_chunking_to_forward( 2025-12-04T08:52:05.7463228Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:52:05.7463625Z return forward_fn(*input_tensors) 2025-12-04T08:52:05.7464087Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 260, in feed_forward_chunk 2025-12-04T08:52:05.7464537Z intermediate_output = self.intermediate(fourier_output) 2025-12-04T08:52:05.7464976Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 218, in forward 2025-12-04T08:52:05.7465376Z hidden_states = self.dense(hidden_states) 2025-12-04T08:52:05.7465520Z 2025-12-04T08:52:05.7465626Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:05.7465987Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:05.7466309Z res = mod(**inputs) 2025-12-04T08:52:05.7466664Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T08:52:05.7467062Z outputs = self.fnet( 2025-12-04T08:52:05.7467421Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T08:52:05.7467806Z encoder_outputs = self.encoder( 2025-12-04T08:52:05.7468185Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T08:52:05.7468586Z layer_outputs = layer_module(hidden_states) 2025-12-04T08:52:05.7468951Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:05.7469319Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:05.7469707Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 251, in forward 2025-12-04T08:52:05.7470106Z layer_output = apply_chunking_to_forward( 2025-12-04T08:52:05.7470504Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:52:05.7470910Z return forward_fn(*input_tensors) 2025-12-04T08:52:05.7471328Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 260, in feed_forward_chunk 2025-12-04T08:52:05.7471777Z intermediate_output = self.intermediate(fourier_output) 2025-12-04T08:52:05.7472205Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 219, in forward 2025-12-04T08:52:05.7472633Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:52:05.7473025Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 62, in forward 2025-12-04T08:52:05.7473463Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-12-04T08:52:05.7473702Z 2025-12-04T08:52:05.7473809Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:05.7474175Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:05.7474503Z res = mod(**inputs) 2025-12-04T08:52:05.7474853Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T08:52:05.7475239Z outputs = self.fnet( 2025-12-04T08:52:05.7475600Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T08:52:05.7475981Z encoder_outputs = self.encoder( 2025-12-04T08:52:05.7476360Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T08:52:05.7476762Z layer_outputs = layer_module(hidden_states) 2025-12-04T08:52:05.7477135Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:05.7477507Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:05.7477943Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 251, in forward 2025-12-04T08:52:05.7478437Z layer_output = apply_chunking_to_forward( 2025-12-04T08:52:05.7478904Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:52:05.7479368Z return forward_fn(*input_tensors) 2025-12-04T08:52:05.7479834Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-12-04T08:52:05.7480341Z layer_output = self.output(intermediate_output, fourier_output) 2025-12-04T08:52:05.7480805Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 232, in forward 2025-12-04T08:52:05.7481263Z hidden_states = self.dense(hidden_states) 2025-12-04T08:52:05.7481433Z 2025-12-04T08:52:05.7481520Z cudagraph partition due to non gpu ops 2025-12-04T08:52:05.7481772Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:05.7482141Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:05.7482476Z res = mod(**inputs) 2025-12-04T08:52:05.7482844Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T08:52:05.7483228Z outputs = self.fnet( 2025-12-04T08:52:05.7483593Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T08:52:05.7483990Z encoder_outputs = self.encoder( 2025-12-04T08:52:05.7484380Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T08:52:05.7484786Z layer_outputs = layer_module(hidden_states) 2025-12-04T08:52:05.7485171Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:05.7485546Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:05.7485942Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T08:52:05.7486362Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T08:52:05.7486799Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T08:52:05.7487230Z self_outputs = self.self(hidden_states) 2025-12-04T08:52:05.7487650Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T08:52:05.7488104Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T08:52:05.7488282Z 2025-12-04T08:52:05.7488574Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:05.7488955Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:05.7489300Z res = mod(**inputs) 2025-12-04T08:52:05.7489686Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T08:52:05.7490109Z outputs = self.fnet( 2025-12-04T08:52:05.7490502Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T08:52:05.7490918Z encoder_outputs = self.encoder( 2025-12-04T08:52:05.7491332Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T08:52:05.7491773Z layer_outputs = layer_module(hidden_states) 2025-12-04T08:52:05.7492170Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:05.7492574Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:05.7493032Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T08:52:05.7493492Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T08:52:05.7493925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T08:52:05.7494369Z self_outputs = self.self(hidden_states) 2025-12-04T08:52:05.7494808Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T08:52:05.7495260Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T08:52:05.7495429Z 2025-12-04T08:52:05.7495545Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:05.7495938Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:05.7496314Z res = mod(**inputs) 2025-12-04T08:52:05.7496700Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T08:52:05.7497120Z outputs = self.fnet( 2025-12-04T08:52:05.7497512Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T08:52:05.7498103Z encoder_outputs = self.encoder( 2025-12-04T08:52:05.7498515Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T08:52:05.7498952Z layer_outputs = layer_module(hidden_states) 2025-12-04T08:52:05.7499202Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:05.7499299Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:05.7499571Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T08:52:05.7499691Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T08:52:05.7499964Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T08:52:05.7500052Z self_outputs = self.self(hidden_states) 2025-12-04T08:52:05.7500328Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T08:52:05.7500433Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T08:52:05.7500437Z 2025-12-04T08:52:05.7500547Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:05.7500761Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:05.7500828Z res = mod(**inputs) 2025-12-04T08:52:05.7501101Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T08:52:05.7501172Z outputs = self.fnet( 2025-12-04T08:52:05.7501448Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T08:52:05.7501526Z encoder_outputs = self.encoder( 2025-12-04T08:52:05.7501783Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T08:52:05.7501882Z layer_outputs = layer_module(hidden_states) 2025-12-04T08:52:05.7502122Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:05.7502202Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:05.7502463Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T08:52:05.7502563Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T08:52:05.7502828Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T08:52:05.7502948Z self_outputs = self.self(hidden_states) 2025-12-04T08:52:05.7503201Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T08:52:05.7503334Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T08:52:05.7503338Z 2025-12-04T08:52:05.7503443Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:05.7503656Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:05.7503723Z res = mod(**inputs) 2025-12-04T08:52:05.7503978Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T08:52:05.7504053Z outputs = self.fnet( 2025-12-04T08:52:05.7504309Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T08:52:05.7504403Z encoder_outputs = self.encoder( 2025-12-04T08:52:05.7504666Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T08:52:05.7504753Z layer_outputs = layer_module(hidden_states) 2025-12-04T08:52:05.7504994Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:05.7505075Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:05.7505328Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 251, in forward 2025-12-04T08:52:05.7505422Z layer_output = apply_chunking_to_forward( 2025-12-04T08:52:05.7505695Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:52:05.7505782Z return forward_fn(*input_tensors) 2025-12-04T08:52:05.7506072Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 260, in feed_forward_chunk 2025-12-04T08:52:05.7506192Z intermediate_output = self.intermediate(fourier_output) 2025-12-04T08:52:05.7506451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 218, in forward 2025-12-04T08:52:05.7506538Z hidden_states = self.dense(hidden_states) 2025-12-04T08:52:05.7506541Z 2025-12-04T08:52:05.7506647Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:05.7506858Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:05.7506925Z res = mod(**inputs) 2025-12-04T08:52:05.7507184Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T08:52:05.7507253Z outputs = self.fnet( 2025-12-04T08:52:05.7507509Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T08:52:05.7507592Z encoder_outputs = self.encoder( 2025-12-04T08:52:05.7507845Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T08:52:05.7507939Z layer_outputs = layer_module(hidden_states) 2025-12-04T08:52:05.7508168Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:05.7508246Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:05.7508507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 251, in forward 2025-12-04T08:52:05.7508594Z layer_output = apply_chunking_to_forward( 2025-12-04T08:52:05.7508862Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:52:05.7508947Z return forward_fn(*input_tensors) 2025-12-04T08:52:05.7509278Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 260, in feed_forward_chunk 2025-12-04T08:52:05.7509405Z intermediate_output = self.intermediate(fourier_output) 2025-12-04T08:52:05.7509675Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 219, in forward 2025-12-04T08:52:05.7509786Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:52:05.7510014Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 62, in forward 2025-12-04T08:52:05.7510197Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-12-04T08:52:05.7510201Z 2025-12-04T08:52:05.7510316Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:05.7510540Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:05.7510607Z res = mod(**inputs) 2025-12-04T08:52:05.7510875Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T08:52:05.7510945Z outputs = self.fnet( 2025-12-04T08:52:05.7511208Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T08:52:05.7511294Z encoder_outputs = self.encoder( 2025-12-04T08:52:05.7511564Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T08:52:05.7511662Z layer_outputs = layer_module(hidden_states) 2025-12-04T08:52:05.7511911Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:05.7511992Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:05.7512258Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 251, in forward 2025-12-04T08:52:05.7512344Z layer_output = apply_chunking_to_forward( 2025-12-04T08:52:05.7512619Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:52:05.7512698Z return forward_fn(*input_tensors) 2025-12-04T08:52:05.7512989Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-12-04T08:52:05.7513127Z layer_output = self.output(intermediate_output, fourier_output) 2025-12-04T08:52:05.7513385Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 232, in forward 2025-12-04T08:52:05.7513470Z hidden_states = self.dense(hidden_states) 2025-12-04T08:52:05.7513482Z 2025-12-04T08:52:05.7513567Z cudagraph partition due to non gpu ops 2025-12-04T08:52:05.7513675Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:05.7513891Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:05.7513958Z res = mod(**inputs) 2025-12-04T08:52:05.7514214Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T08:52:05.7514296Z outputs = self.fnet( 2025-12-04T08:52:05.7514570Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T08:52:05.7514650Z encoder_outputs = self.encoder( 2025-12-04T08:52:05.7514926Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T08:52:05.7515021Z layer_outputs = layer_module(hidden_states) 2025-12-04T08:52:05.7515272Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:05.7515359Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:05.7515662Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T08:52:05.7515779Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T08:52:05.7516068Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T08:52:05.7516161Z self_outputs = self.self(hidden_states) 2025-12-04T08:52:05.7516430Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T08:52:05.7516541Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T08:52:05.7516545Z 2025-12-04T08:52:05.7516663Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:05.7516877Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:05.7516965Z res = mod(**inputs) 2025-12-04T08:52:05.7517242Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T08:52:05.7517316Z outputs = self.fnet( 2025-12-04T08:52:05.7517592Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T08:52:05.7517672Z encoder_outputs = self.encoder( 2025-12-04T08:52:05.7517938Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T08:52:05.7518037Z layer_outputs = layer_module(hidden_states) 2025-12-04T08:52:05.7518334Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:05.7518430Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:05.7518703Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T08:52:05.7518815Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T08:52:05.7519097Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T08:52:05.7519188Z self_outputs = self.self(hidden_states) 2025-12-04T08:52:05.7519462Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T08:52:05.7519585Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T08:52:05.7519589Z 2025-12-04T08:52:05.7519703Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:05.7519934Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:05.7520017Z res = mod(**inputs) 2025-12-04T08:52:05.7520285Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T08:52:05.7520369Z outputs = self.fnet( 2025-12-04T08:52:05.7520636Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T08:52:05.7520891Z encoder_outputs = self.encoder( 2025-12-04T08:52:05.7521182Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T08:52:05.7521277Z layer_outputs = layer_module(hidden_states) 2025-12-04T08:52:05.7521527Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:05.7521610Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:05.7521877Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T08:52:05.7521992Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T08:52:05.7522335Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T08:52:05.7522431Z self_outputs = self.self(hidden_states) 2025-12-04T08:52:05.7522704Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T08:52:05.7522838Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T08:52:05.7522843Z 2025-12-04T08:52:05.7522964Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:05.7523178Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:05.7523257Z res = mod(**inputs) 2025-12-04T08:52:05.7523527Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T08:52:05.7523598Z outputs = self.fnet( 2025-12-04T08:52:05.7523918Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T08:52:05.7523996Z encoder_outputs = self.encoder( 2025-12-04T08:52:05.7524266Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T08:52:05.7524369Z layer_outputs = layer_module(hidden_states) 2025-12-04T08:52:05.7524609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:05.7524698Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:05.7524967Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T08:52:05.7525073Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T08:52:05.7525348Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T08:52:05.7525436Z self_outputs = self.self(hidden_states) 2025-12-04T08:52:05.7525711Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T08:52:05.7525827Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T08:52:05.7525832Z 2025-12-04T08:52:05.7525941Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:05.7526162Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:05.7526232Z res = mod(**inputs) 2025-12-04T08:52:05.7526505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T08:52:05.7526584Z outputs = self.fnet( 2025-12-04T08:52:05.7526855Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T08:52:05.7526942Z encoder_outputs = self.encoder( 2025-12-04T08:52:05.7527216Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T08:52:05.7527308Z layer_outputs = layer_module(hidden_states) 2025-12-04T08:52:05.7527557Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:05.7527643Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:05.7527916Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 251, in forward 2025-12-04T08:52:05.7528014Z layer_output = apply_chunking_to_forward( 2025-12-04T08:52:05.7528300Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:52:05.7528391Z return forward_fn(*input_tensors) 2025-12-04T08:52:05.7528697Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 260, in feed_forward_chunk 2025-12-04T08:52:05.7528858Z intermediate_output = self.intermediate(fourier_output) 2025-12-04T08:52:05.7529138Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 218, in forward 2025-12-04T08:52:05.7529250Z hidden_states = self.dense(hidden_states) 2025-12-04T08:52:05.7529253Z 2025-12-04T08:52:05.7529372Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:05.7529587Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:05.7529657Z res = mod(**inputs) 2025-12-04T08:52:05.7529925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T08:52:05.7529991Z outputs = self.fnet( 2025-12-04T08:52:05.7530238Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T08:52:05.7530349Z encoder_outputs = self.encoder( 2025-12-04T08:52:05.7530604Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T08:52:05.7530697Z layer_outputs = layer_module(hidden_states) 2025-12-04T08:52:05.7530921Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:05.7531001Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:05.7531257Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 251, in forward 2025-12-04T08:52:05.7531340Z layer_output = apply_chunking_to_forward( 2025-12-04T08:52:05.7531611Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:52:05.7531688Z return forward_fn(*input_tensors) 2025-12-04T08:52:05.7531970Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 260, in feed_forward_chunk 2025-12-04T08:52:05.7532091Z intermediate_output = self.intermediate(fourier_output) 2025-12-04T08:52:05.7532346Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 219, in forward 2025-12-04T08:52:05.7532460Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:52:05.7532691Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 62, in forward 2025-12-04T08:52:05.7532875Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-12-04T08:52:05.7532879Z 2025-12-04T08:52:05.7532991Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:05.7533196Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:05.7533265Z res = mod(**inputs) 2025-12-04T08:52:05.7533535Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T08:52:05.7533601Z outputs = self.fnet( 2025-12-04T08:52:05.7533856Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T08:52:05.7533933Z encoder_outputs = self.encoder( 2025-12-04T08:52:05.7534187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T08:52:05.7534281Z layer_outputs = layer_module(hidden_states) 2025-12-04T08:52:05.7534511Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:05.7534592Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:05.7534854Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 251, in forward 2025-12-04T08:52:05.7534938Z layer_output = apply_chunking_to_forward( 2025-12-04T08:52:05.7535250Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:52:05.7535330Z return forward_fn(*input_tensors) 2025-12-04T08:52:05.7535633Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-12-04T08:52:05.7535770Z layer_output = self.output(intermediate_output, fourier_output) 2025-12-04T08:52:05.7536029Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 232, in forward 2025-12-04T08:52:05.7536117Z hidden_states = self.dense(hidden_states) 2025-12-04T08:52:05.7536120Z 2025-12-04T08:52:05.7536201Z cudagraph partition due to non gpu ops 2025-12-04T08:52:05.7536302Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:05.7536536Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:05.7536605Z res = mod(**inputs) 2025-12-04T08:52:05.7536855Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T08:52:05.7536930Z outputs = self.fnet( 2025-12-04T08:52:05.7537175Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T08:52:05.7537254Z encoder_outputs = self.encoder( 2025-12-04T08:52:05.7537502Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T08:52:05.7537587Z layer_outputs = layer_module(hidden_states) 2025-12-04T08:52:05.7537831Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:05.7537913Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:05.7538169Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T08:52:05.7538276Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T08:52:05.7538533Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T08:52:05.7538626Z self_outputs = self.self(hidden_states) 2025-12-04T08:52:05.7538879Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T08:52:05.7538982Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T08:52:05.7538986Z 2025-12-04T08:52:05.7539096Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:05.7539300Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:05.7539375Z res = mod(**inputs) 2025-12-04T08:52:05.7539639Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T08:52:05.7539708Z outputs = self.fnet( 2025-12-04T08:52:05.7539974Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T08:52:05.7540051Z encoder_outputs = self.encoder( 2025-12-04T08:52:05.7540304Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T08:52:05.7540396Z layer_outputs = layer_module(hidden_states) 2025-12-04T08:52:05.7540623Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:05.7540707Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:05.7540959Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T08:52:05.7541060Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T08:52:05.7541356Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T08:52:05.7541441Z self_outputs = self.self(hidden_states) 2025-12-04T08:52:05.7541725Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T08:52:05.7541830Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T08:52:05.7541833Z 2025-12-04T08:52:05.7541938Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:05.7542153Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:05.7542218Z res = mod(**inputs) 2025-12-04T08:52:05.7542484Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T08:52:05.7542582Z outputs = self.fnet( 2025-12-04T08:52:05.7542851Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T08:52:05.7542929Z encoder_outputs = self.encoder( 2025-12-04T08:52:05.7543180Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T08:52:05.7543267Z layer_outputs = layer_module(hidden_states) 2025-12-04T08:52:05.7543501Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:05.7543579Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:05.7543835Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T08:52:05.7543933Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T08:52:05.7544193Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T08:52:05.7544286Z self_outputs = self.self(hidden_states) 2025-12-04T08:52:05.7544546Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T08:52:05.7544647Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T08:52:05.7544653Z 2025-12-04T08:52:05.7544764Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:05.7544971Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:05.7545044Z res = mod(**inputs) 2025-12-04T08:52:05.7545305Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T08:52:05.7545373Z outputs = self.fnet( 2025-12-04T08:52:05.7545638Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T08:52:05.7545716Z encoder_outputs = self.encoder( 2025-12-04T08:52:05.7545982Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T08:52:05.7546078Z layer_outputs = layer_module(hidden_states) 2025-12-04T08:52:05.7546313Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:05.7546400Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:05.7546665Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T08:52:05.7546761Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T08:52:05.7547068Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T08:52:05.7547150Z self_outputs = self.self(hidden_states) 2025-12-04T08:52:05.7547444Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T08:52:05.7547546Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T08:52:05.7547550Z 2025-12-04T08:52:05.7547655Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:05.7547884Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:05.7547950Z res = mod(**inputs) 2025-12-04T08:52:05.7548206Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T08:52:05.7548281Z outputs = self.fnet( 2025-12-04T08:52:05.7548537Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T08:52:05.7548617Z encoder_outputs = self.encoder( 2025-12-04T08:52:05.7548872Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T08:52:05.7548980Z layer_outputs = layer_module(hidden_states) 2025-12-04T08:52:05.7549216Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:05.7549296Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:05.7549557Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 251, in forward 2025-12-04T08:52:05.7549643Z layer_output = apply_chunking_to_forward( 2025-12-04T08:52:05.7549913Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:52:05.7549998Z return forward_fn(*input_tensors) 2025-12-04T08:52:05.7550284Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 260, in feed_forward_chunk 2025-12-04T08:52:05.7550402Z intermediate_output = self.intermediate(fourier_output) 2025-12-04T08:52:05.7550666Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 218, in forward 2025-12-04T08:52:05.7550752Z hidden_states = self.dense(hidden_states) 2025-12-04T08:52:05.7550755Z 2025-12-04T08:52:05.7550867Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:05.7551072Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:05.7551138Z res = mod(**inputs) 2025-12-04T08:52:05.7551402Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T08:52:05.7551470Z outputs = self.fnet( 2025-12-04T08:52:05.7551731Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T08:52:05.7551806Z encoder_outputs = self.encoder( 2025-12-04T08:52:05.7552064Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T08:52:05.7552158Z layer_outputs = layer_module(hidden_states) 2025-12-04T08:52:05.7552386Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:05.7552469Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:05.7552734Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 251, in forward 2025-12-04T08:52:05.7552821Z layer_output = apply_chunking_to_forward( 2025-12-04T08:52:05.7553098Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:52:05.7553178Z return forward_fn(*input_tensors) 2025-12-04T08:52:05.7553464Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 260, in feed_forward_chunk 2025-12-04T08:52:05.7553628Z intermediate_output = self.intermediate(fourier_output) 2025-12-04T08:52:05.7553883Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 219, in forward 2025-12-04T08:52:05.7554001Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:52:05.7554299Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 62, in forward 2025-12-04T08:52:05.7554482Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-12-04T08:52:05.7554486Z 2025-12-04T08:52:05.7554599Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:05.7554803Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:05.7554869Z res = mod(**inputs) 2025-12-04T08:52:05.7555129Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T08:52:05.7555216Z outputs = self.fnet( 2025-12-04T08:52:05.7555481Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T08:52:05.7555555Z encoder_outputs = self.encoder( 2025-12-04T08:52:05.7555811Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T08:52:05.7555907Z layer_outputs = layer_module(hidden_states) 2025-12-04T08:52:05.7556137Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:05.7556225Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:05.7556480Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 251, in forward 2025-12-04T08:52:05.7556563Z layer_output = apply_chunking_to_forward( 2025-12-04T08:52:05.7556847Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:52:05.7556926Z return forward_fn(*input_tensors) 2025-12-04T08:52:05.7557214Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-12-04T08:52:05.7557351Z layer_output = self.output(intermediate_output, fourier_output) 2025-12-04T08:52:05.7557607Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 232, in forward 2025-12-04T08:52:05.7557694Z hidden_states = self.dense(hidden_states) 2025-12-04T08:52:05.7557698Z 2025-12-04T08:52:05.7557782Z cudagraph partition due to non gpu ops 2025-12-04T08:52:05.7557886Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:05.7558097Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:05.7558166Z res = mod(**inputs) 2025-12-04T08:52:05.7558507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T08:52:05.7558589Z outputs = self.fnet( 2025-12-04T08:52:05.7558871Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T08:52:05.7558960Z encoder_outputs = self.encoder( 2025-12-04T08:52:05.7559247Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T08:52:05.7559339Z layer_outputs = layer_module(hidden_states) 2025-12-04T08:52:05.7559588Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:05.7559673Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:05.7559949Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T08:52:05.7560100Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T08:52:05.7560355Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T08:52:05.7560468Z self_outputs = self.self(hidden_states) 2025-12-04T08:52:05.7560724Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T08:52:05.7560829Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T08:52:05.7560841Z 2025-12-04T08:52:05.7560946Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:05.7561151Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:05.7561228Z res = mod(**inputs) 2025-12-04T08:52:05.7561483Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T08:52:05.7561572Z outputs = self.fnet( 2025-12-04T08:52:05.7561836Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T08:52:05.7561913Z encoder_outputs = self.encoder( 2025-12-04T08:52:05.7562178Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T08:52:05.7562263Z layer_outputs = layer_module(hidden_states) 2025-12-04T08:52:05.7562493Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:05.7562581Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:05.7562833Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T08:52:05.7562934Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T08:52:05.7563200Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T08:52:05.7563282Z self_outputs = self.self(hidden_states) 2025-12-04T08:52:05.7563541Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T08:52:05.7563647Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T08:52:05.7563651Z 2025-12-04T08:52:05.7563755Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:05.7563966Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:05.7564045Z res = mod(**inputs) 2025-12-04T08:52:05.7564298Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T08:52:05.7564363Z outputs = self.fnet( 2025-12-04T08:52:05.7564626Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T08:52:05.7564709Z encoder_outputs = self.encoder( 2025-12-04T08:52:05.7564955Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T08:52:05.7565040Z layer_outputs = layer_module(hidden_states) 2025-12-04T08:52:05.7565268Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:05.7565344Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:05.7565596Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T08:52:05.7565692Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T08:52:05.7565937Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T08:52:05.7566027Z self_outputs = self.self(hidden_states) 2025-12-04T08:52:05.7566306Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T08:52:05.7566415Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T08:52:05.7566419Z 2025-12-04T08:52:05.7566538Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:05.7566737Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:05.7566809Z res = mod(**inputs) 2025-12-04T08:52:05.7567056Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T08:52:05.7567124Z outputs = self.fnet( 2025-12-04T08:52:05.7567379Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T08:52:05.7567451Z encoder_outputs = self.encoder( 2025-12-04T08:52:05.7567730Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T08:52:05.7567815Z layer_outputs = layer_module(hidden_states) 2025-12-04T08:52:05.7568038Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:05.7568125Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:05.7568376Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T08:52:05.7568471Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T08:52:05.7568728Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T08:52:05.7568809Z self_outputs = self.self(hidden_states) 2025-12-04T08:52:05.7569068Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T08:52:05.7569172Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T08:52:05.7569176Z 2025-12-04T08:52:05.7569277Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:05.7569483Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:05.7569551Z res = mod(**inputs) 2025-12-04T08:52:05.7569810Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T08:52:05.7569874Z outputs = self.fnet( 2025-12-04T08:52:05.7570122Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T08:52:05.7570205Z encoder_outputs = self.encoder( 2025-12-04T08:52:05.7570465Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T08:52:05.7570550Z layer_outputs = layer_module(hidden_states) 2025-12-04T08:52:05.7570786Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:05.7570864Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:05.7571119Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 251, in forward 2025-12-04T08:52:05.7571202Z layer_output = apply_chunking_to_forward( 2025-12-04T08:52:05.7571470Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:52:05.7571553Z return forward_fn(*input_tensors) 2025-12-04T08:52:05.7571835Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 260, in feed_forward_chunk 2025-12-04T08:52:05.7571956Z intermediate_output = self.intermediate(fourier_output) 2025-12-04T08:52:05.7572249Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 218, in forward 2025-12-04T08:52:05.7572333Z hidden_states = self.dense(hidden_states) 2025-12-04T08:52:05.7572337Z 2025-12-04T08:52:05.7572446Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:05.7572643Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:05.7572725Z res = mod(**inputs) 2025-12-04T08:52:05.7572988Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T08:52:05.7573054Z outputs = self.fnet( 2025-12-04T08:52:05.7573307Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T08:52:05.7573379Z encoder_outputs = self.encoder( 2025-12-04T08:52:05.7573624Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T08:52:05.7573734Z layer_outputs = layer_module(hidden_states) 2025-12-04T08:52:05.7573954Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:05.7574038Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:05.7574284Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 251, in forward 2025-12-04T08:52:05.7574368Z layer_output = apply_chunking_to_forward( 2025-12-04T08:52:05.7574636Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:52:05.7574712Z return forward_fn(*input_tensors) 2025-12-04T08:52:05.7574989Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 260, in feed_forward_chunk 2025-12-04T08:52:05.7575113Z intermediate_output = self.intermediate(fourier_output) 2025-12-04T08:52:05.7575365Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 219, in forward 2025-12-04T08:52:05.7575483Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:52:05.7575697Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 62, in forward 2025-12-04T08:52:05.7575875Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-12-04T08:52:05.7575879Z 2025-12-04T08:52:05.7575990Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:05.7576189Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:05.7576262Z res = mod(**inputs) 2025-12-04T08:52:05.7576512Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T08:52:05.7576580Z outputs = self.fnet( 2025-12-04T08:52:05.7576840Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T08:52:05.7576914Z encoder_outputs = self.encoder( 2025-12-04T08:52:05.7577169Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T08:52:05.7577264Z layer_outputs = layer_module(hidden_states) 2025-12-04T08:52:05.7577485Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:05.7577570Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:05.7577816Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 251, in forward 2025-12-04T08:52:05.7577900Z layer_output = apply_chunking_to_forward( 2025-12-04T08:52:05.7578167Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:52:05.7578283Z return forward_fn(*input_tensors) 2025-12-04T08:52:05.7578568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-12-04T08:52:05.7578696Z layer_output = self.output(intermediate_output, fourier_output) 2025-12-04T08:52:05.7578966Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 232, in forward 2025-12-04T08:52:05.7579052Z hidden_states = self.dense(hidden_states) 2025-12-04T08:52:05.7579055Z 2025-12-04T08:52:05.7579137Z cudagraph partition due to non gpu ops 2025-12-04T08:52:05.7579237Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:05.7579439Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:05.7579503Z res = mod(**inputs) 2025-12-04T08:52:05.7579771Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T08:52:05.7579836Z outputs = self.fnet( 2025-12-04T08:52:05.7580076Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T08:52:05.7580158Z encoder_outputs = self.encoder( 2025-12-04T08:52:05.7580400Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T08:52:05.7580482Z layer_outputs = layer_module(hidden_states) 2025-12-04T08:52:05.7580709Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:05.7580784Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:05.7581040Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T08:52:05.7581136Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T08:52:05.7581379Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T08:52:05.7581467Z self_outputs = self.self(hidden_states) 2025-12-04T08:52:05.7581710Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T08:52:05.7581816Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T08:52:05.7581819Z 2025-12-04T08:52:05.7581919Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:05.7582113Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:05.7582183Z res = mod(**inputs) 2025-12-04T08:52:05.7582424Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T08:52:05.7582491Z outputs = self.fnet( 2025-12-04T08:52:05.7582740Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T08:52:05.7582811Z encoder_outputs = self.encoder( 2025-12-04T08:52:05.7583066Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T08:52:05.7583149Z layer_outputs = layer_module(hidden_states) 2025-12-04T08:52:05.7583372Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:05.7583458Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:05.7583703Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T08:52:05.7583805Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T08:52:05.7584054Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T08:52:05.7584170Z self_outputs = self.self(hidden_states) 2025-12-04T08:52:05.7584432Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T08:52:05.7584535Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T08:52:05.7584556Z 2025-12-04T08:52:05.7584660Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:05.7584876Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:05.7584941Z res = mod(**inputs) 2025-12-04T08:52:05.7585194Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T08:52:05.7585259Z outputs = self.fnet( 2025-12-04T08:52:05.7585503Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T08:52:05.7585602Z encoder_outputs = self.encoder( 2025-12-04T08:52:05.7585851Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T08:52:05.7585940Z layer_outputs = layer_module(hidden_states) 2025-12-04T08:52:05.7586164Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:05.7586242Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:05.7586496Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T08:52:05.7586592Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T08:52:05.7586838Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T08:52:05.7586925Z self_outputs = self.self(hidden_states) 2025-12-04T08:52:05.7587176Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T08:52:05.7587289Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T08:52:05.7587293Z 2025-12-04T08:52:05.7587396Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:05.7587598Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:05.7587673Z res = mod(**inputs) 2025-12-04T08:52:05.7587932Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T08:52:05.7588006Z outputs = self.fnet( 2025-12-04T08:52:05.7588252Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T08:52:05.7588325Z encoder_outputs = self.encoder( 2025-12-04T08:52:05.7588577Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T08:52:05.7588665Z layer_outputs = layer_module(hidden_states) 2025-12-04T08:52:05.7588891Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:05.7588975Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:05.7589224Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T08:52:05.7589327Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T08:52:05.7589576Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T08:52:05.7589656Z self_outputs = self.self(hidden_states) 2025-12-04T08:52:05.7589913Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T08:52:05.7590015Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T08:52:05.7590018Z 2025-12-04T08:52:05.7590168Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:05.7590373Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:05.7590439Z res = mod(**inputs) 2025-12-04T08:52:05.7590724Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T08:52:05.7590792Z outputs = self.fnet( 2025-12-04T08:52:05.7591050Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T08:52:05.7591133Z encoder_outputs = self.encoder( 2025-12-04T08:52:05.7591388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T08:52:05.7591481Z layer_outputs = layer_module(hidden_states) 2025-12-04T08:52:05.7591732Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:05.7591815Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:05.7592090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 251, in forward 2025-12-04T08:52:05.7592178Z layer_output = apply_chunking_to_forward( 2025-12-04T08:52:05.7592451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:52:05.7592538Z return forward_fn(*input_tensors) 2025-12-04T08:52:05.7592825Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 260, in feed_forward_chunk 2025-12-04T08:52:05.7592948Z intermediate_output = self.intermediate(fourier_output) 2025-12-04T08:52:05.7593199Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 218, in forward 2025-12-04T08:52:05.7593285Z hidden_states = self.dense(hidden_states) 2025-12-04T08:52:05.7593292Z 2025-12-04T08:52:05.7593404Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:05.7593606Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:05.7593681Z res = mod(**inputs) 2025-12-04T08:52:05.7593936Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T08:52:05.7594008Z outputs = self.fnet( 2025-12-04T08:52:05.7594284Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T08:52:05.7594361Z encoder_outputs = self.encoder( 2025-12-04T08:52:05.7594629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T08:52:05.7594729Z layer_outputs = layer_module(hidden_states) 2025-12-04T08:52:05.7594973Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:05.7595062Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:05.7595329Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 251, in forward 2025-12-04T08:52:05.7595422Z layer_output = apply_chunking_to_forward( 2025-12-04T08:52:05.7595713Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:52:05.7595795Z return forward_fn(*input_tensors) 2025-12-04T08:52:05.7596102Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 260, in feed_forward_chunk 2025-12-04T08:52:05.7596225Z intermediate_output = self.intermediate(fourier_output) 2025-12-04T08:52:05.7596491Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 219, in forward 2025-12-04T08:52:05.7596658Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:52:05.7596893Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 62, in forward 2025-12-04T08:52:05.7597106Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-12-04T08:52:05.7597117Z 2025-12-04T08:52:05.7597231Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:05.7597446Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:05.7597524Z res = mod(**inputs) 2025-12-04T08:52:05.7597794Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T08:52:05.7597867Z outputs = self.fnet( 2025-12-04T08:52:05.7598166Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T08:52:05.7598480Z encoder_outputs = self.encoder( 2025-12-04T08:52:05.7598768Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T08:52:05.7598863Z layer_outputs = layer_module(hidden_states) 2025-12-04T08:52:05.7599113Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:05.7599210Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:05.7599487Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 251, in forward 2025-12-04T08:52:05.7599581Z layer_output = apply_chunking_to_forward( 2025-12-04T08:52:05.7599885Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:52:05.7599975Z return forward_fn(*input_tensors) 2025-12-04T08:52:05.7600294Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-12-04T08:52:05.7600424Z layer_output = self.output(intermediate_output, fourier_output) 2025-12-04T08:52:05.7600678Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 232, in forward 2025-12-04T08:52:05.7600771Z hidden_states = self.dense(hidden_states) 2025-12-04T08:52:05.7600775Z 2025-12-04T08:52:05.7600859Z cudagraph partition due to non gpu ops 2025-12-04T08:52:05.7600973Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:05.7601186Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:05.7601254Z res = mod(**inputs) 2025-12-04T08:52:05.7601519Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T08:52:05.7601592Z outputs = self.fnet( 2025-12-04T08:52:05.7601850Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T08:52:05.7601936Z encoder_outputs = self.encoder( 2025-12-04T08:52:05.7602189Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T08:52:05.7602283Z layer_outputs = layer_module(hidden_states) 2025-12-04T08:52:05.7602516Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:05.7602597Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:05.7602854Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T08:52:05.7602954Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T08:52:05.7603262Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T08:52:05.7603352Z self_outputs = self.self(hidden_states) 2025-12-04T08:52:05.7603607Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T08:52:05.7603737Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T08:52:05.7603741Z 2025-12-04T08:52:05.7603845Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:05.7604049Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:05.7604123Z res = mod(**inputs) 2025-12-04T08:52:05.7604377Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T08:52:05.7604451Z outputs = self.fnet( 2025-12-04T08:52:05.7604706Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T08:52:05.7604802Z encoder_outputs = self.encoder( 2025-12-04T08:52:05.7605064Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T08:52:05.7605149Z layer_outputs = layer_module(hidden_states) 2025-12-04T08:52:05.7605382Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:05.7605468Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:05.7605723Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T08:52:05.7605832Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T08:52:05.7606088Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T08:52:05.7606172Z self_outputs = self.self(hidden_states) 2025-12-04T08:52:05.7606434Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T08:52:05.7606539Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T08:52:05.7606543Z 2025-12-04T08:52:05.7606654Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:05.7606867Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:05.7606934Z res = mod(**inputs) 2025-12-04T08:52:05.7607197Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T08:52:05.7607267Z outputs = self.fnet( 2025-12-04T08:52:05.7607522Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T08:52:05.7607603Z encoder_outputs = self.encoder( 2025-12-04T08:52:05.7607863Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T08:52:05.7607958Z layer_outputs = layer_module(hidden_states) 2025-12-04T08:52:05.7608186Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:05.7608272Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:05.7608546Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T08:52:05.7608653Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T08:52:05.7608930Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T08:52:05.7609017Z self_outputs = self.self(hidden_states) 2025-12-04T08:52:05.7609285Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T08:52:05.7609402Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T08:52:05.7609444Z 2025-12-04T08:52:05.7609556Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:05.7609771Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:05.7609867Z res = mod(**inputs) 2025-12-04T08:52:05.7610138Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T08:52:05.7610218Z outputs = self.fnet( 2025-12-04T08:52:05.7610491Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T08:52:05.7610565Z encoder_outputs = self.encoder( 2025-12-04T08:52:05.7610826Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T08:52:05.7610931Z layer_outputs = layer_module(hidden_states) 2025-12-04T08:52:05.7611162Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:05.7611250Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:05.7611502Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 248, in forward 2025-12-04T08:52:05.7611611Z self_fourier_outputs = self.fourier(hidden_states) 2025-12-04T08:52:05.7611881Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 201, in forward 2025-12-04T08:52:05.7611967Z self_outputs = self.self(hidden_states) 2025-12-04T08:52:05.7612246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 180, in forward 2025-12-04T08:52:05.7612354Z outputs = self.fourier_transform(hidden_states).real 2025-12-04T08:52:05.7612358Z 2025-12-04T08:52:05.7612477Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:05.7612698Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:05.7612768Z res = mod(**inputs) 2025-12-04T08:52:05.7613050Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T08:52:05.7613123Z outputs = self.fnet( 2025-12-04T08:52:05.7613392Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T08:52:05.7613479Z encoder_outputs = self.encoder( 2025-12-04T08:52:05.7613749Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T08:52:05.7613845Z layer_outputs = layer_module(hidden_states) 2025-12-04T08:52:05.7614085Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:05.7614171Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:05.7614451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 251, in forward 2025-12-04T08:52:05.7614541Z layer_output = apply_chunking_to_forward( 2025-12-04T08:52:05.7614831Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:52:05.7614916Z return forward_fn(*input_tensors) 2025-12-04T08:52:05.7615219Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 260, in feed_forward_chunk 2025-12-04T08:52:05.7615353Z intermediate_output = self.intermediate(fourier_output) 2025-12-04T08:52:05.7615623Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 218, in forward 2025-12-04T08:52:05.7615712Z hidden_states = self.dense(hidden_states) 2025-12-04T08:52:05.7615724Z 2025-12-04T08:52:05.7615833Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:05.7616085Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:05.7616166Z res = mod(**inputs) 2025-12-04T08:52:05.7616448Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T08:52:05.7616546Z outputs = self.fnet( 2025-12-04T08:52:05.7616830Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T08:52:05.7616908Z encoder_outputs = self.encoder( 2025-12-04T08:52:05.7617190Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T08:52:05.7617279Z layer_outputs = layer_module(hidden_states) 2025-12-04T08:52:05.7617525Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:05.7617637Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:05.7617910Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 251, in forward 2025-12-04T08:52:05.7618001Z layer_output = apply_chunking_to_forward( 2025-12-04T08:52:05.7618297Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:52:05.7618379Z return forward_fn(*input_tensors) 2025-12-04T08:52:05.7618687Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 260, in feed_forward_chunk 2025-12-04T08:52:05.7618812Z intermediate_output = self.intermediate(fourier_output) 2025-12-04T08:52:05.7619082Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 219, in forward 2025-12-04T08:52:05.7619218Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:52:05.7619437Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 62, in forward 2025-12-04T08:52:05.7619627Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-12-04T08:52:05.7619632Z 2025-12-04T08:52:05.7619735Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:05.7619936Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:05.7620008Z res = mod(**inputs) 2025-12-04T08:52:05.7620253Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 670, in forward 2025-12-04T08:52:05.7620320Z outputs = self.fnet( 2025-12-04T08:52:05.7620573Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 517, in forward 2025-12-04T08:52:05.7620649Z encoder_outputs = self.encoder( 2025-12-04T08:52:05.7621038Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 279, in forward 2025-12-04T08:52:05.7621129Z layer_outputs = layer_module(hidden_states) 2025-12-04T08:52:05.7621352Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:05.7621440Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:05.7621693Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 251, in forward 2025-12-04T08:52:05.7621786Z layer_output = apply_chunking_to_forward( 2025-12-04T08:52:05.7622055Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:52:05.7622133Z return forward_fn(*input_tensors) 2025-12-04T08:52:05.7622427Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-12-04T08:52:05.7622622Z layer_output = self.output(intermediate_output, fourier_output) 2025-12-04T08:52:05.7622883Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 232, in forward 2025-12-04T08:52:05.7623001Z hidden_states = self.dense(hidden_states) 2025-12-04T08:52:05.7623005Z 2025-12-04T08:52:05.7623112Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:05.7623323Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:05.7623389Z res = mod(**inputs) 2025-12-04T08:52:05.7623647Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 680, in forward 2025-12-04T08:52:05.7623751Z prediction_scores = self.cls(sequence_output) 2025-12-04T08:52:05.7624034Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 358, in forward 2025-12-04T08:52:05.7624159Z prediction_scores = self.predictions(sequence_output) 2025-12-04T08:52:05.7624411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 339, in forward 2025-12-04T08:52:05.7624506Z hidden_states = self.transform(hidden_states) 2025-12-04T08:52:05.7624764Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 320, in forward 2025-12-04T08:52:05.7624847Z hidden_states = self.dense(hidden_states) 2025-12-04T08:52:05.7624850Z 2025-12-04T08:52:05.7624953Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:05.7625162Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:05.7625225Z res = mod(**inputs) 2025-12-04T08:52:05.7625481Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 680, in forward 2025-12-04T08:52:05.7625578Z prediction_scores = self.cls(sequence_output) 2025-12-04T08:52:05.7625827Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 358, in forward 2025-12-04T08:52:05.7625947Z prediction_scores = self.predictions(sequence_output) 2025-12-04T08:52:05.7626199Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 340, in forward 2025-12-04T08:52:05.7626291Z hidden_states = self.decoder(hidden_states) 2025-12-04T08:52:05.7626295Z 2025-12-04T08:52:05.7626397Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:05.7626594Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:05.7626668Z res = mod(**inputs) 2025-12-04T08:52:05.7626916Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 685, in forward 2025-12-04T08:52:05.7627112Z masked_lm_loss = loss_fct(prediction_scores.view(-1, self.config.vocab_size), labels.view(-1)) 2025-12-04T08:52:05.7627127Z 2025-12-04T08:52:15.8409140Z Compilation time (from dynamo_timed): 14.589471743 2025-12-04T08:52:15.8544045Z pass 2025-12-04T08:52:15.8544469Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T08:52:15.8545356Z TIMING: _recursive_pre_grad_passes:0.00557 _recursive_joint_graph_passes:0.22048 _recursive_post_grad_passes:0.06342 async_compile.wait:0.79616 code_gen:9.56967 inductor_compile:10.76275 backend_compile:12.62842 gc:0.0015 entire_frame_compile:14.58947 total_wall_time:14.58947 2025-12-04T08:52:15.8546502Z STATS: call_* op count: 232 | FakeTensorMode.__torch_dispatch__:4224 | FakeTensor.__torch_dispatch__:3268 | ProxyTorchDispatchMode.__torch_dispatch__:1259 2025-12-04T08:52:15.8547012Z Dynamo produced 1 graphs covering 232 ops with 0 graph breaks (0 unique) 2025-12-04T08:52:18.1556417Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/__init__.py:65: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-12-04T08:52:18.1557519Z import pynvml # type: ignore[import] 2025-12-04T08:52:21.5613899Z 2025-12-04T08:52:22.9015851Z loading model: 0it [00:00, ?it/s] 2025-12-04T08:52:22.9016297Z loading model: 0it [00:01, ?it/s] 2025-12-04T08:52:22.9032860Z cpu eval LayoutLMForMaskedLM 2025-12-04T08:52:24.4628768Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T08:52:25.1235396Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T08:52:25.7908953Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T08:52:33.8508913Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:33.8509428Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:33.8509821Z res = mod(**inputs) 2025-12-04T08:52:33.8510253Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8510671Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8511129Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T08:52:33.8511642Z outputs = self.layoutlm( 2025-12-04T08:52:33.8512030Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8512529Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8513017Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T08:52:33.8513483Z encoder_outputs = self.encoder( 2025-12-04T08:52:33.8513878Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8514270Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8514719Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T08:52:33.8515186Z layer_outputs = layer_module( 2025-12-04T08:52:33.8515589Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:33.8515999Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:33.8516468Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 318, in forward 2025-12-04T08:52:33.8516941Z self_attention_outputs = self.attention( 2025-12-04T08:52:33.8517414Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 257, in forward 2025-12-04T08:52:33.8517867Z self_outputs = self.self( 2025-12-04T08:52:33.8518388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 183, in forward 2025-12-04T08:52:33.8518951Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-12-04T08:52:33.8519186Z 2025-12-04T08:52:33.8519318Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:33.8519728Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:33.8520105Z res = mod(**inputs) 2025-12-04T08:52:33.8520469Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8521465Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8522280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T08:52:33.8522731Z outputs = self.layoutlm( 2025-12-04T08:52:33.8523106Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8523550Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8523995Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T08:52:33.8524446Z encoder_outputs = self.encoder( 2025-12-04T08:52:33.8524822Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8525208Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8525651Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T08:52:33.8526150Z layer_outputs = layer_module( 2025-12-04T08:52:33.8526534Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:33.8526928Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:33.8527357Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 318, in forward 2025-12-04T08:52:33.8527794Z self_attention_outputs = self.attention( 2025-12-04T08:52:33.8528220Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 257, in forward 2025-12-04T08:52:33.8528639Z self_outputs = self.self( 2025-12-04T08:52:33.8529041Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 184, in forward 2025-12-04T08:52:33.8529516Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-12-04T08:52:33.8529727Z 2025-12-04T08:52:33.8529836Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:33.8530216Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:33.8530560Z res = mod(**inputs) 2025-12-04T08:52:33.8530902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8531297Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8531875Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T08:52:33.8532334Z outputs = self.layoutlm( 2025-12-04T08:52:33.8532685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8533054Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8533475Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T08:52:33.8533895Z encoder_outputs = self.encoder( 2025-12-04T08:52:33.8534267Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8534640Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8535064Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T08:52:33.8535497Z layer_outputs = layer_module( 2025-12-04T08:52:33.8535878Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:33.8536287Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:33.8536749Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 318, in forward 2025-12-04T08:52:33.8537198Z self_attention_outputs = self.attention( 2025-12-04T08:52:33.8537681Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 257, in forward 2025-12-04T08:52:33.8538106Z self_outputs = self.self( 2025-12-04T08:52:33.8538511Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 185, in forward 2025-12-04T08:52:33.8539063Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-12-04T08:52:33.8539291Z 2025-12-04T08:52:33.8539378Z cudagraph partition due to non gpu ops 2025-12-04T08:52:33.8539610Z cudagraph partition due to non gpu ops 2025-12-04T08:52:33.8539853Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:33.8540239Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:33.8540586Z res = mod(**inputs) 2025-12-04T08:52:33.8540926Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8541347Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8541775Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T08:52:33.8542193Z outputs = self.layoutlm( 2025-12-04T08:52:33.8542533Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8542910Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8543323Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T08:52:33.8543740Z encoder_outputs = self.encoder( 2025-12-04T08:52:33.8544094Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8544464Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8544876Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T08:52:33.8545294Z layer_outputs = layer_module( 2025-12-04T08:52:33.8545655Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:33.8546033Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:33.8546456Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 318, in forward 2025-12-04T08:52:33.8546877Z self_attention_outputs = self.attention( 2025-12-04T08:52:33.8547304Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 264, in forward 2025-12-04T08:52:33.8547777Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T08:52:33.8548252Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 217, in forward 2025-12-04T08:52:33.8548670Z hidden_states = self.dense(hidden_states) 2025-12-04T08:52:33.8548823Z 2025-12-04T08:52:33.8548935Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:33.8549308Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:33.8549644Z res = mod(**inputs) 2025-12-04T08:52:33.8549983Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8550354Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8550774Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T08:52:33.8551189Z outputs = self.layoutlm( 2025-12-04T08:52:33.8551538Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8551933Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8552380Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T08:52:33.8552800Z encoder_outputs = self.encoder( 2025-12-04T08:52:33.8553154Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8553550Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8553961Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T08:52:33.8554381Z layer_outputs = layer_module( 2025-12-04T08:52:33.8554741Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:33.8555140Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:33.8555598Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 328, in forward 2025-12-04T08:52:33.8556079Z layer_output = apply_chunking_to_forward( 2025-12-04T08:52:33.8556533Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:52:33.8556969Z return forward_fn(*input_tensors) 2025-12-04T08:52:33.8557458Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 336, in feed_forward_chunk 2025-12-04T08:52:33.8557989Z intermediate_output = self.intermediate(attention_output) 2025-12-04T08:52:33.8558672Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 280, in forward 2025-12-04T08:52:33.8559133Z hidden_states = self.dense(hidden_states) 2025-12-04T08:52:33.8559296Z 2025-12-04T08:52:33.8559412Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:33.8559811Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:33.8560183Z res = mod(**inputs) 2025-12-04T08:52:33.8560544Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8560933Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8561375Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T08:52:33.8561817Z outputs = self.layoutlm( 2025-12-04T08:52:33.8562184Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8562572Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8563009Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T08:52:33.8563443Z encoder_outputs = self.encoder( 2025-12-04T08:52:33.8563829Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8564215Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8564641Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T08:52:33.8565079Z layer_outputs = layer_module( 2025-12-04T08:52:33.8565459Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:33.8565859Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:33.8566294Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 328, in forward 2025-12-04T08:52:33.8566749Z layer_output = apply_chunking_to_forward( 2025-12-04T08:52:33.8567187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:52:33.8567616Z return forward_fn(*input_tensors) 2025-12-04T08:52:33.8568124Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 336, in feed_forward_chunk 2025-12-04T08:52:33.8568661Z intermediate_output = self.intermediate(attention_output) 2025-12-04T08:52:33.8569150Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 281, in forward 2025-12-04T08:52:33.8569611Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:52:33.8570013Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T08:52:33.8570376Z return self.act(input) 2025-12-04T08:52:33.8570492Z 2025-12-04T08:52:33.8570610Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:33.8570979Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:33.8571337Z res = mod(**inputs) 2025-12-04T08:52:33.8571681Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8572039Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8572457Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T08:52:33.8572874Z outputs = self.layoutlm( 2025-12-04T08:52:33.8573222Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8573581Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8573995Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T08:52:33.8574416Z encoder_outputs = self.encoder( 2025-12-04T08:52:33.8574769Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8575135Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8575553Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T08:52:33.8575966Z layer_outputs = layer_module( 2025-12-04T08:52:33.8576323Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:33.8576697Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:33.8577123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 328, in forward 2025-12-04T08:52:33.8577551Z layer_output = apply_chunking_to_forward( 2025-12-04T08:52:33.8577959Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:52:33.8578367Z return forward_fn(*input_tensors) 2025-12-04T08:52:33.8578818Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 337, in feed_forward_chunk 2025-12-04T08:52:33.8579317Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T08:52:33.8579795Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 294, in forward 2025-12-04T08:52:33.8580222Z hidden_states = self.dense(hidden_states) 2025-12-04T08:52:33.8580364Z 2025-12-04T08:52:33.8580481Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:33.8580847Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:33.8581182Z res = mod(**inputs) 2025-12-04T08:52:33.8581520Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8581885Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8582329Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T08:52:33.8582746Z outputs = self.layoutlm( 2025-12-04T08:52:33.8583096Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8583477Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8583897Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T08:52:33.8584328Z encoder_outputs = self.encoder( 2025-12-04T08:52:33.8584698Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8585066Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8585501Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T08:52:33.8585969Z layer_outputs = layer_module( 2025-12-04T08:52:33.8586329Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:33.8586709Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:33.8587164Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 318, in forward 2025-12-04T08:52:33.8587631Z self_attention_outputs = self.attention( 2025-12-04T08:52:33.8588080Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 257, in forward 2025-12-04T08:52:33.8588531Z self_outputs = self.self( 2025-12-04T08:52:33.8588966Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 183, in forward 2025-12-04T08:52:33.8589502Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-12-04T08:52:33.8589725Z 2025-12-04T08:52:33.8589840Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:33.8590239Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:33.8590598Z res = mod(**inputs) 2025-12-04T08:52:33.8590950Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8591339Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8591789Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T08:52:33.8592241Z outputs = self.layoutlm( 2025-12-04T08:52:33.8592608Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8592993Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8593429Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T08:52:33.8593874Z encoder_outputs = self.encoder( 2025-12-04T08:52:33.8594246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8594632Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8595070Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T08:52:33.8595511Z layer_outputs = layer_module( 2025-12-04T08:52:33.8595894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:33.8596292Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:33.8596745Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 318, in forward 2025-12-04T08:52:33.8597187Z self_attention_outputs = self.attention( 2025-12-04T08:52:33.8597683Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 257, in forward 2025-12-04T08:52:33.8598130Z self_outputs = self.self( 2025-12-04T08:52:33.8598716Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 184, in forward 2025-12-04T08:52:33.8599293Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-12-04T08:52:33.8599513Z 2025-12-04T08:52:33.8599629Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:33.8600033Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:33.8600362Z res = mod(**inputs) 2025-12-04T08:52:33.8600700Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8601072Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8601526Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T08:52:33.8601960Z outputs = self.layoutlm( 2025-12-04T08:52:33.8602332Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8602727Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8603160Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T08:52:33.8603608Z encoder_outputs = self.encoder( 2025-12-04T08:52:33.8603990Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8604379Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8604813Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T08:52:33.8605260Z layer_outputs = layer_module( 2025-12-04T08:52:33.8605649Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:33.8606050Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:33.8606501Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 318, in forward 2025-12-04T08:52:33.8606956Z self_attention_outputs = self.attention( 2025-12-04T08:52:33.8607408Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 257, in forward 2025-12-04T08:52:33.8607840Z self_outputs = self.self( 2025-12-04T08:52:33.8608276Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 185, in forward 2025-12-04T08:52:33.8608798Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-12-04T08:52:33.8609023Z 2025-12-04T08:52:33.8609122Z cudagraph partition due to non gpu ops 2025-12-04T08:52:33.8609360Z cudagraph partition due to non gpu ops 2025-12-04T08:52:33.8609625Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:33.8610031Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:33.8610388Z res = mod(**inputs) 2025-12-04T08:52:33.8610755Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8611150Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8611593Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T08:52:33.8612039Z outputs = self.layoutlm( 2025-12-04T08:52:33.8612409Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8612804Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8613276Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T08:52:33.8613722Z encoder_outputs = self.encoder( 2025-12-04T08:52:33.8614102Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8614475Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8614877Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T08:52:33.8615272Z layer_outputs = layer_module( 2025-12-04T08:52:33.8615615Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:33.8615977Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:33.8616380Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 318, in forward 2025-12-04T08:52:33.8616806Z self_attention_outputs = self.attention( 2025-12-04T08:52:33.8617218Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 264, in forward 2025-12-04T08:52:33.8617679Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T08:52:33.8618123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 217, in forward 2025-12-04T08:52:33.8618525Z hidden_states = self.dense(hidden_states) 2025-12-04T08:52:33.8618661Z 2025-12-04T08:52:33.8618773Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:33.8619130Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:33.8619458Z res = mod(**inputs) 2025-12-04T08:52:33.8619786Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8620135Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8620541Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T08:52:33.8621129Z outputs = self.layoutlm( 2025-12-04T08:52:33.8621480Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8621834Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8622240Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T08:52:33.8622769Z encoder_outputs = self.encoder( 2025-12-04T08:52:33.8623123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8623469Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8623865Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T08:52:33.8624274Z layer_outputs = layer_module( 2025-12-04T08:52:33.8624622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:33.8624993Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:33.8625406Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 328, in forward 2025-12-04T08:52:33.8625903Z layer_output = apply_chunking_to_forward( 2025-12-04T08:52:33.8626294Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:52:33.8626697Z return forward_fn(*input_tensors) 2025-12-04T08:52:33.8627123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 336, in feed_forward_chunk 2025-12-04T08:52:33.8627597Z intermediate_output = self.intermediate(attention_output) 2025-12-04T08:52:33.8628133Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 280, in forward 2025-12-04T08:52:33.8628554Z hidden_states = self.dense(hidden_states) 2025-12-04T08:52:33.8628724Z 2025-12-04T08:52:33.8628834Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:33.8629182Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:33.8629512Z res = mod(**inputs) 2025-12-04T08:52:33.8629841Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8630202Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8630600Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T08:52:33.8631035Z outputs = self.layoutlm( 2025-12-04T08:52:33.8631384Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8631728Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8632135Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T08:52:33.8632547Z encoder_outputs = self.encoder( 2025-12-04T08:52:33.8632897Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8633251Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8633656Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T08:52:33.8634059Z layer_outputs = layer_module( 2025-12-04T08:52:33.8634409Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:33.8634777Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:33.8635191Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 328, in forward 2025-12-04T08:52:33.8635619Z layer_output = apply_chunking_to_forward( 2025-12-04T08:52:33.8636042Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:52:33.8636439Z return forward_fn(*input_tensors) 2025-12-04T08:52:33.8636877Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 336, in feed_forward_chunk 2025-12-04T08:52:33.8637368Z intermediate_output = self.intermediate(attention_output) 2025-12-04T08:52:33.8637828Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 281, in forward 2025-12-04T08:52:33.8638340Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:52:33.8638752Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T08:52:33.8639136Z return self.act(input) 2025-12-04T08:52:33.8639261Z 2025-12-04T08:52:33.8639379Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:33.8639780Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:33.8640132Z res = mod(**inputs) 2025-12-04T08:52:33.8640473Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8640836Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8641246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T08:52:33.8641652Z outputs = self.layoutlm( 2025-12-04T08:52:33.8641988Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8642380Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8642788Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T08:52:33.8643208Z encoder_outputs = self.encoder( 2025-12-04T08:52:33.8643561Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8643924Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8644327Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T08:52:33.8644719Z layer_outputs = layer_module( 2025-12-04T08:52:33.8645072Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:33.8645460Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:33.8645871Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 328, in forward 2025-12-04T08:52:33.8646298Z layer_output = apply_chunking_to_forward( 2025-12-04T08:52:33.8646702Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:52:33.8647101Z return forward_fn(*input_tensors) 2025-12-04T08:52:33.8647531Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 337, in feed_forward_chunk 2025-12-04T08:52:33.8648030Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T08:52:33.8648502Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 294, in forward 2025-12-04T08:52:33.8648919Z hidden_states = self.dense(hidden_states) 2025-12-04T08:52:33.8649061Z 2025-12-04T08:52:33.8649166Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:33.8649534Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:33.8649868Z res = mod(**inputs) 2025-12-04T08:52:33.8650193Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8650550Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8650957Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T08:52:33.8651382Z outputs = self.layoutlm( 2025-12-04T08:52:33.8651715Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8652071Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8652475Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T08:52:33.8652891Z encoder_outputs = self.encoder( 2025-12-04T08:52:33.8653237Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8653599Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8653994Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T08:52:33.8654383Z layer_outputs = layer_module( 2025-12-04T08:52:33.8654724Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:33.8655080Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:33.8655481Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 318, in forward 2025-12-04T08:52:33.8655880Z self_attention_outputs = self.attention( 2025-12-04T08:52:33.8656327Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 257, in forward 2025-12-04T08:52:33.8656725Z self_outputs = self.self( 2025-12-04T08:52:33.8657107Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 183, in forward 2025-12-04T08:52:33.8657590Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-12-04T08:52:33.8657798Z 2025-12-04T08:52:33.8657902Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:33.8658258Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:33.8658569Z res = mod(**inputs) 2025-12-04T08:52:33.8658897Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8659254Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8659689Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T08:52:33.8660077Z outputs = self.layoutlm( 2025-12-04T08:52:33.8660407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8660757Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8661144Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T08:52:33.8661545Z encoder_outputs = self.encoder( 2025-12-04T08:52:33.8661885Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8662229Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8662612Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T08:52:33.8663008Z layer_outputs = layer_module( 2025-12-04T08:52:33.8663352Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:33.8663705Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:33.8664097Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 318, in forward 2025-12-04T08:52:33.8664502Z self_attention_outputs = self.attention( 2025-12-04T08:52:33.8664901Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 257, in forward 2025-12-04T08:52:33.8665290Z self_outputs = self.self( 2025-12-04T08:52:33.8665669Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 184, in forward 2025-12-04T08:52:33.8666122Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-12-04T08:52:33.8666311Z 2025-12-04T08:52:33.8666421Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:33.8666773Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:33.8667096Z res = mod(**inputs) 2025-12-04T08:52:33.8667425Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8667781Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8668187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T08:52:33.8668586Z outputs = self.layoutlm( 2025-12-04T08:52:33.8668920Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8669268Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8669675Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T08:52:33.8670143Z encoder_outputs = self.encoder( 2025-12-04T08:52:33.8670495Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8670842Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8671262Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T08:52:33.8671666Z layer_outputs = layer_module( 2025-12-04T08:52:33.8672009Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:33.8672376Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:33.8672783Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 318, in forward 2025-12-04T08:52:33.8673199Z self_attention_outputs = self.attention( 2025-12-04T08:52:33.8673627Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 257, in forward 2025-12-04T08:52:33.8674035Z self_outputs = self.self( 2025-12-04T08:52:33.8674428Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 185, in forward 2025-12-04T08:52:33.8674909Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-12-04T08:52:33.8675113Z 2025-12-04T08:52:33.8675197Z cudagraph partition due to non gpu ops 2025-12-04T08:52:33.8675423Z cudagraph partition due to non gpu ops 2025-12-04T08:52:33.8675670Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:33.8676039Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:33.8676379Z res = mod(**inputs) 2025-12-04T08:52:33.8676721Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8677094Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8677508Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T08:52:33.8677923Z outputs = self.layoutlm( 2025-12-04T08:52:33.8678344Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8678755Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8679220Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T08:52:33.8679668Z encoder_outputs = self.encoder( 2025-12-04T08:52:33.8680060Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8680409Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8680826Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T08:52:33.8681235Z layer_outputs = layer_module( 2025-12-04T08:52:33.8681583Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:33.8681955Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:33.8682371Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 318, in forward 2025-12-04T08:52:33.8682794Z self_attention_outputs = self.attention( 2025-12-04T08:52:33.8683206Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 264, in forward 2025-12-04T08:52:33.8683686Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T08:52:33.8684144Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 217, in forward 2025-12-04T08:52:33.8684566Z hidden_states = self.dense(hidden_states) 2025-12-04T08:52:33.8684739Z 2025-12-04T08:52:33.8684846Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:33.8685216Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:33.8685569Z res = mod(**inputs) 2025-12-04T08:52:33.8685890Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8686249Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8686657Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T08:52:33.8687122Z outputs = self.layoutlm( 2025-12-04T08:52:33.8687443Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8687814Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8688219Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T08:52:33.8688618Z encoder_outputs = self.encoder( 2025-12-04T08:52:33.8688967Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8689325Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8689731Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T08:52:33.8690137Z layer_outputs = layer_module( 2025-12-04T08:52:33.8690482Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:33.8690839Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:33.8691240Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 328, in forward 2025-12-04T08:52:33.8691657Z layer_output = apply_chunking_to_forward( 2025-12-04T08:52:33.8692067Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:52:33.8692467Z return forward_fn(*input_tensors) 2025-12-04T08:52:33.8692897Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 336, in feed_forward_chunk 2025-12-04T08:52:33.8693384Z intermediate_output = self.intermediate(attention_output) 2025-12-04T08:52:33.8693831Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 280, in forward 2025-12-04T08:52:33.8694243Z hidden_states = self.dense(hidden_states) 2025-12-04T08:52:33.8694383Z 2025-12-04T08:52:33.8694488Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:33.8694860Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:33.8695204Z res = mod(**inputs) 2025-12-04T08:52:33.8695545Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8695912Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8696337Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T08:52:33.8696742Z outputs = self.layoutlm( 2025-12-04T08:52:33.8697084Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8697442Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8697848Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T08:52:33.8698258Z encoder_outputs = self.encoder( 2025-12-04T08:52:33.8698638Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8699006Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8699411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T08:52:33.8699820Z layer_outputs = layer_module( 2025-12-04T08:52:33.8700167Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:33.8700531Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:33.8700937Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 328, in forward 2025-12-04T08:52:33.8701343Z layer_output = apply_chunking_to_forward( 2025-12-04T08:52:33.8701750Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:52:33.8702187Z return forward_fn(*input_tensors) 2025-12-04T08:52:33.8702642Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 336, in feed_forward_chunk 2025-12-04T08:52:33.8703129Z intermediate_output = self.intermediate(attention_output) 2025-12-04T08:52:33.8703596Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 281, in forward 2025-12-04T08:52:33.8704054Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:52:33.8704446Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T08:52:33.8704805Z return self.act(input) 2025-12-04T08:52:33.8704928Z 2025-12-04T08:52:33.8705035Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:33.8705410Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:33.8705747Z res = mod(**inputs) 2025-12-04T08:52:33.8706107Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8706498Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8706916Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T08:52:33.8707326Z outputs = self.layoutlm( 2025-12-04T08:52:33.8707681Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8708069Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8708495Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T08:52:33.8708914Z encoder_outputs = self.encoder( 2025-12-04T08:52:33.8709274Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8709645Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8710050Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T08:52:33.8710474Z layer_outputs = layer_module( 2025-12-04T08:52:33.8710840Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:33.8711209Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:33.8711630Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 328, in forward 2025-12-04T08:52:33.8712061Z layer_output = apply_chunking_to_forward( 2025-12-04T08:52:33.8712482Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:52:33.8712890Z return forward_fn(*input_tensors) 2025-12-04T08:52:33.8713367Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 337, in feed_forward_chunk 2025-12-04T08:52:33.8713878Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T08:52:33.8714357Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 294, in forward 2025-12-04T08:52:33.8714798Z hidden_states = self.dense(hidden_states) 2025-12-04T08:52:33.8714949Z 2025-12-04T08:52:33.8715060Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:33.8715436Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:33.8715767Z res = mod(**inputs) 2025-12-04T08:52:33.8716105Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8716494Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8716916Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T08:52:33.8717326Z outputs = self.layoutlm( 2025-12-04T08:52:33.8717681Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8718073Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8718603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T08:52:33.8719069Z encoder_outputs = self.encoder( 2025-12-04T08:52:33.8719464Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8719866Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8720305Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T08:52:33.8720895Z layer_outputs = layer_module( 2025-12-04T08:52:33.8721302Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:33.8721719Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:33.8722181Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 318, in forward 2025-12-04T08:52:33.8722655Z self_attention_outputs = self.attention( 2025-12-04T08:52:33.8723125Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 257, in forward 2025-12-04T08:52:33.8723583Z self_outputs = self.self( 2025-12-04T08:52:33.8724033Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 183, in forward 2025-12-04T08:52:33.8724568Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-12-04T08:52:33.8724807Z 2025-12-04T08:52:33.8724937Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:33.8725339Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:33.8725706Z res = mod(**inputs) 2025-12-04T08:52:33.8726073Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8726471Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8726926Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T08:52:33.8727371Z outputs = self.layoutlm( 2025-12-04T08:52:33.8727751Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8728101Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8728509Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T08:52:33.8729054Z encoder_outputs = self.encoder( 2025-12-04T08:52:33.8729415Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8729769Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8730220Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T08:52:33.8730638Z layer_outputs = layer_module( 2025-12-04T08:52:33.8730990Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:33.8731366Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:33.8731790Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 318, in forward 2025-12-04T08:52:33.8732231Z self_attention_outputs = self.attention( 2025-12-04T08:52:33.8732639Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 257, in forward 2025-12-04T08:52:33.8733048Z self_outputs = self.self( 2025-12-04T08:52:33.8733441Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 184, in forward 2025-12-04T08:52:33.8733915Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-12-04T08:52:33.8734106Z 2025-12-04T08:52:33.8734212Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:33.8734581Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:33.8734911Z res = mod(**inputs) 2025-12-04T08:52:33.8735232Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8735593Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8736015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T08:52:33.8736409Z outputs = self.layoutlm( 2025-12-04T08:52:33.8736735Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8737083Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8737528Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T08:52:33.8737933Z encoder_outputs = self.encoder( 2025-12-04T08:52:33.8738277Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8738635Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8739039Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T08:52:33.8739438Z layer_outputs = layer_module( 2025-12-04T08:52:33.8739791Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:33.8740161Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:33.8740574Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 318, in forward 2025-12-04T08:52:33.8740983Z self_attention_outputs = self.attention( 2025-12-04T08:52:33.8741397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 257, in forward 2025-12-04T08:52:33.8741797Z self_outputs = self.self( 2025-12-04T08:52:33.8742178Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 185, in forward 2025-12-04T08:52:33.8742657Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-12-04T08:52:33.8742867Z 2025-12-04T08:52:33.8742950Z cudagraph partition due to non gpu ops 2025-12-04T08:52:33.8743210Z cudagraph partition due to non gpu ops 2025-12-04T08:52:33.8743449Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:33.8743820Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:33.8744179Z res = mod(**inputs) 2025-12-04T08:52:33.8744518Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8744899Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8745327Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T08:52:33.8745761Z outputs = self.layoutlm( 2025-12-04T08:52:33.8746122Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8746531Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8746981Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T08:52:33.8747421Z encoder_outputs = self.encoder( 2025-12-04T08:52:33.8747788Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8748155Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8748568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T08:52:33.8748974Z layer_outputs = layer_module( 2025-12-04T08:52:33.8749335Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:33.8749709Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:33.8750131Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 318, in forward 2025-12-04T08:52:33.8750636Z self_attention_outputs = self.attention( 2025-12-04T08:52:33.8751062Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 264, in forward 2025-12-04T08:52:33.8751538Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T08:52:33.8752011Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 217, in forward 2025-12-04T08:52:33.8752435Z hidden_states = self.dense(hidden_states) 2025-12-04T08:52:33.8752588Z 2025-12-04T08:52:33.8752695Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:33.8753070Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:33.8753402Z res = mod(**inputs) 2025-12-04T08:52:33.8753738Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8754110Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8754524Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T08:52:33.8754934Z outputs = self.layoutlm( 2025-12-04T08:52:33.8755284Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8755666Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8756105Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T08:52:33.8756558Z encoder_outputs = self.encoder( 2025-12-04T08:52:33.8756947Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8757340Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8757814Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T08:52:33.8758330Z layer_outputs = layer_module( 2025-12-04T08:52:33.8758722Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:33.8759164Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:33.8759629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 328, in forward 2025-12-04T08:52:33.8760102Z layer_output = apply_chunking_to_forward( 2025-12-04T08:52:33.8760559Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:52:33.8760962Z return forward_fn(*input_tensors) 2025-12-04T08:52:33.8761405Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 336, in feed_forward_chunk 2025-12-04T08:52:33.8761961Z intermediate_output = self.intermediate(attention_output) 2025-12-04T08:52:33.8762487Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 280, in forward 2025-12-04T08:52:33.8762956Z hidden_states = self.dense(hidden_states) 2025-12-04T08:52:33.8763124Z 2025-12-04T08:52:33.8763242Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:33.8763650Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:33.8764014Z res = mod(**inputs) 2025-12-04T08:52:33.8764383Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8764786Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8765247Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T08:52:33.8765699Z outputs = self.layoutlm( 2025-12-04T08:52:33.8766081Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8766481Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8766939Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T08:52:33.8767394Z encoder_outputs = self.encoder( 2025-12-04T08:52:33.8767786Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8768184Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8768605Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T08:52:33.8769015Z layer_outputs = layer_module( 2025-12-04T08:52:33.8769368Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:33.8769741Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:33.8770144Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 328, in forward 2025-12-04T08:52:33.8770575Z layer_output = apply_chunking_to_forward( 2025-12-04T08:52:33.8771001Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:52:33.8771411Z return forward_fn(*input_tensors) 2025-12-04T08:52:33.8771852Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 336, in feed_forward_chunk 2025-12-04T08:52:33.8772346Z intermediate_output = self.intermediate(attention_output) 2025-12-04T08:52:33.8772811Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 281, in forward 2025-12-04T08:52:33.8773262Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:52:33.8773699Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T08:52:33.8774062Z return self.act(input) 2025-12-04T08:52:33.8774177Z 2025-12-04T08:52:33.8774310Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:33.8774687Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:33.8775027Z res = mod(**inputs) 2025-12-04T08:52:33.8775369Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8775731Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8776149Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T08:52:33.8776582Z outputs = self.layoutlm( 2025-12-04T08:52:33.8776931Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8777293Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8777697Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T08:52:33.8778107Z encoder_outputs = self.encoder( 2025-12-04T08:52:33.8778457Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8778807Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8779213Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T08:52:33.8779619Z layer_outputs = layer_module( 2025-12-04T08:52:33.8779966Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:33.8780338Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:33.8780752Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 328, in forward 2025-12-04T08:52:33.8781171Z layer_output = apply_chunking_to_forward( 2025-12-04T08:52:33.8781571Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:52:33.8781969Z return forward_fn(*input_tensors) 2025-12-04T08:52:33.8782402Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 337, in feed_forward_chunk 2025-12-04T08:52:33.8782899Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T08:52:33.8783359Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 294, in forward 2025-12-04T08:52:33.8783775Z hidden_states = self.dense(hidden_states) 2025-12-04T08:52:33.8783913Z 2025-12-04T08:52:33.8784030Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:33.8784390Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:33.8784721Z res = mod(**inputs) 2025-12-04T08:52:33.8785049Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8785408Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8785804Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T08:52:33.8786214Z outputs = self.layoutlm( 2025-12-04T08:52:33.8786562Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8786921Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8787345Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T08:52:33.8787841Z encoder_outputs = self.encoder( 2025-12-04T08:52:33.8788226Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8788634Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8789061Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T08:52:33.8789488Z layer_outputs = layer_module( 2025-12-04T08:52:33.8789868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:33.8790271Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:33.8790731Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 318, in forward 2025-12-04T08:52:33.8791203Z self_attention_outputs = self.attention( 2025-12-04T08:52:33.8791654Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 257, in forward 2025-12-04T08:52:33.8792105Z self_outputs = self.self( 2025-12-04T08:52:33.8792534Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 183, in forward 2025-12-04T08:52:33.8793062Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-12-04T08:52:33.8793281Z 2025-12-04T08:52:33.8793395Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:33.8793790Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:33.8794148Z res = mod(**inputs) 2025-12-04T08:52:33.8794501Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8794895Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8795357Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T08:52:33.8795816Z outputs = self.layoutlm( 2025-12-04T08:52:33.8796198Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8796606Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8797068Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T08:52:33.8797533Z encoder_outputs = self.encoder( 2025-12-04T08:52:33.8797928Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8798407Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8798865Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T08:52:33.8799323Z layer_outputs = layer_module( 2025-12-04T08:52:33.8799727Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:33.8800128Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:33.8800582Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 318, in forward 2025-12-04T08:52:33.8801076Z self_attention_outputs = self.attention( 2025-12-04T08:52:33.8801529Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 257, in forward 2025-12-04T08:52:33.8801974Z self_outputs = self.self( 2025-12-04T08:52:33.8802400Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 184, in forward 2025-12-04T08:52:33.8802909Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-12-04T08:52:33.8803124Z 2025-12-04T08:52:33.8803278Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:33.8803680Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:33.8804027Z res = mod(**inputs) 2025-12-04T08:52:33.8804381Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8804792Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8805247Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T08:52:33.8805657Z outputs = self.layoutlm( 2025-12-04T08:52:33.8806015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8806401Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8806841Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T08:52:33.8807308Z encoder_outputs = self.encoder( 2025-12-04T08:52:33.8807687Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8808067Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8808496Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T08:52:33.8808944Z layer_outputs = layer_module( 2025-12-04T08:52:33.8809327Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:33.8809724Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:33.8810151Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 318, in forward 2025-12-04T08:52:33.8810574Z self_attention_outputs = self.attention( 2025-12-04T08:52:33.8810997Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 257, in forward 2025-12-04T08:52:33.8811403Z self_outputs = self.self( 2025-12-04T08:52:33.8811827Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 185, in forward 2025-12-04T08:52:33.8812347Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-12-04T08:52:33.8812554Z 2025-12-04T08:52:33.8812645Z cudagraph partition due to non gpu ops 2025-12-04T08:52:33.8812859Z cudagraph partition due to non gpu ops 2025-12-04T08:52:33.8813103Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:33.8813472Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:33.8813799Z res = mod(**inputs) 2025-12-04T08:52:33.8814134Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8814527Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8814968Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T08:52:33.8815394Z outputs = self.layoutlm( 2025-12-04T08:52:33.8815763Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8816146Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8816574Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T08:52:33.8817013Z encoder_outputs = self.encoder( 2025-12-04T08:52:33.8817390Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8817774Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8818256Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T08:52:33.8818703Z layer_outputs = layer_module( 2025-12-04T08:52:33.8819085Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:33.8819502Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:33.8819936Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 318, in forward 2025-12-04T08:52:33.8820387Z self_attention_outputs = self.attention( 2025-12-04T08:52:33.8820951Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 264, in forward 2025-12-04T08:52:33.8821452Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T08:52:33.8821952Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 217, in forward 2025-12-04T08:52:33.8822458Z hidden_states = self.dense(hidden_states) 2025-12-04T08:52:33.8822610Z 2025-12-04T08:52:33.8822730Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:33.8823117Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:33.8823472Z res = mod(**inputs) 2025-12-04T08:52:33.8823834Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8824219Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8824660Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T08:52:33.8825101Z outputs = self.layoutlm( 2025-12-04T08:52:33.8825469Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8825849Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8826299Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T08:52:33.8826741Z encoder_outputs = self.encoder( 2025-12-04T08:52:33.8827117Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8827496Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8827942Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T08:52:33.8828392Z layer_outputs = layer_module( 2025-12-04T08:52:33.8828765Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:33.8829162Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:33.8829604Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 328, in forward 2025-12-04T08:52:33.8830061Z layer_output = apply_chunking_to_forward( 2025-12-04T08:52:33.8830493Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:52:33.8830927Z return forward_fn(*input_tensors) 2025-12-04T08:52:33.8831409Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 336, in feed_forward_chunk 2025-12-04T08:52:33.8831907Z intermediate_output = self.intermediate(attention_output) 2025-12-04T08:52:33.8832362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 280, in forward 2025-12-04T08:52:33.8832785Z hidden_states = self.dense(hidden_states) 2025-12-04T08:52:33.8832927Z 2025-12-04T08:52:33.8833043Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:33.8833684Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:33.8834091Z res = mod(**inputs) 2025-12-04T08:52:33.8834454Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8834843Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8835319Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T08:52:33.8835765Z outputs = self.layoutlm( 2025-12-04T08:52:33.8836135Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8836527Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8836971Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T08:52:33.8837476Z encoder_outputs = self.encoder( 2025-12-04T08:52:33.8837882Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8838335Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8838795Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T08:52:33.8839252Z layer_outputs = layer_module( 2025-12-04T08:52:33.8839648Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:33.8840060Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:33.8840510Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 328, in forward 2025-12-04T08:52:33.8840966Z layer_output = apply_chunking_to_forward( 2025-12-04T08:52:33.8841400Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:52:33.8841835Z return forward_fn(*input_tensors) 2025-12-04T08:52:33.8842288Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 336, in feed_forward_chunk 2025-12-04T08:52:33.8842769Z intermediate_output = self.intermediate(attention_output) 2025-12-04T08:52:33.8843216Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 281, in forward 2025-12-04T08:52:33.8843665Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:52:33.8844065Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T08:52:33.8844421Z return self.act(input) 2025-12-04T08:52:33.8844537Z 2025-12-04T08:52:33.8844646Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:33.8845025Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:33.8845365Z res = mod(**inputs) 2025-12-04T08:52:33.8845715Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8846116Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8846537Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T08:52:33.8846975Z outputs = self.layoutlm( 2025-12-04T08:52:33.8847344Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8847734Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8848175Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T08:52:33.8848596Z encoder_outputs = self.encoder( 2025-12-04T08:52:33.8848953Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8849385Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8849830Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T08:52:33.8850293Z layer_outputs = layer_module( 2025-12-04T08:52:33.8850676Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:33.8851092Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:33.8851547Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 328, in forward 2025-12-04T08:52:33.8852005Z layer_output = apply_chunking_to_forward( 2025-12-04T08:52:33.8852444Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:52:33.8852893Z return forward_fn(*input_tensors) 2025-12-04T08:52:33.8853359Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 337, in feed_forward_chunk 2025-12-04T08:52:33.8853894Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T08:52:33.8854404Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 294, in forward 2025-12-04T08:52:33.8854865Z hidden_states = self.dense(hidden_states) 2025-12-04T08:52:33.8855015Z 2025-12-04T08:52:33.8855129Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:33.8855522Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:33.8855876Z res = mod(**inputs) 2025-12-04T08:52:33.8856229Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8856611Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8857052Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T08:52:33.8857482Z outputs = self.layoutlm( 2025-12-04T08:52:33.8857822Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8858190Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8858600Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T08:52:33.8859016Z encoder_outputs = self.encoder( 2025-12-04T08:52:33.8859368Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8859728Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8860139Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T08:52:33.8860555Z layer_outputs = layer_module( 2025-12-04T08:52:33.8860908Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:33.8861291Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:33.8861712Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 318, in forward 2025-12-04T08:52:33.8862129Z self_attention_outputs = self.attention( 2025-12-04T08:52:33.8862551Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 257, in forward 2025-12-04T08:52:33.8862962Z self_outputs = self.self( 2025-12-04T08:52:33.8863360Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 183, in forward 2025-12-04T08:52:33.8863841Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-12-04T08:52:33.8864058Z 2025-12-04T08:52:33.8864202Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:33.8864582Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:33.8864917Z res = mod(**inputs) 2025-12-04T08:52:33.8865268Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8865637Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8866051Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T08:52:33.8866461Z outputs = self.layoutlm( 2025-12-04T08:52:33.8866810Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8867175Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8867616Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T08:52:33.8868032Z encoder_outputs = self.encoder( 2025-12-04T08:52:33.8868392Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8868762Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8869172Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T08:52:33.8869591Z layer_outputs = layer_module( 2025-12-04T08:52:33.8869953Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:33.8870333Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:33.8870742Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 318, in forward 2025-12-04T08:52:33.8871162Z self_attention_outputs = self.attention( 2025-12-04T08:52:33.8871578Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 257, in forward 2025-12-04T08:52:33.8871984Z self_outputs = self.self( 2025-12-04T08:52:33.8872369Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 184, in forward 2025-12-04T08:52:33.8872839Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-12-04T08:52:33.8873034Z 2025-12-04T08:52:33.8873150Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:33.8873511Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:33.8873840Z res = mod(**inputs) 2025-12-04T08:52:33.8874174Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8874536Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8874943Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T08:52:33.8875352Z outputs = self.layoutlm( 2025-12-04T08:52:33.8875694Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8876043Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8876450Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T08:52:33.8876857Z encoder_outputs = self.encoder( 2025-12-04T08:52:33.8877202Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8877550Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8877959Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T08:52:33.8878459Z layer_outputs = layer_module( 2025-12-04T08:52:33.8878886Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:33.8879293Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:33.8879781Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 318, in forward 2025-12-04T08:52:33.8880249Z self_attention_outputs = self.attention( 2025-12-04T08:52:33.8880705Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 257, in forward 2025-12-04T08:52:33.8881200Z self_outputs = self.self( 2025-12-04T08:52:33.8881648Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 185, in forward 2025-12-04T08:52:33.8882150Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-12-04T08:52:33.8882377Z 2025-12-04T08:52:33.8882465Z cudagraph partition due to non gpu ops 2025-12-04T08:52:33.8882690Z cudagraph partition due to non gpu ops 2025-12-04T08:52:33.8882937Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:33.8883304Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:33.8883648Z res = mod(**inputs) 2025-12-04T08:52:33.8883981Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8884342Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8884742Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T08:52:33.8885146Z outputs = self.layoutlm( 2025-12-04T08:52:33.8885483Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8885838Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8886250Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T08:52:33.8886664Z encoder_outputs = self.encoder( 2025-12-04T08:52:33.8887024Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8887382Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8887798Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T08:52:33.8888210Z layer_outputs = layer_module( 2025-12-04T08:52:33.8888569Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:33.8888937Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:33.8889359Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 318, in forward 2025-12-04T08:52:33.8889790Z self_attention_outputs = self.attention( 2025-12-04T08:52:33.8890216Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 264, in forward 2025-12-04T08:52:33.8890681Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T08:52:33.8891151Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 217, in forward 2025-12-04T08:52:33.8891582Z hidden_states = self.dense(hidden_states) 2025-12-04T08:52:33.8891725Z 2025-12-04T08:52:33.8891832Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:33.8892207Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:33.8892546Z res = mod(**inputs) 2025-12-04T08:52:33.8892889Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8893291Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8893711Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T08:52:33.8894142Z outputs = self.layoutlm( 2025-12-04T08:52:33.8894485Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8894850Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8895268Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T08:52:33.8895689Z encoder_outputs = self.encoder( 2025-12-04T08:52:33.8896040Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8896426Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8896840Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T08:52:33.8897246Z layer_outputs = layer_module( 2025-12-04T08:52:33.8897605Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:33.8897979Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:33.8898398Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 328, in forward 2025-12-04T08:52:33.8898817Z layer_output = apply_chunking_to_forward( 2025-12-04T08:52:33.8899235Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:52:33.8899661Z return forward_fn(*input_tensors) 2025-12-04T08:52:33.8900103Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 336, in feed_forward_chunk 2025-12-04T08:52:33.8900596Z intermediate_output = self.intermediate(attention_output) 2025-12-04T08:52:33.8901062Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 280, in forward 2025-12-04T08:52:33.8901491Z hidden_states = self.dense(hidden_states) 2025-12-04T08:52:33.8901634Z 2025-12-04T08:52:33.8901742Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:33.8902123Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:33.8902459Z res = mod(**inputs) 2025-12-04T08:52:33.8902796Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8903155Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8903571Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T08:52:33.8903988Z outputs = self.layoutlm( 2025-12-04T08:52:33.8904331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8904697Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8905112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T08:52:33.8905525Z encoder_outputs = self.encoder( 2025-12-04T08:52:33.8905875Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8906240Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8906649Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T08:52:33.8907059Z layer_outputs = layer_module( 2025-12-04T08:52:33.8907416Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:33.8907834Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:33.8908259Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 328, in forward 2025-12-04T08:52:33.8908716Z layer_output = apply_chunking_to_forward( 2025-12-04T08:52:33.8909134Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:52:33.8909543Z return forward_fn(*input_tensors) 2025-12-04T08:52:33.8909990Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 336, in feed_forward_chunk 2025-12-04T08:52:33.8910479Z intermediate_output = self.intermediate(attention_output) 2025-12-04T08:52:33.8910943Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 281, in forward 2025-12-04T08:52:33.8911426Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:52:33.8911829Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T08:52:33.8912181Z return self.act(input) 2025-12-04T08:52:33.8912306Z 2025-12-04T08:52:33.8912414Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:33.8912789Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:33.8913115Z res = mod(**inputs) 2025-12-04T08:52:33.8913453Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8913822Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8914240Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T08:52:33.8914648Z outputs = self.layoutlm( 2025-12-04T08:52:33.8915000Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8915387Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8915820Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T08:52:33.8916264Z encoder_outputs = self.encoder( 2025-12-04T08:52:33.8916644Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8917030Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8917470Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T08:52:33.8917919Z layer_outputs = layer_module( 2025-12-04T08:52:33.8918392Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:33.8918865Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:33.8919317Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 328, in forward 2025-12-04T08:52:33.8919792Z layer_output = apply_chunking_to_forward( 2025-12-04T08:52:33.8920253Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:52:33.8920678Z return forward_fn(*input_tensors) 2025-12-04T08:52:33.8922105Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 337, in feed_forward_chunk 2025-12-04T08:52:33.8922866Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T08:52:33.8923407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 294, in forward 2025-12-04T08:52:33.8923550Z hidden_states = self.dense(hidden_states) 2025-12-04T08:52:33.8923560Z 2025-12-04T08:52:33.8924142Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:33.8924499Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:33.8924583Z res = mod(**inputs) 2025-12-04T08:52:33.8924927Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8925016Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8925325Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T08:52:33.8925421Z outputs = self.layoutlm( 2025-12-04T08:52:33.8925697Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8925780Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8926135Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T08:52:33.8926219Z encoder_outputs = self.encoder( 2025-12-04T08:52:33.8926465Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8926549Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8926843Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T08:52:33.8926932Z layer_outputs = layer_module( 2025-12-04T08:52:33.8927195Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:33.8927296Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:33.8927590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 318, in forward 2025-12-04T08:52:33.8927689Z self_attention_outputs = self.attention( 2025-12-04T08:52:33.8927993Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 257, in forward 2025-12-04T08:52:33.8928075Z self_outputs = self.self( 2025-12-04T08:52:33.8928385Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 183, in forward 2025-12-04T08:52:33.8928561Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-12-04T08:52:33.8928566Z 2025-12-04T08:52:33.8928685Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:33.8928915Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:33.8928989Z res = mod(**inputs) 2025-12-04T08:52:33.8929229Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8929319Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8929614Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T08:52:33.8929700Z outputs = self.layoutlm( 2025-12-04T08:52:33.8929938Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8930019Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8930367Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T08:52:33.8930455Z encoder_outputs = self.encoder( 2025-12-04T08:52:33.8930701Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8930786Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8931078Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T08:52:33.8931200Z layer_outputs = layer_module( 2025-12-04T08:52:33.8931450Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:33.8931537Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:33.8931872Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 318, in forward 2025-12-04T08:52:33.8931970Z self_attention_outputs = self.attention( 2025-12-04T08:52:33.8932279Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 257, in forward 2025-12-04T08:52:33.8932367Z self_outputs = self.self( 2025-12-04T08:52:33.8932692Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 184, in forward 2025-12-04T08:52:33.8932878Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-12-04T08:52:33.8932883Z 2025-12-04T08:52:33.8933006Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:33.8933234Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:33.8933317Z res = mod(**inputs) 2025-12-04T08:52:33.8933565Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8933652Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8933962Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T08:52:33.8934040Z outputs = self.layoutlm( 2025-12-04T08:52:33.8934300Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8934382Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8934692Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T08:52:33.8934782Z encoder_outputs = self.encoder( 2025-12-04T08:52:33.8935034Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8935121Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8935409Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T08:52:33.8935482Z layer_outputs = layer_module( 2025-12-04T08:52:33.8935721Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:33.8935801Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:33.8936083Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 318, in forward 2025-12-04T08:52:33.8936169Z self_attention_outputs = self.attention( 2025-12-04T08:52:33.8936448Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 257, in forward 2025-12-04T08:52:33.8936528Z self_outputs = self.self( 2025-12-04T08:52:33.8936809Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 185, in forward 2025-12-04T08:52:33.8936960Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-12-04T08:52:33.8936970Z 2025-12-04T08:52:33.8937057Z cudagraph partition due to non gpu ops 2025-12-04T08:52:33.8937137Z cudagraph partition due to non gpu ops 2025-12-04T08:52:33.8937250Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:33.8937461Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:33.8937528Z res = mod(**inputs) 2025-12-04T08:52:33.8937762Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8937900Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8938176Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T08:52:33.8938277Z outputs = self.layoutlm( 2025-12-04T08:52:33.8938502Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8938585Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8938861Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T08:52:33.8938936Z encoder_outputs = self.encoder( 2025-12-04T08:52:33.8939168Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8939259Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8939541Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T08:52:33.8939616Z layer_outputs = layer_module( 2025-12-04T08:52:33.8939847Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:33.8939937Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:33.8940216Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 318, in forward 2025-12-04T08:52:33.8940301Z self_attention_outputs = self.attention( 2025-12-04T08:52:33.8940587Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 264, in forward 2025-12-04T08:52:33.8940721Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T08:52:33.8941004Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 217, in forward 2025-12-04T08:52:33.8941095Z hidden_states = self.dense(hidden_states) 2025-12-04T08:52:33.8941099Z 2025-12-04T08:52:33.8941204Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:33.8941417Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:33.8941485Z res = mod(**inputs) 2025-12-04T08:52:33.8941724Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8941799Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8942082Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T08:52:33.8942163Z outputs = self.layoutlm( 2025-12-04T08:52:33.8942409Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8942493Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8942794Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T08:52:33.8942874Z encoder_outputs = self.encoder( 2025-12-04T08:52:33.8943117Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8943203Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8943478Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T08:52:33.8943558Z layer_outputs = layer_module( 2025-12-04T08:52:33.8943783Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:33.8943862Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:33.8944144Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 328, in forward 2025-12-04T08:52:33.8944266Z layer_output = apply_chunking_to_forward( 2025-12-04T08:52:33.8944545Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:52:33.8944644Z return forward_fn(*input_tensors) 2025-12-04T08:52:33.8944958Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 336, in feed_forward_chunk 2025-12-04T08:52:33.8945103Z intermediate_output = self.intermediate(attention_output) 2025-12-04T08:52:33.8945397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 280, in forward 2025-12-04T08:52:33.8945496Z hidden_states = self.dense(hidden_states) 2025-12-04T08:52:33.8945500Z 2025-12-04T08:52:33.8945610Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:33.8945848Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:33.8945933Z res = mod(**inputs) 2025-12-04T08:52:33.8946184Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8946260Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8946546Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T08:52:33.8946624Z outputs = self.layoutlm( 2025-12-04T08:52:33.8946858Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8946939Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8947233Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T08:52:33.8947324Z encoder_outputs = self.encoder( 2025-12-04T08:52:33.8947565Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8947648Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8947938Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T08:52:33.8948019Z layer_outputs = layer_module( 2025-12-04T08:52:33.8948265Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:33.8948351Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:33.8948642Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 328, in forward 2025-12-04T08:52:33.8948742Z layer_output = apply_chunking_to_forward( 2025-12-04T08:52:33.8949026Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:52:33.8949124Z return forward_fn(*input_tensors) 2025-12-04T08:52:33.8949505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 336, in feed_forward_chunk 2025-12-04T08:52:33.8949651Z intermediate_output = self.intermediate(attention_output) 2025-12-04T08:52:33.8949948Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 281, in forward 2025-12-04T08:52:33.8950077Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:52:33.8950322Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T08:52:33.8950400Z return self.act(input) 2025-12-04T08:52:33.8950404Z 2025-12-04T08:52:33.8950518Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:33.8950746Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:33.8950857Z res = mod(**inputs) 2025-12-04T08:52:33.8951096Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8951185Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8951494Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T08:52:33.8951579Z outputs = self.layoutlm( 2025-12-04T08:52:33.8951814Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8951893Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8952191Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T08:52:33.8952272Z encoder_outputs = self.encoder( 2025-12-04T08:52:33.8952534Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8952628Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8952927Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T08:52:33.8953012Z layer_outputs = layer_module( 2025-12-04T08:52:33.8953256Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:33.8953340Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:33.8953640Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 328, in forward 2025-12-04T08:52:33.8953732Z layer_output = apply_chunking_to_forward( 2025-12-04T08:52:33.8954025Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:52:33.8954109Z return forward_fn(*input_tensors) 2025-12-04T08:52:33.8954441Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 337, in feed_forward_chunk 2025-12-04T08:52:33.8954596Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T08:52:33.8954893Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 294, in forward 2025-12-04T08:52:33.8954988Z hidden_states = self.dense(hidden_states) 2025-12-04T08:52:33.8954993Z 2025-12-04T08:52:33.8955106Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:33.8955328Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:33.8955408Z res = mod(**inputs) 2025-12-04T08:52:33.8955647Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8955729Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8956034Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T08:52:33.8956109Z outputs = self.layoutlm( 2025-12-04T08:52:33.8956358Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8956440Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8956731Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T08:52:33.8956826Z encoder_outputs = self.encoder( 2025-12-04T08:52:33.8957065Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8957145Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8957449Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T08:52:33.8957577Z layer_outputs = layer_module( 2025-12-04T08:52:33.8957840Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:33.8957928Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:33.8958419Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 318, in forward 2025-12-04T08:52:33.8958535Z self_attention_outputs = self.attention( 2025-12-04T08:52:33.8958844Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 257, in forward 2025-12-04T08:52:33.8958932Z self_outputs = self.self( 2025-12-04T08:52:33.8959234Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 183, in forward 2025-12-04T08:52:33.8959448Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-12-04T08:52:33.8959452Z 2025-12-04T08:52:33.8959585Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:33.8959807Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:33.8959883Z res = mod(**inputs) 2025-12-04T08:52:33.8960137Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8960220Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8960529Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T08:52:33.8960611Z outputs = self.layoutlm( 2025-12-04T08:52:33.8960853Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8960942Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8961249Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T08:52:33.8961341Z encoder_outputs = self.encoder( 2025-12-04T08:52:33.8961588Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8961674Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8962073Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T08:52:33.8962157Z layer_outputs = layer_module( 2025-12-04T08:52:33.8962397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:33.8962491Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:33.8962784Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 318, in forward 2025-12-04T08:52:33.8962905Z self_attention_outputs = self.attention( 2025-12-04T08:52:33.8963201Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 257, in forward 2025-12-04T08:52:33.8963279Z self_outputs = self.self( 2025-12-04T08:52:33.8963579Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 184, in forward 2025-12-04T08:52:33.8963730Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-12-04T08:52:33.8963735Z 2025-12-04T08:52:33.8963854Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:33.8964069Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:33.8964141Z res = mod(**inputs) 2025-12-04T08:52:33.8964387Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8964468Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8964812Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T08:52:33.8964897Z outputs = self.layoutlm( 2025-12-04T08:52:33.8965135Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8965246Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8965536Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T08:52:33.8965616Z encoder_outputs = self.encoder( 2025-12-04T08:52:33.8965858Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8965934Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8966223Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T08:52:33.8966334Z layer_outputs = layer_module( 2025-12-04T08:52:33.8966577Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:33.8966669Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:33.8966962Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 318, in forward 2025-12-04T08:52:33.8967050Z self_attention_outputs = self.attention( 2025-12-04T08:52:33.8967346Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 257, in forward 2025-12-04T08:52:33.8967421Z self_outputs = self.self( 2025-12-04T08:52:33.8967718Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 185, in forward 2025-12-04T08:52:33.8967882Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-12-04T08:52:33.8967886Z 2025-12-04T08:52:33.8967981Z cudagraph partition due to non gpu ops 2025-12-04T08:52:33.8968072Z cudagraph partition due to non gpu ops 2025-12-04T08:52:33.8968183Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:33.8968397Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:33.8968476Z res = mod(**inputs) 2025-12-04T08:52:33.8968716Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8968800Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8969089Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T08:52:33.8969164Z outputs = self.layoutlm( 2025-12-04T08:52:33.8969407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8969488Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8969781Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T08:52:33.8969867Z encoder_outputs = self.encoder( 2025-12-04T08:52:33.8970103Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8970187Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8970489Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T08:52:33.8970566Z layer_outputs = layer_module( 2025-12-04T08:52:33.8970822Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:33.8970907Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:33.8971257Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 318, in forward 2025-12-04T08:52:33.8971351Z self_attention_outputs = self.attention( 2025-12-04T08:52:33.8971655Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 264, in forward 2025-12-04T08:52:33.8971841Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T08:52:33.8972137Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 217, in forward 2025-12-04T08:52:33.8972230Z hidden_states = self.dense(hidden_states) 2025-12-04T08:52:33.8972243Z 2025-12-04T08:52:33.8972356Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:33.8972572Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:33.8972649Z res = mod(**inputs) 2025-12-04T08:52:33.8972917Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8973003Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8973306Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T08:52:33.8973384Z outputs = self.layoutlm( 2025-12-04T08:52:33.8973628Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8973709Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8974006Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T08:52:33.8974092Z encoder_outputs = self.encoder( 2025-12-04T08:52:33.8974338Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8974420Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8974714Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T08:52:33.8974787Z layer_outputs = layer_module( 2025-12-04T08:52:33.8975024Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:33.8975106Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:33.8975385Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 328, in forward 2025-12-04T08:52:33.8975481Z layer_output = apply_chunking_to_forward( 2025-12-04T08:52:33.8975756Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:52:33.8975835Z return forward_fn(*input_tensors) 2025-12-04T08:52:33.8976165Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 336, in feed_forward_chunk 2025-12-04T08:52:33.8976294Z intermediate_output = self.intermediate(attention_output) 2025-12-04T08:52:33.8976577Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 280, in forward 2025-12-04T08:52:33.8976668Z hidden_states = self.dense(hidden_states) 2025-12-04T08:52:33.8976671Z 2025-12-04T08:52:33.8976781Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:33.8976994Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:33.8977060Z res = mod(**inputs) 2025-12-04T08:52:33.8977293Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8977367Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8977644Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T08:52:33.8977772Z outputs = self.layoutlm( 2025-12-04T08:52:33.8978008Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8978083Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8978394Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T08:52:33.8978470Z encoder_outputs = self.encoder( 2025-12-04T08:52:33.8978702Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8978775Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8979062Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T08:52:33.8979143Z layer_outputs = layer_module( 2025-12-04T08:52:33.8979392Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:33.8979480Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:33.8979758Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 328, in forward 2025-12-04T08:52:33.8979855Z layer_output = apply_chunking_to_forward( 2025-12-04T08:52:33.8980133Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:52:33.8980214Z return forward_fn(*input_tensors) 2025-12-04T08:52:33.8980537Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 336, in feed_forward_chunk 2025-12-04T08:52:33.8980677Z intermediate_output = self.intermediate(attention_output) 2025-12-04T08:52:33.8980966Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 281, in forward 2025-12-04T08:52:33.8981105Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:52:33.8981339Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T08:52:33.8981419Z return self.act(input) 2025-12-04T08:52:33.8981425Z 2025-12-04T08:52:33.8981545Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:33.8981762Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:33.8981838Z res = mod(**inputs) 2025-12-04T08:52:33.8982079Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8982161Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8982457Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T08:52:33.8982536Z outputs = self.layoutlm( 2025-12-04T08:52:33.8982775Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8982862Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8983154Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T08:52:33.8983243Z encoder_outputs = self.encoder( 2025-12-04T08:52:33.8983479Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8983556Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8983854Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T08:52:33.8983931Z layer_outputs = layer_module( 2025-12-04T08:52:33.8984175Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:33.8984302Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:33.8984574Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 328, in forward 2025-12-04T08:52:33.8984667Z layer_output = apply_chunking_to_forward( 2025-12-04T08:52:33.8984980Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:52:33.8985061Z return forward_fn(*input_tensors) 2025-12-04T08:52:33.8985392Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 337, in feed_forward_chunk 2025-12-04T08:52:33.8985538Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T08:52:33.8985836Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 294, in forward 2025-12-04T08:52:33.8985947Z hidden_states = self.dense(hidden_states) 2025-12-04T08:52:33.8985956Z 2025-12-04T08:52:33.8986068Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:33.8986297Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:33.8986370Z res = mod(**inputs) 2025-12-04T08:52:33.8986613Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8986692Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8986986Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T08:52:33.8987068Z outputs = self.layoutlm( 2025-12-04T08:52:33.8987303Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8987381Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8987685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T08:52:33.8987764Z encoder_outputs = self.encoder( 2025-12-04T08:52:33.8988009Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8988090Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8988381Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T08:52:33.8988467Z layer_outputs = layer_module( 2025-12-04T08:52:33.8988710Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:33.8988796Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:33.8989099Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 318, in forward 2025-12-04T08:52:33.8989194Z self_attention_outputs = self.attention( 2025-12-04T08:52:33.8989495Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 257, in forward 2025-12-04T08:52:33.8989573Z self_outputs = self.self( 2025-12-04T08:52:33.8989865Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 183, in forward 2025-12-04T08:52:33.8990030Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-12-04T08:52:33.8990035Z 2025-12-04T08:52:33.8990148Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:33.8990369Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:33.8990438Z res = mod(**inputs) 2025-12-04T08:52:33.8990677Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8990768Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8991113Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T08:52:33.8991193Z outputs = self.layoutlm( 2025-12-04T08:52:33.8991439Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8991545Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8991844Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T08:52:33.8991929Z encoder_outputs = self.encoder( 2025-12-04T08:52:33.8992182Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8992267Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8992559Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T08:52:33.8992682Z layer_outputs = layer_module( 2025-12-04T08:52:33.8992927Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:33.8993013Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:33.8993312Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 318, in forward 2025-12-04T08:52:33.8993402Z self_attention_outputs = self.attention( 2025-12-04T08:52:33.8993710Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 257, in forward 2025-12-04T08:52:33.8993794Z self_outputs = self.self( 2025-12-04T08:52:33.8994083Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 184, in forward 2025-12-04T08:52:33.8994245Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-12-04T08:52:33.8994249Z 2025-12-04T08:52:33.8994363Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:33.8994579Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:33.8994658Z res = mod(**inputs) 2025-12-04T08:52:33.8994906Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8994994Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8995297Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T08:52:33.8995375Z outputs = self.layoutlm( 2025-12-04T08:52:33.8995626Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8995707Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8996011Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T08:52:33.8996103Z encoder_outputs = self.encoder( 2025-12-04T08:52:33.8996352Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8996438Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8996727Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T08:52:33.8996806Z layer_outputs = layer_module( 2025-12-04T08:52:33.8997060Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:33.8997151Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:33.8997451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 318, in forward 2025-12-04T08:52:33.8997553Z self_attention_outputs = self.attention( 2025-12-04T08:52:33.8997900Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 257, in forward 2025-12-04T08:52:33.8997992Z self_outputs = self.self( 2025-12-04T08:52:33.8998417Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 185, in forward 2025-12-04T08:52:33.8998601Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-12-04T08:52:33.8998606Z 2025-12-04T08:52:33.8998707Z cudagraph partition due to non gpu ops 2025-12-04T08:52:33.8998797Z cudagraph partition due to non gpu ops 2025-12-04T08:52:33.8998921Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:33.8999146Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:33.8999218Z res = mod(**inputs) 2025-12-04T08:52:33.8999516Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.8999611Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.8999903Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T08:52:33.8999992Z outputs = self.layoutlm( 2025-12-04T08:52:33.9000229Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.9000314Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.9000607Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T08:52:33.9000687Z encoder_outputs = self.encoder( 2025-12-04T08:52:33.9000930Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.9001009Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.9001306Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T08:52:33.9001394Z layer_outputs = layer_module( 2025-12-04T08:52:33.9001641Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:33.9001735Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:33.9002027Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 318, in forward 2025-12-04T08:52:33.9002115Z self_attention_outputs = self.attention( 2025-12-04T08:52:33.9002419Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 264, in forward 2025-12-04T08:52:33.9002558Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T08:52:33.9002870Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 217, in forward 2025-12-04T08:52:33.9002968Z hidden_states = self.dense(hidden_states) 2025-12-04T08:52:33.9002971Z 2025-12-04T08:52:33.9003081Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:33.9003311Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:33.9003381Z res = mod(**inputs) 2025-12-04T08:52:33.9003622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.9003713Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.9004023Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T08:52:33.9004107Z outputs = self.layoutlm( 2025-12-04T08:52:33.9004342Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.9004476Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.9004776Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T08:52:33.9004854Z encoder_outputs = self.encoder( 2025-12-04T08:52:33.9005123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.9005197Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.9005472Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T08:52:33.9005551Z layer_outputs = layer_module( 2025-12-04T08:52:33.9005784Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:33.9005862Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:33.9006171Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 328, in forward 2025-12-04T08:52:33.9006260Z layer_output = apply_chunking_to_forward( 2025-12-04T08:52:33.9006533Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:52:33.9006615Z return forward_fn(*input_tensors) 2025-12-04T08:52:33.9006922Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 336, in feed_forward_chunk 2025-12-04T08:52:33.9007053Z intermediate_output = self.intermediate(attention_output) 2025-12-04T08:52:33.9007326Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 280, in forward 2025-12-04T08:52:33.9007419Z hidden_states = self.dense(hidden_states) 2025-12-04T08:52:33.9007423Z 2025-12-04T08:52:33.9007530Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:33.9007738Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:33.9007811Z res = mod(**inputs) 2025-12-04T08:52:33.9008035Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.9008112Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.9008396Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T08:52:33.9008467Z outputs = self.layoutlm( 2025-12-04T08:52:33.9008696Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.9008771Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.9009046Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T08:52:33.9009133Z encoder_outputs = self.encoder( 2025-12-04T08:52:33.9009370Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.9009447Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.9009742Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T08:52:33.9009821Z layer_outputs = layer_module( 2025-12-04T08:52:33.9010072Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:33.9010160Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:33.9010461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 328, in forward 2025-12-04T08:52:33.9010564Z layer_output = apply_chunking_to_forward( 2025-12-04T08:52:33.9010850Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:52:33.9010994Z return forward_fn(*input_tensors) 2025-12-04T08:52:33.9011323Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 336, in feed_forward_chunk 2025-12-04T08:52:33.9011473Z intermediate_output = self.intermediate(attention_output) 2025-12-04T08:52:33.9011771Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 281, in forward 2025-12-04T08:52:33.9011902Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:52:33.9012124Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T08:52:33.9012202Z return self.act(input) 2025-12-04T08:52:33.9012206Z 2025-12-04T08:52:33.9012309Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:33.9012541Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:33.9012609Z res = mod(**inputs) 2025-12-04T08:52:33.9012833Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.9012914Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.9013189Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T08:52:33.9013268Z outputs = self.layoutlm( 2025-12-04T08:52:33.9013493Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.9013571Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.9013876Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T08:52:33.9013958Z encoder_outputs = self.encoder( 2025-12-04T08:52:33.9014196Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.9014279Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.9014567Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T08:52:33.9014655Z layer_outputs = layer_module( 2025-12-04T08:52:33.9014903Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:33.9014985Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:33.9015267Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 328, in forward 2025-12-04T08:52:33.9015354Z layer_output = apply_chunking_to_forward( 2025-12-04T08:52:33.9015625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:52:33.9015708Z return forward_fn(*input_tensors) 2025-12-04T08:52:33.9016017Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 337, in feed_forward_chunk 2025-12-04T08:52:33.9016160Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T08:52:33.9016439Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 294, in forward 2025-12-04T08:52:33.9016522Z hidden_states = self.dense(hidden_states) 2025-12-04T08:52:33.9016534Z 2025-12-04T08:52:33.9016639Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:33.9016843Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:33.9016915Z res = mod(**inputs) 2025-12-04T08:52:33.9017140Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.9017220Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.9017541Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T08:52:33.9017614Z outputs = self.layoutlm( 2025-12-04T08:52:33.9017863Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.9017937Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.9018215Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T08:52:33.9018296Z encoder_outputs = self.encoder( 2025-12-04T08:52:33.9018521Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.9018593Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.9018911Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T08:52:33.9018987Z layer_outputs = layer_module( 2025-12-04T08:52:33.9019226Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:33.9019308Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:33.9019582Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 318, in forward 2025-12-04T08:52:33.9019676Z self_attention_outputs = self.attention( 2025-12-04T08:52:33.9019952Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 257, in forward 2025-12-04T08:52:33.9020024Z self_outputs = self.self( 2025-12-04T08:52:33.9020315Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 183, in forward 2025-12-04T08:52:33.9020465Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-12-04T08:52:33.9020472Z 2025-12-04T08:52:33.9020584Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:33.9020982Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:33.9021060Z res = mod(**inputs) 2025-12-04T08:52:33.9021297Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.9021372Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.9021655Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T08:52:33.9021727Z outputs = self.layoutlm( 2025-12-04T08:52:33.9021952Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.9022038Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.9022314Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T08:52:33.9022392Z encoder_outputs = self.encoder( 2025-12-04T08:52:33.9022620Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.9022696Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.9022982Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T08:52:33.9023057Z layer_outputs = layer_module( 2025-12-04T08:52:33.9023290Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:33.9023379Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:33.9023663Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 318, in forward 2025-12-04T08:52:33.9023757Z self_attention_outputs = self.attention( 2025-12-04T08:52:33.9024111Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 257, in forward 2025-12-04T08:52:33.9024186Z self_outputs = self.self( 2025-12-04T08:52:33.9024496Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 184, in forward 2025-12-04T08:52:33.9024643Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-12-04T08:52:33.9024648Z 2025-12-04T08:52:33.9024755Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:33.9024967Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:33.9025036Z res = mod(**inputs) 2025-12-04T08:52:33.9025265Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.9025426Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.9025702Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T08:52:33.9025781Z outputs = self.layoutlm( 2025-12-04T08:52:33.9026004Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.9026087Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.9026359Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T08:52:33.9026433Z encoder_outputs = self.encoder( 2025-12-04T08:52:33.9026662Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.9026736Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.9027013Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T08:52:33.9027102Z layer_outputs = layer_module( 2025-12-04T08:52:33.9027331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:33.9027420Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:33.9027694Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 318, in forward 2025-12-04T08:52:33.9027781Z self_attention_outputs = self.attention( 2025-12-04T08:52:33.9028065Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 257, in forward 2025-12-04T08:52:33.9028140Z self_outputs = self.self( 2025-12-04T08:52:33.9028419Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 185, in forward 2025-12-04T08:52:33.9028577Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-12-04T08:52:33.9028581Z 2025-12-04T08:52:33.9028667Z cudagraph partition due to non gpu ops 2025-12-04T08:52:33.9028757Z cudagraph partition due to non gpu ops 2025-12-04T08:52:33.9028861Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:33.9029064Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:33.9029139Z res = mod(**inputs) 2025-12-04T08:52:33.9029372Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.9029452Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.9029719Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T08:52:33.9029788Z outputs = self.layoutlm( 2025-12-04T08:52:33.9030010Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.9030131Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.9030402Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T08:52:33.9030485Z encoder_outputs = self.encoder( 2025-12-04T08:52:33.9030724Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.9030803Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.9031077Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T08:52:33.9031148Z layer_outputs = layer_module( 2025-12-04T08:52:33.9031384Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:33.9031463Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:33.9031760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 318, in forward 2025-12-04T08:52:33.9031852Z self_attention_outputs = self.attention( 2025-12-04T08:52:33.9032127Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 264, in forward 2025-12-04T08:52:33.9032267Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T08:52:33.9032543Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 217, in forward 2025-12-04T08:52:33.9032628Z hidden_states = self.dense(hidden_states) 2025-12-04T08:52:33.9032632Z 2025-12-04T08:52:33.9032744Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:33.9032955Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:33.9033029Z res = mod(**inputs) 2025-12-04T08:52:33.9033259Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.9033332Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.9033615Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T08:52:33.9033687Z outputs = self.layoutlm( 2025-12-04T08:52:33.9033913Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.9033995Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.9034276Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T08:52:33.9034359Z encoder_outputs = self.encoder( 2025-12-04T08:52:33.9034584Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.9034660Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.9034965Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T08:52:33.9035041Z layer_outputs = layer_module( 2025-12-04T08:52:33.9035290Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:33.9035376Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:33.9035669Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 328, in forward 2025-12-04T08:52:33.9035767Z layer_output = apply_chunking_to_forward( 2025-12-04T08:52:33.9036051Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:52:33.9036133Z return forward_fn(*input_tensors) 2025-12-04T08:52:33.9036518Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 336, in feed_forward_chunk 2025-12-04T08:52:33.9036652Z intermediate_output = self.intermediate(attention_output) 2025-12-04T08:52:33.9036954Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 280, in forward 2025-12-04T08:52:33.9037063Z hidden_states = self.dense(hidden_states) 2025-12-04T08:52:33.9037068Z 2025-12-04T08:52:33.9037178Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:33.9037402Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:33.9037471Z res = mod(**inputs) 2025-12-04T08:52:33.9037720Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.9037801Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.9038112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T08:52:33.9038200Z outputs = self.layoutlm( 2025-12-04T08:52:33.9038510Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.9038596Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.9038910Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T08:52:33.9038991Z encoder_outputs = self.encoder( 2025-12-04T08:52:33.9039241Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.9039323Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.9039632Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T08:52:33.9039722Z layer_outputs = layer_module( 2025-12-04T08:52:33.9039967Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:33.9040053Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:33.9040354Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 328, in forward 2025-12-04T08:52:33.9040450Z layer_output = apply_chunking_to_forward( 2025-12-04T08:52:33.9040751Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:52:33.9040834Z return forward_fn(*input_tensors) 2025-12-04T08:52:33.9041163Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 336, in feed_forward_chunk 2025-12-04T08:52:33.9041303Z intermediate_output = self.intermediate(attention_output) 2025-12-04T08:52:33.9041600Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 281, in forward 2025-12-04T08:52:33.9041744Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:52:33.9041964Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T08:52:33.9042040Z return self.act(input) 2025-12-04T08:52:33.9042044Z 2025-12-04T08:52:33.9042158Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:33.9042368Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:33.9042444Z res = mod(**inputs) 2025-12-04T08:52:33.9042681Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.9042760Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.9043057Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T08:52:33.9043137Z outputs = self.layoutlm( 2025-12-04T08:52:33.9043421Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.9043508Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.9043799Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T08:52:33.9043904Z encoder_outputs = self.encoder( 2025-12-04T08:52:33.9044142Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.9044220Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.9044523Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T08:52:33.9044600Z layer_outputs = layer_module( 2025-12-04T08:52:33.9044864Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:33.9044961Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:33.9045256Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 328, in forward 2025-12-04T08:52:33.9045359Z layer_output = apply_chunking_to_forward( 2025-12-04T08:52:33.9045644Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:52:33.9045725Z return forward_fn(*input_tensors) 2025-12-04T08:52:33.9046060Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 337, in feed_forward_chunk 2025-12-04T08:52:33.9046204Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T08:52:33.9046504Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 294, in forward 2025-12-04T08:52:33.9046605Z hidden_states = self.dense(hidden_states) 2025-12-04T08:52:33.9046608Z 2025-12-04T08:52:33.9046721Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:33.9046948Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:33.9047024Z res = mod(**inputs) 2025-12-04T08:52:33.9047263Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.9047350Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.9047642Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T08:52:33.9047726Z outputs = self.layoutlm( 2025-12-04T08:52:33.9047963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.9048045Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.9048346Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T08:52:33.9048424Z encoder_outputs = self.encoder( 2025-12-04T08:52:33.9048669Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.9048755Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.9049047Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T08:52:33.9049129Z layer_outputs = layer_module( 2025-12-04T08:52:33.9049374Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:33.9049468Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:33.9049823Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 318, in forward 2025-12-04T08:52:33.9049956Z self_attention_outputs = self.attention( 2025-12-04T08:52:33.9050252Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 257, in forward 2025-12-04T08:52:33.9050330Z self_outputs = self.self( 2025-12-04T08:52:33.9050640Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 183, in forward 2025-12-04T08:52:33.9050802Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-12-04T08:52:33.9050806Z 2025-12-04T08:52:33.9050915Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:33.9051135Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:33.9051205Z res = mod(**inputs) 2025-12-04T08:52:33.9051441Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.9051551Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.9051846Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T08:52:33.9051925Z outputs = self.layoutlm( 2025-12-04T08:52:33.9052169Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.9052247Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.9052541Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T08:52:33.9052620Z encoder_outputs = self.encoder( 2025-12-04T08:52:33.9052851Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.9052931Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.9053206Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T08:52:33.9053278Z layer_outputs = layer_module( 2025-12-04T08:52:33.9053505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:33.9053586Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:33.9053859Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 318, in forward 2025-12-04T08:52:33.9053941Z self_attention_outputs = self.attention( 2025-12-04T08:52:33.9054208Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 257, in forward 2025-12-04T08:52:33.9054286Z self_outputs = self.self( 2025-12-04T08:52:33.9054553Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 184, in forward 2025-12-04T08:52:33.9054703Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-12-04T08:52:33.9054710Z 2025-12-04T08:52:33.9054811Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:33.9055009Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:33.9055086Z res = mod(**inputs) 2025-12-04T08:52:33.9055306Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.9055378Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.9055654Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T08:52:33.9055727Z outputs = self.layoutlm( 2025-12-04T08:52:33.9055955Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.9056032Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.9056347Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T08:52:33.9056435Z encoder_outputs = self.encoder( 2025-12-04T08:52:33.9056654Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.9056756Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.9057031Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T08:52:33.9057105Z layer_outputs = layer_module( 2025-12-04T08:52:33.9057344Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:33.9057425Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:33.9057705Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 318, in forward 2025-12-04T08:52:33.9057833Z self_attention_outputs = self.attention( 2025-12-04T08:52:33.9058120Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 257, in forward 2025-12-04T08:52:33.9058195Z self_outputs = self.self( 2025-12-04T08:52:33.9058463Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 185, in forward 2025-12-04T08:52:33.9058607Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-12-04T08:52:33.9058610Z 2025-12-04T08:52:33.9058710Z cudagraph partition due to non gpu ops 2025-12-04T08:52:33.9058790Z cudagraph partition due to non gpu ops 2025-12-04T08:52:33.9058891Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:33.9059098Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:33.9059164Z res = mod(**inputs) 2025-12-04T08:52:33.9059393Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.9059465Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.9059734Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T08:52:33.9059814Z outputs = self.layoutlm( 2025-12-04T08:52:33.9060034Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.9060120Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.9060389Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T08:52:33.9060462Z encoder_outputs = self.encoder( 2025-12-04T08:52:33.9060687Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.9060763Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.9061034Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T08:52:33.9061112Z layer_outputs = layer_module( 2025-12-04T08:52:33.9061332Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:33.9061420Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:33.9061689Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 318, in forward 2025-12-04T08:52:33.9061770Z self_attention_outputs = self.attention( 2025-12-04T08:52:33.9062047Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 264, in forward 2025-12-04T08:52:33.9062174Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T08:52:33.9062486Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 217, in forward 2025-12-04T08:52:33.9062572Z hidden_states = self.dense(hidden_states) 2025-12-04T08:52:33.9062575Z 2025-12-04T08:52:33.9062677Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:33.9062901Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:33.9062965Z res = mod(**inputs) 2025-12-04T08:52:33.9063184Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.9063272Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.9063541Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T08:52:33.9063618Z outputs = self.layoutlm( 2025-12-04T08:52:33.9063856Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.9063933Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.9064210Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T08:52:33.9064285Z encoder_outputs = self.encoder( 2025-12-04T08:52:33.9064499Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.9064577Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.9064845Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T08:52:33.9064926Z layer_outputs = layer_module( 2025-12-04T08:52:33.9065156Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:33.9065236Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:33.9065514Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 328, in forward 2025-12-04T08:52:33.9065601Z layer_output = apply_chunking_to_forward( 2025-12-04T08:52:33.9065867Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:52:33.9065944Z return forward_fn(*input_tensors) 2025-12-04T08:52:33.9066252Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 336, in feed_forward_chunk 2025-12-04T08:52:33.9066381Z intermediate_output = self.intermediate(attention_output) 2025-12-04T08:52:33.9066653Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 280, in forward 2025-12-04T08:52:33.9066739Z hidden_states = self.dense(hidden_states) 2025-12-04T08:52:33.9066752Z 2025-12-04T08:52:33.9066856Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:33.9067060Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:33.9067132Z res = mod(**inputs) 2025-12-04T08:52:33.9067355Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.9067432Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.9067711Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T08:52:33.9067783Z outputs = self.layoutlm( 2025-12-04T08:52:33.9068012Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.9068086Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.9068359Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T08:52:33.9068448Z encoder_outputs = self.encoder( 2025-12-04T08:52:33.9068717Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.9068793Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.9069094Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T08:52:33.9069166Z layer_outputs = layer_module( 2025-12-04T08:52:33.9069412Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:33.9069493Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:33.9069784Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 328, in forward 2025-12-04T08:52:33.9069878Z layer_output = apply_chunking_to_forward( 2025-12-04T08:52:33.9070183Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:52:33.9070266Z return forward_fn(*input_tensors) 2025-12-04T08:52:33.9070572Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 336, in feed_forward_chunk 2025-12-04T08:52:33.9070694Z intermediate_output = self.intermediate(attention_output) 2025-12-04T08:52:33.9070971Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 281, in forward 2025-12-04T08:52:33.9071085Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:52:33.9071304Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T08:52:33.9071381Z return self.act(input) 2025-12-04T08:52:33.9071384Z 2025-12-04T08:52:33.9071489Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:33.9071707Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:33.9071771Z res = mod(**inputs) 2025-12-04T08:52:33.9071989Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.9072076Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.9072347Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T08:52:33.9072428Z outputs = self.layoutlm( 2025-12-04T08:52:33.9072647Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.9072720Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.9072997Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T08:52:33.9073075Z encoder_outputs = self.encoder( 2025-12-04T08:52:33.9073295Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.9073376Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.9073648Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T08:52:33.9073729Z layer_outputs = layer_module( 2025-12-04T08:52:33.9073951Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:33.9074029Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:33.9074302Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 328, in forward 2025-12-04T08:52:33.9074385Z layer_output = apply_chunking_to_forward( 2025-12-04T08:52:33.9074645Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:52:33.9074766Z return forward_fn(*input_tensors) 2025-12-04T08:52:33.9075065Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 337, in feed_forward_chunk 2025-12-04T08:52:33.9075203Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T08:52:33.9075492Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 294, in forward 2025-12-04T08:52:33.9075573Z hidden_states = self.dense(hidden_states) 2025-12-04T08:52:33.9075577Z 2025-12-04T08:52:33.9075689Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:33.9075890Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:33.9075963Z res = mod(**inputs) 2025-12-04T08:52:33.9076180Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.9076278Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.9076558Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T08:52:33.9076628Z outputs = self.layoutlm( 2025-12-04T08:52:33.9076849Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.9076931Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.9077195Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T08:52:33.9077277Z encoder_outputs = self.encoder( 2025-12-04T08:52:33.9077494Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.9077566Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.9077843Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T08:52:33.9077915Z layer_outputs = layer_module( 2025-12-04T08:52:33.9078141Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:33.9078303Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:33.9078616Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 318, in forward 2025-12-04T08:52:33.9078718Z self_attention_outputs = self.attention( 2025-12-04T08:52:33.9079016Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 257, in forward 2025-12-04T08:52:33.9079098Z self_outputs = self.self( 2025-12-04T08:52:33.9079408Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 183, in forward 2025-12-04T08:52:33.9079580Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-12-04T08:52:33.9079584Z 2025-12-04T08:52:33.9079721Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:33.9079927Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:33.9079997Z res = mod(**inputs) 2025-12-04T08:52:33.9080229Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.9080305Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.9080585Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T08:52:33.9080658Z outputs = self.layoutlm( 2025-12-04T08:52:33.9080879Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.9080963Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.9081279Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T08:52:33.9081357Z encoder_outputs = self.encoder( 2025-12-04T08:52:33.9081589Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.9081696Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.9081977Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T08:52:33.9082050Z layer_outputs = layer_module( 2025-12-04T08:52:33.9082276Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:33.9082363Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:33.9082635Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 318, in forward 2025-12-04T08:52:33.9082743Z self_attention_outputs = self.attention( 2025-12-04T08:52:33.9083024Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 257, in forward 2025-12-04T08:52:33.9083096Z self_outputs = self.self( 2025-12-04T08:52:33.9083379Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 184, in forward 2025-12-04T08:52:33.9083522Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-12-04T08:52:33.9083526Z 2025-12-04T08:52:33.9083633Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:33.9083844Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:33.9083910Z res = mod(**inputs) 2025-12-04T08:52:33.9084140Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.9084216Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.9084497Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T08:52:33.9084577Z outputs = self.layoutlm( 2025-12-04T08:52:33.9084803Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.9084877Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.9085161Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T08:52:33.9085239Z encoder_outputs = self.encoder( 2025-12-04T08:52:33.9085469Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.9085543Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.9085823Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T08:52:33.9085903Z layer_outputs = layer_module( 2025-12-04T08:52:33.9086131Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:33.9086221Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:33.9086495Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 318, in forward 2025-12-04T08:52:33.9086580Z self_attention_outputs = self.attention( 2025-12-04T08:52:33.9086865Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 257, in forward 2025-12-04T08:52:33.9086939Z self_outputs = self.self( 2025-12-04T08:52:33.9087213Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 185, in forward 2025-12-04T08:52:33.9087375Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-12-04T08:52:33.9087416Z 2025-12-04T08:52:33.9087504Z cudagraph partition due to non gpu ops 2025-12-04T08:52:33.9087595Z cudagraph partition due to non gpu ops 2025-12-04T08:52:33.9087703Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:33.9087929Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:33.9088004Z res = mod(**inputs) 2025-12-04T08:52:33.9088237Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.9088314Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.9088605Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T08:52:33.9088676Z outputs = self.layoutlm( 2025-12-04T08:52:33.9088925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.9089003Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.9089282Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T08:52:33.9089367Z encoder_outputs = self.encoder( 2025-12-04T08:52:33.9089589Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.9089668Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.9089942Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T08:52:33.9090015Z layer_outputs = layer_module( 2025-12-04T08:52:33.9090251Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:33.9090332Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:33.9090609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 318, in forward 2025-12-04T08:52:33.9090704Z self_attention_outputs = self.attention( 2025-12-04T08:52:33.9090980Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 264, in forward 2025-12-04T08:52:33.9091118Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T08:52:33.9091394Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 217, in forward 2025-12-04T08:52:33.9091481Z hidden_states = self.dense(hidden_states) 2025-12-04T08:52:33.9091484Z 2025-12-04T08:52:33.9091597Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:33.9091805Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:33.9091883Z res = mod(**inputs) 2025-12-04T08:52:33.9092109Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.9092185Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.9092465Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T08:52:33.9092539Z outputs = self.layoutlm( 2025-12-04T08:52:33.9092762Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.9092844Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.9093119Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T08:52:33.9093202Z encoder_outputs = self.encoder( 2025-12-04T08:52:33.9093425Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.9093502Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.9093826Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T08:52:33.9093901Z layer_outputs = layer_module( 2025-12-04T08:52:33.9094159Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:33.9094244Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:33.9094521Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 328, in forward 2025-12-04T08:52:33.9094616Z layer_output = apply_chunking_to_forward( 2025-12-04T08:52:33.9094884Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:52:33.9094961Z return forward_fn(*input_tensors) 2025-12-04T08:52:33.9095297Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 336, in feed_forward_chunk 2025-12-04T08:52:33.9095423Z intermediate_output = self.intermediate(attention_output) 2025-12-04T08:52:33.9095708Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 280, in forward 2025-12-04T08:52:33.9095794Z hidden_states = self.dense(hidden_states) 2025-12-04T08:52:33.9095798Z 2025-12-04T08:52:33.9095904Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:33.9096115Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:33.9096180Z res = mod(**inputs) 2025-12-04T08:52:33.9096403Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.9096484Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.9096761Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T08:52:33.9096838Z outputs = self.layoutlm( 2025-12-04T08:52:33.9097061Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.9097136Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.9097417Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T08:52:33.9097493Z encoder_outputs = self.encoder( 2025-12-04T08:52:33.9097721Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.9097794Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.9098071Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T08:52:33.9098153Z layer_outputs = layer_module( 2025-12-04T08:52:33.9098383Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:33.9098471Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:33.9098744Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 328, in forward 2025-12-04T08:52:33.9098826Z layer_output = apply_chunking_to_forward( 2025-12-04T08:52:33.9099087Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:52:33.9099159Z return forward_fn(*input_tensors) 2025-12-04T08:52:33.9099455Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 336, in feed_forward_chunk 2025-12-04T08:52:33.9099577Z intermediate_output = self.intermediate(attention_output) 2025-12-04T08:52:33.9099878Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 281, in forward 2025-12-04T08:52:33.9099996Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:52:33.9100206Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T08:52:33.9100292Z return self.act(input) 2025-12-04T08:52:33.9100297Z 2025-12-04T08:52:33.9100405Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:33.9100598Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:33.9100661Z res = mod(**inputs) 2025-12-04T08:52:33.9100883Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.9100954Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.9101233Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 721, in forward 2025-12-04T08:52:33.9101326Z outputs = self.layoutlm( 2025-12-04T08:52:33.9101547Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.9101627Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.9101902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 613, in forward 2025-12-04T08:52:33.9101983Z encoder_outputs = self.encoder( 2025-12-04T08:52:33.9102203Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.9102275Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.9102565Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 369, in forward 2025-12-04T08:52:33.9102636Z layer_outputs = layer_module( 2025-12-04T08:52:33.9102868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:52:33.9102954Z return super().__call__(*args, **kwargs) 2025-12-04T08:52:33.9103228Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 328, in forward 2025-12-04T08:52:33.9103322Z layer_output = apply_chunking_to_forward( 2025-12-04T08:52:33.9103586Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:52:33.9103661Z return forward_fn(*input_tensors) 2025-12-04T08:52:33.9103973Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 337, in feed_forward_chunk 2025-12-04T08:52:33.9104106Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T08:52:33.9104384Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 294, in forward 2025-12-04T08:52:33.9104471Z hidden_states = self.dense(hidden_states) 2025-12-04T08:52:33.9104475Z 2025-12-04T08:52:33.9104578Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:33.9104785Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:33.9104852Z res = mod(**inputs) 2025-12-04T08:52:33.9105075Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.9105159Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.9105439Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 735, in forward 2025-12-04T08:52:33.9105541Z prediction_scores = self.cls(sequence_output) 2025-12-04T08:52:33.9105824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 456, in forward 2025-12-04T08:52:33.9105991Z prediction_scores = self.predictions(sequence_output) 2025-12-04T08:52:33.9106279Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 444, in forward 2025-12-04T08:52:33.9106385Z hidden_states = self.transform(hidden_states) 2025-12-04T08:52:33.9106680Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 419, in forward 2025-12-04T08:52:33.9106763Z hidden_states = self.dense(hidden_states) 2025-12-04T08:52:33.9106767Z 2025-12-04T08:52:33.9106868Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:33.9107074Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:33.9107139Z res = mod(**inputs) 2025-12-04T08:52:33.9107359Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.9107469Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.9107748Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 735, in forward 2025-12-04T08:52:33.9107855Z prediction_scores = self.cls(sequence_output) 2025-12-04T08:52:33.9108135Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 456, in forward 2025-12-04T08:52:33.9108252Z prediction_scores = self.predictions(sequence_output) 2025-12-04T08:52:33.9108540Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 445, in forward 2025-12-04T08:52:33.9108636Z hidden_states = self.decoder(hidden_states) 2025-12-04T08:52:33.9108640Z 2025-12-04T08:52:33.9108754Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:52:33.9108963Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:52:33.9109035Z res = mod(**inputs) 2025-12-04T08:52:33.9109271Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:52:33.9109351Z output = func(self, *args, **kwargs) 2025-12-04T08:52:33.9109630Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 740, in forward 2025-12-04T08:52:33.9109718Z masked_lm_loss = loss_fct( 2025-12-04T08:52:33.9109722Z 2025-12-04T08:52:44.2903556Z Compilation time (from dynamo_timed): 17.371067213 2025-12-04T08:52:44.3019262Z pass 2025-12-04T08:52:44.3019874Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T08:52:44.3020938Z TIMING: _recursive_pre_grad_passes:0.00784 _recursive_joint_graph_passes:0.47465 _recursive_post_grad_passes:0.07392 async_compile.wait:0.66247 code_gen:9.44026 inductor_compile:10.76243 backend_compile:14.31986 gc:0.00194 entire_frame_compile:17.37107 total_wall_time:17.37107 2025-12-04T08:52:44.3021963Z STATS: call_* op count: 432 | FakeTensorMode.__torch_dispatch__:8841 | FakeTensor.__torch_dispatch__:4457 | ProxyTorchDispatchMode.__torch_dispatch__:2621 2025-12-04T08:52:44.3022475Z Dynamo produced 1 graphs covering 432 ops with 0 graph breaks (0 unique) 2025-12-04T08:52:46.7438397Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/__init__.py:65: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-12-04T08:52:46.7439334Z import pynvml # type: ignore[import] 2025-12-04T08:52:50.1803247Z 2025-12-04T08:52:56.4106375Z loading model: 0it [00:00, ?it/s] 2025-12-04T08:52:56.4106939Z loading model: 0it [00:06, ?it/s] 2025-12-04T08:52:56.4134571Z cpu eval M2M100ForConditionalGeneration 2025-12-04T08:52:58.3187914Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T08:52:59.1946961Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T08:53:00.0892562Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T08:53:16.0951367Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.0951898Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.0952243Z res = mod(**inputs) 2025-12-04T08:53:16.0952661Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.0953104Z outputs = self.model( 2025-12-04T08:53:16.0953524Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T08:53:16.0954301Z encoder_outputs = self.encoder( 2025-12-04T08:53:16.0954724Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 849, in forward 2025-12-04T08:53:16.0955176Z embed_pos = self.embed_positions(input_ids, inputs_embeds) 2025-12-04T08:53:16.0955596Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context 2025-12-04T08:53:16.0955998Z return func(*args, **kwargs) 2025-12-04T08:53:16.0956393Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 149, in forward 2025-12-04T08:53:16.0956933Z position_ids = create_position_ids_from_input_ids(input_ids, self.padding_idx, past_key_values_length).to( 2025-12-04T08:53:16.0957533Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 81, in create_position_ids_from_input_ids 2025-12-04T08:53:16.0958002Z mask = input_ids.ne(padding_idx).int() 2025-12-04T08:53:16.0958159Z 2025-12-04T08:53:16.0958280Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.0958867Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.0959250Z res = mod(**inputs) 2025-12-04T08:53:16.0959699Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.0960142Z outputs = self.model( 2025-12-04T08:53:16.0960555Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.0961030Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.0974783Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1098, in forward 2025-12-04T08:53:16.0975451Z positions = self.embed_positions(input_ids, inputs_embeds, past_key_values_length) 2025-12-04T08:53:16.0975983Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context 2025-12-04T08:53:16.0976365Z return func(*args, **kwargs) 2025-12-04T08:53:16.0976765Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 149, in forward 2025-12-04T08:53:16.0977309Z position_ids = create_position_ids_from_input_ids(input_ids, self.padding_idx, past_key_values_length).to( 2025-12-04T08:53:16.0977914Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 81, in create_position_ids_from_input_ids 2025-12-04T08:53:16.0978379Z mask = input_ids.ne(padding_idx).int() 2025-12-04T08:53:16.0978534Z 2025-12-04T08:53:16.0978620Z cudagraph partition due to non gpu ops 2025-12-04T08:53:16.0978841Z cudagraph partition due to non gpu ops 2025-12-04T08:53:16.0979051Z cudagraph partition due to non gpu ops 2025-12-04T08:53:16.0979266Z cudagraph partition due to non gpu ops 2025-12-04T08:53:16.0979648Z cudagraph partition due to non gpu ops 2025-12-04T08:53:16.0979864Z cudagraph partition due to non gpu ops 2025-12-04T08:53:16.0980066Z cudagraph partition due to non gpu ops 2025-12-04T08:53:16.0980275Z cudagraph partition due to non gpu ops 2025-12-04T08:53:16.0980533Z cudagraph partition due to non gpu ops 2025-12-04T08:53:16.0980733Z cudagraph partition due to non gpu ops 2025-12-04T08:53:16.0980946Z cudagraph partition due to non gpu ops 2025-12-04T08:53:16.0981155Z cudagraph partition due to non gpu ops 2025-12-04T08:53:16.0981394Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.0981776Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.0982122Z res = mod(**inputs) 2025-12-04T08:53:16.0982499Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.0982909Z outputs = self.model( 2025-12-04T08:53:16.0983284Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T08:53:16.0983671Z encoder_outputs = self.encoder( 2025-12-04T08:53:16.0984053Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 849, in forward 2025-12-04T08:53:16.0984490Z embed_pos = self.embed_positions(input_ids, inputs_embeds) 2025-12-04T08:53:16.0984908Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context 2025-12-04T08:53:16.0985287Z return func(*args, **kwargs) 2025-12-04T08:53:16.0985666Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 149, in forward 2025-12-04T08:53:16.0986209Z position_ids = create_position_ids_from_input_ids(input_ids, self.padding_idx, past_key_values_length).to( 2025-12-04T08:53:16.0986791Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 82, in create_position_ids_from_input_ids 2025-12-04T08:53:16.0987348Z incremental_indices = (torch.cumsum(mask, dim=1).type_as(mask) + past_key_values_length) * mask 2025-12-04T08:53:16.0987590Z 2025-12-04T08:53:16.0987699Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.0988071Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.0988406Z res = mod(**inputs) 2025-12-04T08:53:16.0988766Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.0989149Z outputs = self.model( 2025-12-04T08:53:16.0989511Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T08:53:16.0989899Z encoder_outputs = self.encoder( 2025-12-04T08:53:16.0990274Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 849, in forward 2025-12-04T08:53:16.0990701Z embed_pos = self.embed_positions(input_ids, inputs_embeds) 2025-12-04T08:53:16.0991114Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context 2025-12-04T08:53:16.0991487Z return func(*args, **kwargs) 2025-12-04T08:53:16.0991869Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 149, in forward 2025-12-04T08:53:16.0992397Z position_ids = create_position_ids_from_input_ids(input_ids, self.padding_idx, past_key_values_length).to( 2025-12-04T08:53:16.0992984Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 82, in create_position_ids_from_input_ids 2025-12-04T08:53:16.0993552Z incremental_indices = (torch.cumsum(mask, dim=1).type_as(mask) + past_key_values_length) * mask 2025-12-04T08:53:16.0993822Z 2025-12-04T08:53:16.0993930Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.0994296Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.0994668Z res = mod(**inputs) 2025-12-04T08:53:16.0995032Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.0995415Z outputs = self.model( 2025-12-04T08:53:16.0995789Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T08:53:16.0996185Z encoder_outputs = self.encoder( 2025-12-04T08:53:16.0996566Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T08:53:16.0996979Z layer_outputs = encoder_layer( 2025-12-04T08:53:16.0997347Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.0997727Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.0998125Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T08:53:16.0998697Z hidden_states, attn_weights = self.self_attn( 2025-12-04T08:53:16.0999158Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.0999583Z return func(*args, **kwargs) 2025-12-04T08:53:16.0999970Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-12-04T08:53:16.1000451Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T08:53:16.1000669Z 2025-12-04T08:53:16.1000788Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1001164Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1001492Z res = mod(**inputs) 2025-12-04T08:53:16.1001864Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1002261Z outputs = self.model( 2025-12-04T08:53:16.1002635Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T08:53:16.1003073Z encoder_outputs = self.encoder( 2025-12-04T08:53:16.1003500Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T08:53:16.1003906Z layer_outputs = encoder_layer( 2025-12-04T08:53:16.1004303Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1005164Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1005606Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T08:53:16.1006051Z hidden_states, attn_weights = self.self_attn( 2025-12-04T08:53:16.1006485Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1006866Z return func(*args, **kwargs) 2025-12-04T08:53:16.1007236Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-12-04T08:53:16.1007638Z key_states = self.k_proj(current_states) 2025-12-04T08:53:16.1007783Z 2025-12-04T08:53:16.1007888Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1008251Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1008571Z res = mod(**inputs) 2025-12-04T08:53:16.1008984Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1009371Z outputs = self.model( 2025-12-04T08:53:16.1009745Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T08:53:16.1010167Z encoder_outputs = self.encoder( 2025-12-04T08:53:16.1010568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T08:53:16.1010984Z layer_outputs = encoder_layer( 2025-12-04T08:53:16.1011368Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1011761Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1012184Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T08:53:16.1012605Z hidden_states, attn_weights = self.self_attn( 2025-12-04T08:53:16.1013008Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1013401Z return func(*args, **kwargs) 2025-12-04T08:53:16.1013789Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 304, in forward 2025-12-04T08:53:16.1014194Z value_states = self.v_proj(current_states) 2025-12-04T08:53:16.1014347Z 2025-12-04T08:53:16.1014430Z cudagraph partition due to non gpu ops 2025-12-04T08:53:16.1014678Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1015047Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1015374Z res = mod(**inputs) 2025-12-04T08:53:16.1015758Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1016174Z outputs = self.model( 2025-12-04T08:53:16.1016550Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T08:53:16.1016947Z encoder_outputs = self.encoder( 2025-12-04T08:53:16.1017340Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T08:53:16.1017743Z layer_outputs = encoder_layer( 2025-12-04T08:53:16.1018098Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1018476Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1018900Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T08:53:16.1019315Z hidden_states, attn_weights = self.self_attn( 2025-12-04T08:53:16.1019717Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1020108Z return func(*args, **kwargs) 2025-12-04T08:53:16.1020496Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 322, in forward 2025-12-04T08:53:16.1021236Z attn_output, attn_weights = attention_interface( 2025-12-04T08:53:16.1021713Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T08:53:16.1022225Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T08:53:16.1022421Z 2025-12-04T08:53:16.1022541Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1022915Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1023267Z res = mod(**inputs) 2025-12-04T08:53:16.1023733Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1024141Z outputs = self.model( 2025-12-04T08:53:16.1024516Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T08:53:16.1024946Z encoder_outputs = self.encoder( 2025-12-04T08:53:16.1025426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T08:53:16.1025846Z layer_outputs = encoder_layer( 2025-12-04T08:53:16.1026205Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1026590Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1027034Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T08:53:16.1027450Z hidden_states, attn_weights = self.self_attn( 2025-12-04T08:53:16.1027877Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1028269Z return func(*args, **kwargs) 2025-12-04T08:53:16.1028662Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 336, in forward 2025-12-04T08:53:16.1029071Z attn_output = self.out_proj(attn_output) 2025-12-04T08:53:16.1029222Z 2025-12-04T08:53:16.1029329Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1029705Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1030031Z res = mod(**inputs) 2025-12-04T08:53:16.1030413Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1030811Z outputs = self.model( 2025-12-04T08:53:16.1031191Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T08:53:16.1031588Z encoder_outputs = self.encoder( 2025-12-04T08:53:16.1031983Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T08:53:16.1032380Z layer_outputs = encoder_layer( 2025-12-04T08:53:16.1032740Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1033109Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1033539Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 392, in forward 2025-12-04T08:53:16.1034014Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:53:16.1034205Z 2025-12-04T08:53:16.1034321Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1034718Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1035072Z res = mod(**inputs) 2025-12-04T08:53:16.1035470Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1035907Z outputs = self.model( 2025-12-04T08:53:16.1036308Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T08:53:16.1036736Z encoder_outputs = self.encoder( 2025-12-04T08:53:16.1037151Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T08:53:16.1037567Z layer_outputs = encoder_layer( 2025-12-04T08:53:16.1037950Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1038458Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1038893Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 392, in forward 2025-12-04T08:53:16.1039398Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:53:16.1039650Z 2025-12-04T08:53:16.1039766Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1040166Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1040513Z res = mod(**inputs) 2025-12-04T08:53:16.1040916Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1041344Z outputs = self.model( 2025-12-04T08:53:16.1041746Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T08:53:16.1042212Z encoder_outputs = self.encoder( 2025-12-04T08:53:16.1042613Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T08:53:16.1043005Z layer_outputs = encoder_layer( 2025-12-04T08:53:16.1043354Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1043736Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1044142Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 394, in forward 2025-12-04T08:53:16.1044550Z hidden_states = self.fc2(hidden_states) 2025-12-04T08:53:16.1044692Z 2025-12-04T08:53:16.1044800Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1045176Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1045520Z res = mod(**inputs) 2025-12-04T08:53:16.1045893Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1046292Z outputs = self.model( 2025-12-04T08:53:16.1046671Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T08:53:16.1047083Z encoder_outputs = self.encoder( 2025-12-04T08:53:16.1047483Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T08:53:16.1047881Z layer_outputs = encoder_layer( 2025-12-04T08:53:16.1048246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1048623Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1049025Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T08:53:16.1049444Z hidden_states, attn_weights = self.self_attn( 2025-12-04T08:53:16.1049861Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1050245Z return func(*args, **kwargs) 2025-12-04T08:53:16.1050632Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-12-04T08:53:16.1051119Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T08:53:16.1051331Z 2025-12-04T08:53:16.1051450Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1051825Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1052181Z res = mod(**inputs) 2025-12-04T08:53:16.1052584Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1053058Z outputs = self.model( 2025-12-04T08:53:16.1053455Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T08:53:16.1053898Z encoder_outputs = self.encoder( 2025-12-04T08:53:16.1054321Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T08:53:16.1054712Z layer_outputs = encoder_layer( 2025-12-04T08:53:16.1055083Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1055463Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1055865Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T08:53:16.1056301Z hidden_states, attn_weights = self.self_attn( 2025-12-04T08:53:16.1056739Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1057153Z return func(*args, **kwargs) 2025-12-04T08:53:16.1057556Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-12-04T08:53:16.1057990Z key_states = self.k_proj(current_states) 2025-12-04T08:53:16.1058143Z 2025-12-04T08:53:16.1058249Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1058621Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1058942Z res = mod(**inputs) 2025-12-04T08:53:16.1059331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1059754Z outputs = self.model( 2025-12-04T08:53:16.1060161Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T08:53:16.1060579Z encoder_outputs = self.encoder( 2025-12-04T08:53:16.1061000Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T08:53:16.1061399Z layer_outputs = encoder_layer( 2025-12-04T08:53:16.1061760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1062130Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1062531Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T08:53:16.1062960Z hidden_states, attn_weights = self.self_attn( 2025-12-04T08:53:16.1063379Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1063772Z return func(*args, **kwargs) 2025-12-04T08:53:16.1064166Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 304, in forward 2025-12-04T08:53:16.1064577Z value_states = self.v_proj(current_states) 2025-12-04T08:53:16.1064725Z 2025-12-04T08:53:16.1064809Z cudagraph partition due to non gpu ops 2025-12-04T08:53:16.1065056Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1065435Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1065763Z res = mod(**inputs) 2025-12-04T08:53:16.1066146Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1066545Z outputs = self.model( 2025-12-04T08:53:16.1066922Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T08:53:16.1067319Z encoder_outputs = self.encoder( 2025-12-04T08:53:16.1067741Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T08:53:16.1068149Z layer_outputs = encoder_layer( 2025-12-04T08:53:16.1068530Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1068904Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1069306Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T08:53:16.1069722Z hidden_states, attn_weights = self.self_attn( 2025-12-04T08:53:16.1070133Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1070521Z return func(*args, **kwargs) 2025-12-04T08:53:16.1070933Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 322, in forward 2025-12-04T08:53:16.1071368Z attn_output, attn_weights = attention_interface( 2025-12-04T08:53:16.1071838Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T08:53:16.1072344Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T08:53:16.1072537Z 2025-12-04T08:53:16.1072650Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1073025Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1073353Z res = mod(**inputs) 2025-12-04T08:53:16.1073786Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1074221Z outputs = self.model( 2025-12-04T08:53:16.1074624Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T08:53:16.1075055Z encoder_outputs = self.encoder( 2025-12-04T08:53:16.1075476Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T08:53:16.1075906Z layer_outputs = encoder_layer( 2025-12-04T08:53:16.1076283Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1076828Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1077267Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T08:53:16.1077716Z hidden_states, attn_weights = self.self_attn( 2025-12-04T08:53:16.1078168Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1078651Z return func(*args, **kwargs) 2025-12-04T08:53:16.1079073Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 336, in forward 2025-12-04T08:53:16.1079510Z attn_output = self.out_proj(attn_output) 2025-12-04T08:53:16.1079674Z 2025-12-04T08:53:16.1079795Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1080208Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1080565Z res = mod(**inputs) 2025-12-04T08:53:16.1080958Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1081385Z outputs = self.model( 2025-12-04T08:53:16.1081797Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T08:53:16.1082226Z encoder_outputs = self.encoder( 2025-12-04T08:53:16.1083093Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T08:53:16.1083517Z layer_outputs = encoder_layer( 2025-12-04T08:53:16.1083901Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1084313Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1084744Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 392, in forward 2025-12-04T08:53:16.1085224Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:53:16.1085412Z 2025-12-04T08:53:16.1085533Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1085918Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1086270Z res = mod(**inputs) 2025-12-04T08:53:16.1086689Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1087103Z outputs = self.model( 2025-12-04T08:53:16.1087510Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T08:53:16.1087937Z encoder_outputs = self.encoder( 2025-12-04T08:53:16.1088356Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T08:53:16.1088773Z layer_outputs = encoder_layer( 2025-12-04T08:53:16.1089156Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1089561Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1089985Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 392, in forward 2025-12-04T08:53:16.1090473Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:53:16.1090671Z 2025-12-04T08:53:16.1090783Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1091177Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1091530Z res = mod(**inputs) 2025-12-04T08:53:16.1091915Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1092336Z outputs = self.model( 2025-12-04T08:53:16.1092738Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T08:53:16.1093165Z encoder_outputs = self.encoder( 2025-12-04T08:53:16.1093581Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T08:53:16.1094000Z layer_outputs = encoder_layer( 2025-12-04T08:53:16.1094383Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1094772Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1095198Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 394, in forward 2025-12-04T08:53:16.1095607Z hidden_states = self.fc2(hidden_states) 2025-12-04T08:53:16.1095750Z 2025-12-04T08:53:16.1095862Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1096222Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1096553Z res = mod(**inputs) 2025-12-04T08:53:16.1096929Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1097320Z outputs = self.model( 2025-12-04T08:53:16.1097731Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T08:53:16.1098133Z encoder_outputs = self.encoder( 2025-12-04T08:53:16.1098528Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T08:53:16.1098938Z layer_outputs = encoder_layer( 2025-12-04T08:53:16.1099306Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1099708Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1100128Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 396, in forward 2025-12-04T08:53:16.1100541Z hidden_states = residual + hidden_states 2025-12-04T08:53:16.1100690Z 2025-12-04T08:53:16.1100797Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1101183Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1101517Z res = mod(**inputs) 2025-12-04T08:53:16.1101892Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1102293Z outputs = self.model( 2025-12-04T08:53:16.1102657Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T08:53:16.1103062Z encoder_outputs = self.encoder( 2025-12-04T08:53:16.1103453Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T08:53:16.1103854Z layer_outputs = encoder_layer( 2025-12-04T08:53:16.1104207Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1104586Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1104992Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T08:53:16.1105414Z hidden_states, attn_weights = self.self_attn( 2025-12-04T08:53:16.1105819Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1106210Z return func(*args, **kwargs) 2025-12-04T08:53:16.1106597Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-12-04T08:53:16.1107072Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T08:53:16.1107292Z 2025-12-04T08:53:16.1107398Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1107767Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1108098Z res = mod(**inputs) 2025-12-04T08:53:16.1108469Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1108865Z outputs = self.model( 2025-12-04T08:53:16.1109240Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T08:53:16.1109644Z encoder_outputs = self.encoder( 2025-12-04T08:53:16.1110038Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T08:53:16.1110445Z layer_outputs = encoder_layer( 2025-12-04T08:53:16.1110793Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1111156Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1111543Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T08:53:16.1111959Z hidden_states, attn_weights = self.self_attn( 2025-12-04T08:53:16.1112413Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1112798Z return func(*args, **kwargs) 2025-12-04T08:53:16.1113203Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-12-04T08:53:16.1113614Z key_states = self.k_proj(current_states) 2025-12-04T08:53:16.1113753Z 2025-12-04T08:53:16.1113860Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1114247Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1114611Z res = mod(**inputs) 2025-12-04T08:53:16.1115003Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1115439Z outputs = self.model( 2025-12-04T08:53:16.1115844Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T08:53:16.1116275Z encoder_outputs = self.encoder( 2025-12-04T08:53:16.1116687Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T08:53:16.1117101Z layer_outputs = encoder_layer( 2025-12-04T08:53:16.1117482Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1117883Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1118306Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T08:53:16.1118857Z hidden_states, attn_weights = self.self_attn( 2025-12-04T08:53:16.1119310Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1119744Z return func(*args, **kwargs) 2025-12-04T08:53:16.1120159Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 304, in forward 2025-12-04T08:53:16.1120611Z value_states = self.v_proj(current_states) 2025-12-04T08:53:16.1120958Z 2025-12-04T08:53:16.1121057Z cudagraph partition due to non gpu ops 2025-12-04T08:53:16.1121304Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1121678Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1122027Z res = mod(**inputs) 2025-12-04T08:53:16.1122396Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1122779Z outputs = self.model( 2025-12-04T08:53:16.1123155Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T08:53:16.1123550Z encoder_outputs = self.encoder( 2025-12-04T08:53:16.1123940Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T08:53:16.1124333Z layer_outputs = encoder_layer( 2025-12-04T08:53:16.1124694Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1125080Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1125461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T08:53:16.1125866Z hidden_states, attn_weights = self.self_attn( 2025-12-04T08:53:16.1126263Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1126643Z return func(*args, **kwargs) 2025-12-04T08:53:16.1127079Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 322, in forward 2025-12-04T08:53:16.1127497Z attn_output, attn_weights = attention_interface( 2025-12-04T08:53:16.1127950Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T08:53:16.1128464Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T08:53:16.1128668Z 2025-12-04T08:53:16.1128776Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1129145Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1129476Z res = mod(**inputs) 2025-12-04T08:53:16.1129843Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1130259Z outputs = self.model( 2025-12-04T08:53:16.1130631Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T08:53:16.1131024Z encoder_outputs = self.encoder( 2025-12-04T08:53:16.1131403Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T08:53:16.1131799Z layer_outputs = encoder_layer( 2025-12-04T08:53:16.1132143Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1132494Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1132875Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T08:53:16.1133274Z hidden_states, attn_weights = self.self_attn( 2025-12-04T08:53:16.1133661Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1134028Z return func(*args, **kwargs) 2025-12-04T08:53:16.1134398Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 336, in forward 2025-12-04T08:53:16.1134798Z attn_output = self.out_proj(attn_output) 2025-12-04T08:53:16.1134937Z 2025-12-04T08:53:16.1135046Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1135412Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1135742Z res = mod(**inputs) 2025-12-04T08:53:16.1136105Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1136489Z outputs = self.model( 2025-12-04T08:53:16.1136857Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T08:53:16.1137250Z encoder_outputs = self.encoder( 2025-12-04T08:53:16.1137634Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T08:53:16.1138012Z layer_outputs = encoder_layer( 2025-12-04T08:53:16.1138363Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1138729Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1139112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 392, in forward 2025-12-04T08:53:16.1139543Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:53:16.1139721Z 2025-12-04T08:53:16.1139825Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1140186Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1140506Z res = mod(**inputs) 2025-12-04T08:53:16.1140912Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1141317Z outputs = self.model( 2025-12-04T08:53:16.1141703Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T08:53:16.1142113Z encoder_outputs = self.encoder( 2025-12-04T08:53:16.1142510Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T08:53:16.1142910Z layer_outputs = encoder_layer( 2025-12-04T08:53:16.1143268Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1143640Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1144042Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 392, in forward 2025-12-04T08:53:16.1144512Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:53:16.1144689Z 2025-12-04T08:53:16.1144797Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1145165Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1145502Z res = mod(**inputs) 2025-12-04T08:53:16.1145872Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1146274Z outputs = self.model( 2025-12-04T08:53:16.1146655Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T08:53:16.1147059Z encoder_outputs = self.encoder( 2025-12-04T08:53:16.1147445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T08:53:16.1147848Z layer_outputs = encoder_layer( 2025-12-04T08:53:16.1148213Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1148590Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1148987Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 394, in forward 2025-12-04T08:53:16.1149400Z hidden_states = self.fc2(hidden_states) 2025-12-04T08:53:16.1149542Z 2025-12-04T08:53:16.1149659Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1150022Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1150356Z res = mod(**inputs) 2025-12-04T08:53:16.1150731Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1151133Z outputs = self.model( 2025-12-04T08:53:16.1151507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T08:53:16.1151915Z encoder_outputs = self.encoder( 2025-12-04T08:53:16.1152307Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T08:53:16.1152700Z layer_outputs = encoder_layer( 2025-12-04T08:53:16.1153062Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1153438Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1153841Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T08:53:16.1154259Z hidden_states, attn_weights = self.self_attn( 2025-12-04T08:53:16.1154687Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1155128Z return func(*args, **kwargs) 2025-12-04T08:53:16.1155536Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-12-04T08:53:16.1156034Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T08:53:16.1156284Z 2025-12-04T08:53:16.1156399Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1156794Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1157138Z res = mod(**inputs) 2025-12-04T08:53:16.1157540Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1157961Z outputs = self.model( 2025-12-04T08:53:16.1158422Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T08:53:16.1158901Z encoder_outputs = self.encoder( 2025-12-04T08:53:16.1159385Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T08:53:16.1159811Z layer_outputs = encoder_layer( 2025-12-04T08:53:16.1160201Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1160596Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1161018Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T08:53:16.1161447Z hidden_states, attn_weights = self.self_attn( 2025-12-04T08:53:16.1161851Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1162246Z return func(*args, **kwargs) 2025-12-04T08:53:16.1162640Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-12-04T08:53:16.1163053Z key_states = self.k_proj(current_states) 2025-12-04T08:53:16.1163196Z 2025-12-04T08:53:16.1163303Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1163679Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1164017Z res = mod(**inputs) 2025-12-04T08:53:16.1164383Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1164771Z outputs = self.model( 2025-12-04T08:53:16.1165144Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T08:53:16.1165539Z encoder_outputs = self.encoder( 2025-12-04T08:53:16.1165918Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T08:53:16.1166310Z layer_outputs = encoder_layer( 2025-12-04T08:53:16.1166664Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1167038Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1167424Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T08:53:16.1167843Z hidden_states, attn_weights = self.self_attn( 2025-12-04T08:53:16.1168266Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1168670Z return func(*args, **kwargs) 2025-12-04T08:53:16.1169047Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 304, in forward 2025-12-04T08:53:16.1169478Z value_states = self.v_proj(current_states) 2025-12-04T08:53:16.1169630Z 2025-12-04T08:53:16.1169750Z cudagraph partition due to non gpu ops 2025-12-04T08:53:16.1169987Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1170349Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1170695Z res = mod(**inputs) 2025-12-04T08:53:16.1171062Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1171446Z outputs = self.model( 2025-12-04T08:53:16.1171817Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T08:53:16.1172208Z encoder_outputs = self.encoder( 2025-12-04T08:53:16.1172587Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T08:53:16.1172996Z layer_outputs = encoder_layer( 2025-12-04T08:53:16.1173353Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1173722Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1174108Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T08:53:16.1174521Z hidden_states, attn_weights = self.self_attn( 2025-12-04T08:53:16.1174918Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1175294Z return func(*args, **kwargs) 2025-12-04T08:53:16.1175667Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 322, in forward 2025-12-04T08:53:16.1176083Z attn_output, attn_weights = attention_interface( 2025-12-04T08:53:16.1176540Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T08:53:16.1177018Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T08:53:16.1177212Z 2025-12-04T08:53:16.1177317Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1177679Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1178001Z res = mod(**inputs) 2025-12-04T08:53:16.1178360Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1178746Z outputs = self.model( 2025-12-04T08:53:16.1179117Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T08:53:16.1179505Z encoder_outputs = self.encoder( 2025-12-04T08:53:16.1179891Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T08:53:16.1180283Z layer_outputs = encoder_layer( 2025-12-04T08:53:16.1180635Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1181004Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1181414Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T08:53:16.1181828Z hidden_states, attn_weights = self.self_attn( 2025-12-04T08:53:16.1182228Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1182601Z return func(*args, **kwargs) 2025-12-04T08:53:16.1182981Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 336, in forward 2025-12-04T08:53:16.1183388Z attn_output = self.out_proj(attn_output) 2025-12-04T08:53:16.1183527Z 2025-12-04T08:53:16.1183662Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1184027Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1184357Z res = mod(**inputs) 2025-12-04T08:53:16.1184743Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1185119Z outputs = self.model( 2025-12-04T08:53:16.1185482Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T08:53:16.1185873Z encoder_outputs = self.encoder( 2025-12-04T08:53:16.1186242Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T08:53:16.1186627Z layer_outputs = encoder_layer( 2025-12-04T08:53:16.1186999Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1187369Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1187900Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 392, in forward 2025-12-04T08:53:16.1188345Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:53:16.1188526Z 2025-12-04T08:53:16.1188629Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1188986Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1189297Z res = mod(**inputs) 2025-12-04T08:53:16.1189654Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1190036Z outputs = self.model( 2025-12-04T08:53:16.1190392Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T08:53:16.1190782Z encoder_outputs = self.encoder( 2025-12-04T08:53:16.1191167Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T08:53:16.1191559Z layer_outputs = encoder_layer( 2025-12-04T08:53:16.1191902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1192271Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1192661Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 392, in forward 2025-12-04T08:53:16.1193096Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:53:16.1193266Z 2025-12-04T08:53:16.1193380Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1193729Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1194046Z res = mod(**inputs) 2025-12-04T08:53:16.1194394Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1194781Z outputs = self.model( 2025-12-04T08:53:16.1195153Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T08:53:16.1195544Z encoder_outputs = self.encoder( 2025-12-04T08:53:16.1195916Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T08:53:16.1196302Z layer_outputs = encoder_layer( 2025-12-04T08:53:16.1196652Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1197008Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1197439Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 394, in forward 2025-12-04T08:53:16.1197837Z hidden_states = self.fc2(hidden_states) 2025-12-04T08:53:16.1197977Z 2025-12-04T08:53:16.1198089Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1198556Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1198927Z res = mod(**inputs) 2025-12-04T08:53:16.1199334Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1199770Z outputs = self.model( 2025-12-04T08:53:16.1200163Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T08:53:16.1200558Z encoder_outputs = self.encoder( 2025-12-04T08:53:16.1200971Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T08:53:16.1201354Z layer_outputs = encoder_layer( 2025-12-04T08:53:16.1201721Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1202100Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1202503Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 396, in forward 2025-12-04T08:53:16.1202910Z hidden_states = residual + hidden_states 2025-12-04T08:53:16.1203054Z 2025-12-04T08:53:16.1203159Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1203523Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1203841Z res = mod(**inputs) 2025-12-04T08:53:16.1204206Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1204595Z outputs = self.model( 2025-12-04T08:53:16.1204966Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T08:53:16.1205347Z encoder_outputs = self.encoder( 2025-12-04T08:53:16.1205734Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T08:53:16.1206121Z layer_outputs = encoder_layer( 2025-12-04T08:53:16.1206476Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1206832Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1207221Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T08:53:16.1207631Z hidden_states, attn_weights = self.self_attn( 2025-12-04T08:53:16.1208031Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1208417Z return func(*args, **kwargs) 2025-12-04T08:53:16.1208805Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-12-04T08:53:16.1209285Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T08:53:16.1209496Z 2025-12-04T08:53:16.1209602Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1209974Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1210319Z res = mod(**inputs) 2025-12-04T08:53:16.1210692Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1211091Z outputs = self.model( 2025-12-04T08:53:16.1211511Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T08:53:16.1211920Z encoder_outputs = self.encoder( 2025-12-04T08:53:16.1212293Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T08:53:16.1212715Z layer_outputs = encoder_layer( 2025-12-04T08:53:16.1213071Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1213438Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1213829Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T08:53:16.1214248Z hidden_states, attn_weights = self.self_attn( 2025-12-04T08:53:16.1214654Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1215059Z return func(*args, **kwargs) 2025-12-04T08:53:16.1215439Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-12-04T08:53:16.1215832Z key_states = self.k_proj(current_states) 2025-12-04T08:53:16.1215972Z 2025-12-04T08:53:16.1216083Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1216434Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1216756Z res = mod(**inputs) 2025-12-04T08:53:16.1217119Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1217519Z outputs = self.model( 2025-12-04T08:53:16.1217887Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T08:53:16.1218292Z encoder_outputs = self.encoder( 2025-12-04T08:53:16.1218677Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T08:53:16.1219059Z layer_outputs = encoder_layer( 2025-12-04T08:53:16.1219412Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1219782Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1220170Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T08:53:16.1220578Z hidden_states, attn_weights = self.self_attn( 2025-12-04T08:53:16.1221107Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1221505Z return func(*args, **kwargs) 2025-12-04T08:53:16.1221891Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 304, in forward 2025-12-04T08:53:16.1222313Z value_states = self.v_proj(current_states) 2025-12-04T08:53:16.1222475Z 2025-12-04T08:53:16.1222558Z cudagraph partition due to non gpu ops 2025-12-04T08:53:16.1222803Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1223160Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1223490Z res = mod(**inputs) 2025-12-04T08:53:16.1223866Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1224264Z outputs = self.model( 2025-12-04T08:53:16.1224647Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T08:53:16.1225062Z encoder_outputs = self.encoder( 2025-12-04T08:53:16.1225469Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T08:53:16.1225977Z layer_outputs = encoder_layer( 2025-12-04T08:53:16.1226343Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1226715Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1227145Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T08:53:16.1227562Z hidden_states, attn_weights = self.self_attn( 2025-12-04T08:53:16.1227974Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1228365Z return func(*args, **kwargs) 2025-12-04T08:53:16.1228762Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 322, in forward 2025-12-04T08:53:16.1229216Z attn_output, attn_weights = attention_interface( 2025-12-04T08:53:16.1229675Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T08:53:16.1230175Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T08:53:16.1230370Z 2025-12-04T08:53:16.1230480Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1230853Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1231181Z res = mod(**inputs) 2025-12-04T08:53:16.1231559Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1231984Z outputs = self.model( 2025-12-04T08:53:16.1232381Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T08:53:16.1232801Z encoder_outputs = self.encoder( 2025-12-04T08:53:16.1233217Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T08:53:16.1233637Z layer_outputs = encoder_layer( 2025-12-04T08:53:16.1234013Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1234410Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1234832Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T08:53:16.1235268Z hidden_states, attn_weights = self.self_attn( 2025-12-04T08:53:16.1235691Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1236100Z return func(*args, **kwargs) 2025-12-04T08:53:16.1236512Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 336, in forward 2025-12-04T08:53:16.1236942Z attn_output = self.out_proj(attn_output) 2025-12-04T08:53:16.1237092Z 2025-12-04T08:53:16.1237204Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1237596Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1237951Z res = mod(**inputs) 2025-12-04T08:53:16.1238336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1238826Z outputs = self.model( 2025-12-04T08:53:16.1239235Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T08:53:16.1239684Z encoder_outputs = self.encoder( 2025-12-04T08:53:16.1240131Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T08:53:16.1240574Z layer_outputs = encoder_layer( 2025-12-04T08:53:16.1241009Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1241383Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1241793Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 392, in forward 2025-12-04T08:53:16.1242266Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:53:16.1242453Z 2025-12-04T08:53:16.1242571Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1242940Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1243278Z res = mod(**inputs) 2025-12-04T08:53:16.1243657Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1244075Z outputs = self.model( 2025-12-04T08:53:16.1244452Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T08:53:16.1244890Z encoder_outputs = self.encoder( 2025-12-04T08:53:16.1245286Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T08:53:16.1245684Z layer_outputs = encoder_layer( 2025-12-04T08:53:16.1246044Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1246416Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1246816Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 392, in forward 2025-12-04T08:53:16.1247252Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:53:16.1247434Z 2025-12-04T08:53:16.1247544Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1247927Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1248250Z res = mod(**inputs) 2025-12-04T08:53:16.1248624Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1249032Z outputs = self.model( 2025-12-04T08:53:16.1249411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T08:53:16.1249820Z encoder_outputs = self.encoder( 2025-12-04T08:53:16.1250214Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T08:53:16.1250608Z layer_outputs = encoder_layer( 2025-12-04T08:53:16.1250960Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1251322Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1251717Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 394, in forward 2025-12-04T08:53:16.1252118Z hidden_states = self.fc2(hidden_states) 2025-12-04T08:53:16.1252254Z 2025-12-04T08:53:16.1252361Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1252716Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1253037Z res = mod(**inputs) 2025-12-04T08:53:16.1253396Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1253772Z outputs = self.model( 2025-12-04T08:53:16.1254140Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T08:53:16.1254532Z encoder_outputs = self.encoder( 2025-12-04T08:53:16.1254952Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T08:53:16.1255348Z layer_outputs = encoder_layer( 2025-12-04T08:53:16.1255700Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1256081Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1256464Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T08:53:16.1256873Z hidden_states, attn_weights = self.self_attn( 2025-12-04T08:53:16.1257272Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1257649Z return func(*args, **kwargs) 2025-12-04T08:53:16.1258018Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-12-04T08:53:16.1258509Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T08:53:16.1258713Z 2025-12-04T08:53:16.1258826Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1259253Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1259590Z res = mod(**inputs) 2025-12-04T08:53:16.1259957Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1260349Z outputs = self.model( 2025-12-04T08:53:16.1260713Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T08:53:16.1261152Z encoder_outputs = self.encoder( 2025-12-04T08:53:16.1261543Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T08:53:16.1261938Z layer_outputs = encoder_layer( 2025-12-04T08:53:16.1262292Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1262662Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1263053Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T08:53:16.1263463Z hidden_states, attn_weights = self.self_attn( 2025-12-04T08:53:16.1263854Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1264222Z return func(*args, **kwargs) 2025-12-04T08:53:16.1264588Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-12-04T08:53:16.1264967Z key_states = self.k_proj(current_states) 2025-12-04T08:53:16.1265109Z 2025-12-04T08:53:16.1265210Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1265567Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1265878Z res = mod(**inputs) 2025-12-04T08:53:16.1266231Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1266611Z outputs = self.model( 2025-12-04T08:53:16.1266968Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T08:53:16.1267341Z encoder_outputs = self.encoder( 2025-12-04T08:53:16.1267714Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T08:53:16.1268094Z layer_outputs = encoder_layer( 2025-12-04T08:53:16.1268437Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1268833Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1269221Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T08:53:16.1269621Z hidden_states, attn_weights = self.self_attn( 2025-12-04T08:53:16.1270051Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1270428Z return func(*args, **kwargs) 2025-12-04T08:53:16.1270798Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 304, in forward 2025-12-04T08:53:16.1271196Z value_states = self.v_proj(current_states) 2025-12-04T08:53:16.1271334Z 2025-12-04T08:53:16.1271413Z cudagraph partition due to non gpu ops 2025-12-04T08:53:16.1271648Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1272025Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1272336Z res = mod(**inputs) 2025-12-04T08:53:16.1272693Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1273075Z outputs = self.model( 2025-12-04T08:53:16.1273438Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T08:53:16.1273814Z encoder_outputs = self.encoder( 2025-12-04T08:53:16.1274187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T08:53:16.1274565Z layer_outputs = encoder_layer( 2025-12-04T08:53:16.1274903Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1275263Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1275652Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T08:53:16.1276061Z hidden_states, attn_weights = self.self_attn( 2025-12-04T08:53:16.1276454Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1276827Z return func(*args, **kwargs) 2025-12-04T08:53:16.1277197Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 322, in forward 2025-12-04T08:53:16.1277755Z attn_output, attn_weights = attention_interface( 2025-12-04T08:53:16.1278208Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T08:53:16.1278755Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T08:53:16.1278971Z 2025-12-04T08:53:16.1279097Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1279514Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1279868Z res = mod(**inputs) 2025-12-04T08:53:16.1280270Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1280699Z outputs = self.model( 2025-12-04T08:53:16.1281096Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T08:53:16.1283872Z encoder_outputs = self.encoder( 2025-12-04T08:53:16.1284300Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T08:53:16.1284708Z layer_outputs = encoder_layer( 2025-12-04T08:53:16.1285086Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1285488Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1285891Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T08:53:16.1286294Z hidden_states, attn_weights = self.self_attn( 2025-12-04T08:53:16.1286722Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1287107Z return func(*args, **kwargs) 2025-12-04T08:53:16.1287512Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 336, in forward 2025-12-04T08:53:16.1287913Z attn_output = self.out_proj(attn_output) 2025-12-04T08:53:16.1288057Z 2025-12-04T08:53:16.1288166Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1288542Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1288906Z res = mod(**inputs) 2025-12-04T08:53:16.1289276Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1289677Z outputs = self.model( 2025-12-04T08:53:16.1290055Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T08:53:16.1290469Z encoder_outputs = self.encoder( 2025-12-04T08:53:16.1290849Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T08:53:16.1291238Z layer_outputs = encoder_layer( 2025-12-04T08:53:16.1291595Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1291961Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1292347Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 392, in forward 2025-12-04T08:53:16.1292787Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:53:16.1292958Z 2025-12-04T08:53:16.1293070Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1293429Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1293757Z res = mod(**inputs) 2025-12-04T08:53:16.1294118Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1294509Z outputs = self.model( 2025-12-04T08:53:16.1294868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T08:53:16.1295270Z encoder_outputs = self.encoder( 2025-12-04T08:53:16.1295654Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T08:53:16.1296044Z layer_outputs = encoder_layer( 2025-12-04T08:53:16.1296389Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1296756Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1297149Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 392, in forward 2025-12-04T08:53:16.1297578Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:53:16.1297757Z 2025-12-04T08:53:16.1297919Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1298294Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1298617Z res = mod(**inputs) 2025-12-04T08:53:16.1298972Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1299359Z outputs = self.model( 2025-12-04T08:53:16.1299740Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T08:53:16.1300136Z encoder_outputs = self.encoder( 2025-12-04T08:53:16.1300534Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T08:53:16.1300908Z layer_outputs = encoder_layer( 2025-12-04T08:53:16.1301251Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1301615Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1301990Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 394, in forward 2025-12-04T08:53:16.1302374Z hidden_states = self.fc2(hidden_states) 2025-12-04T08:53:16.1302534Z 2025-12-04T08:53:16.1302637Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1302990Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1303301Z res = mod(**inputs) 2025-12-04T08:53:16.1303659Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1304038Z outputs = self.model( 2025-12-04T08:53:16.1304397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T08:53:16.1304770Z encoder_outputs = self.encoder( 2025-12-04T08:53:16.1305144Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T08:53:16.1305524Z layer_outputs = encoder_layer( 2025-12-04T08:53:16.1305858Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1306217Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1306595Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 396, in forward 2025-12-04T08:53:16.1306975Z hidden_states = residual + hidden_states 2025-12-04T08:53:16.1307110Z 2025-12-04T08:53:16.1307212Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1307558Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1307873Z res = mod(**inputs) 2025-12-04T08:53:16.1308221Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1308605Z outputs = self.model( 2025-12-04T08:53:16.1308958Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T08:53:16.1309338Z encoder_outputs = self.encoder( 2025-12-04T08:53:16.1309707Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T08:53:16.1310086Z layer_outputs = encoder_layer( 2025-12-04T08:53:16.1310432Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1310789Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1311168Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T08:53:16.1311616Z hidden_states, attn_weights = self.self_attn( 2025-12-04T08:53:16.1312020Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1312393Z return func(*args, **kwargs) 2025-12-04T08:53:16.1312772Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-12-04T08:53:16.1313258Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T08:53:16.1313471Z 2025-12-04T08:53:16.1313584Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1313958Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1314283Z res = mod(**inputs) 2025-12-04T08:53:16.1314648Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1315039Z outputs = self.model( 2025-12-04T08:53:16.1315401Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T08:53:16.1315795Z encoder_outputs = self.encoder( 2025-12-04T08:53:16.1316176Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T08:53:16.1316576Z layer_outputs = encoder_layer( 2025-12-04T08:53:16.1316928Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1317294Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1317705Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T08:53:16.1318119Z hidden_states, attn_weights = self.self_attn( 2025-12-04T08:53:16.1318621Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1319026Z return func(*args, **kwargs) 2025-12-04T08:53:16.1319409Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-12-04T08:53:16.1319823Z key_states = self.k_proj(current_states) 2025-12-04T08:53:16.1319973Z 2025-12-04T08:53:16.1320084Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1320458Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1321005Z res = mod(**inputs) 2025-12-04T08:53:16.1321382Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1321785Z outputs = self.model( 2025-12-04T08:53:16.1322162Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T08:53:16.1322551Z encoder_outputs = self.encoder( 2025-12-04T08:53:16.1322939Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T08:53:16.1323333Z layer_outputs = encoder_layer( 2025-12-04T08:53:16.1323688Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1324061Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1324464Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T08:53:16.1324882Z hidden_states, attn_weights = self.self_attn( 2025-12-04T08:53:16.1325283Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1325674Z return func(*args, **kwargs) 2025-12-04T08:53:16.1326119Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 304, in forward 2025-12-04T08:53:16.1326517Z value_states = self.v_proj(current_states) 2025-12-04T08:53:16.1326659Z 2025-12-04T08:53:16.1326739Z cudagraph partition due to non gpu ops 2025-12-04T08:53:16.1326976Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1327367Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1327686Z res = mod(**inputs) 2025-12-04T08:53:16.1328054Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1328476Z outputs = self.model( 2025-12-04T08:53:16.1328845Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T08:53:16.1329236Z encoder_outputs = self.encoder( 2025-12-04T08:53:16.1329624Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T08:53:16.1330016Z layer_outputs = encoder_layer( 2025-12-04T08:53:16.1330361Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1330756Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1331212Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T08:53:16.1331612Z hidden_states, attn_weights = self.self_attn( 2025-12-04T08:53:16.1332002Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1332388Z return func(*args, **kwargs) 2025-12-04T08:53:16.1332774Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 322, in forward 2025-12-04T08:53:16.1333182Z attn_output, attn_weights = attention_interface( 2025-12-04T08:53:16.1333643Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T08:53:16.1334143Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T08:53:16.1334329Z 2025-12-04T08:53:16.1334445Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1334799Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1335126Z res = mod(**inputs) 2025-12-04T08:53:16.1335491Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1335883Z outputs = self.model( 2025-12-04T08:53:16.1336248Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T08:53:16.1336647Z encoder_outputs = self.encoder( 2025-12-04T08:53:16.1337043Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T08:53:16.1337438Z layer_outputs = encoder_layer( 2025-12-04T08:53:16.1337805Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1338187Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1338596Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T08:53:16.1339023Z hidden_states, attn_weights = self.self_attn( 2025-12-04T08:53:16.1339436Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1339828Z return func(*args, **kwargs) 2025-12-04T08:53:16.1340235Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 336, in forward 2025-12-04T08:53:16.1340635Z attn_output = self.out_proj(attn_output) 2025-12-04T08:53:16.1340780Z 2025-12-04T08:53:16.1340885Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1341253Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1341585Z res = mod(**inputs) 2025-12-04T08:53:16.1341954Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1342337Z outputs = self.model( 2025-12-04T08:53:16.1342732Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T08:53:16.1343121Z encoder_outputs = self.encoder( 2025-12-04T08:53:16.1343505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T08:53:16.1343893Z layer_outputs = encoder_layer( 2025-12-04T08:53:16.1344238Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1344606Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1345016Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 392, in forward 2025-12-04T08:53:16.1345450Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:53:16.1345622Z 2025-12-04T08:53:16.1345726Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1346088Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1346413Z res = mod(**inputs) 2025-12-04T08:53:16.1346779Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1347158Z outputs = self.model( 2025-12-04T08:53:16.1347526Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T08:53:16.1347916Z encoder_outputs = self.encoder( 2025-12-04T08:53:16.1348312Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T08:53:16.1348704Z layer_outputs = encoder_layer( 2025-12-04T08:53:16.1349055Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1349424Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1349806Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 392, in forward 2025-12-04T08:53:16.1350250Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:53:16.1350421Z 2025-12-04T08:53:16.1350531Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1350887Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1351195Z res = mod(**inputs) 2025-12-04T08:53:16.1351548Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1351927Z outputs = self.model( 2025-12-04T08:53:16.1352278Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T08:53:16.1352660Z encoder_outputs = self.encoder( 2025-12-04T08:53:16.1353036Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T08:53:16.1353411Z layer_outputs = encoder_layer( 2025-12-04T08:53:16.1353768Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1354126Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1354506Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 394, in forward 2025-12-04T08:53:16.1354894Z hidden_states = self.fc2(hidden_states) 2025-12-04T08:53:16.1355037Z 2025-12-04T08:53:16.1355155Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1355520Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1355843Z res = mod(**inputs) 2025-12-04T08:53:16.1356217Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1356612Z outputs = self.model( 2025-12-04T08:53:16.1356983Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T08:53:16.1357377Z encoder_outputs = self.encoder( 2025-12-04T08:53:16.1357752Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T08:53:16.1358191Z layer_outputs = encoder_layer( 2025-12-04T08:53:16.1358659Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1359074Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1359540Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T08:53:16.1359999Z hidden_states, attn_weights = self.self_attn( 2025-12-04T08:53:16.1360413Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1360801Z return func(*args, **kwargs) 2025-12-04T08:53:16.1361184Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-12-04T08:53:16.1361658Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T08:53:16.1361869Z 2025-12-04T08:53:16.1361982Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1362343Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1362664Z res = mod(**inputs) 2025-12-04T08:53:16.1363029Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1363415Z outputs = self.model( 2025-12-04T08:53:16.1363785Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T08:53:16.1364181Z encoder_outputs = self.encoder( 2025-12-04T08:53:16.1364568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T08:53:16.1364953Z layer_outputs = encoder_layer( 2025-12-04T08:53:16.1365302Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1365676Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1366064Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T08:53:16.1366473Z hidden_states, attn_weights = self.self_attn( 2025-12-04T08:53:16.1366877Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1367255Z return func(*args, **kwargs) 2025-12-04T08:53:16.1367625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-12-04T08:53:16.1368052Z key_states = self.k_proj(current_states) 2025-12-04T08:53:16.1368191Z 2025-12-04T08:53:16.1368304Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1368668Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1368988Z res = mod(**inputs) 2025-12-04T08:53:16.1369368Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1369761Z outputs = self.model( 2025-12-04T08:53:16.1370127Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T08:53:16.1370542Z encoder_outputs = self.encoder( 2025-12-04T08:53:16.1370926Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T08:53:16.1371323Z layer_outputs = encoder_layer( 2025-12-04T08:53:16.1371678Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1372054Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1372449Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T08:53:16.1372882Z hidden_states, attn_weights = self.self_attn( 2025-12-04T08:53:16.1373294Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1373677Z return func(*args, **kwargs) 2025-12-04T08:53:16.1374054Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 304, in forward 2025-12-04T08:53:16.1374445Z value_states = self.v_proj(current_states) 2025-12-04T08:53:16.1374594Z 2025-12-04T08:53:16.1374676Z cudagraph partition due to non gpu ops 2025-12-04T08:53:16.1374924Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1375287Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1375608Z res = mod(**inputs) 2025-12-04T08:53:16.1375973Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1376362Z outputs = self.model( 2025-12-04T08:53:16.1376723Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T08:53:16.1377111Z encoder_outputs = self.encoder( 2025-12-04T08:53:16.1377495Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T08:53:16.1377879Z layer_outputs = encoder_layer( 2025-12-04T08:53:16.1378223Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1378586Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1378976Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T08:53:16.1379375Z hidden_states, attn_weights = self.self_attn( 2025-12-04T08:53:16.1379777Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1380154Z return func(*args, **kwargs) 2025-12-04T08:53:16.1380555Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 322, in forward 2025-12-04T08:53:16.1380972Z attn_output, attn_weights = attention_interface( 2025-12-04T08:53:16.1381437Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T08:53:16.1381930Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T08:53:16.1382110Z 2025-12-04T08:53:16.1382218Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1382565Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1382889Z res = mod(**inputs) 2025-12-04T08:53:16.1383263Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1383637Z outputs = self.model( 2025-12-04T08:53:16.1383998Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T08:53:16.1384413Z encoder_outputs = self.encoder( 2025-12-04T08:53:16.1384797Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T08:53:16.1385183Z layer_outputs = encoder_layer( 2025-12-04T08:53:16.1385530Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1385886Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1386267Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T08:53:16.1386713Z hidden_states, attn_weights = self.self_attn( 2025-12-04T08:53:16.1387116Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1387498Z return func(*args, **kwargs) 2025-12-04T08:53:16.1387882Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 336, in forward 2025-12-04T08:53:16.1388308Z attn_output = self.out_proj(attn_output) 2025-12-04T08:53:16.1388446Z 2025-12-04T08:53:16.1388557Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1388922Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1389242Z res = mod(**inputs) 2025-12-04T08:53:16.1389611Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1390005Z outputs = self.model( 2025-12-04T08:53:16.1390369Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T08:53:16.1390766Z encoder_outputs = self.encoder( 2025-12-04T08:53:16.1391157Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T08:53:16.1391548Z layer_outputs = encoder_layer( 2025-12-04T08:53:16.1391904Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1392276Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1392673Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 392, in forward 2025-12-04T08:53:16.1393113Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:53:16.1393289Z 2025-12-04T08:53:16.1393394Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1393761Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1394085Z res = mod(**inputs) 2025-12-04T08:53:16.1394456Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1394857Z outputs = self.model( 2025-12-04T08:53:16.1395238Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T08:53:16.1395643Z encoder_outputs = self.encoder( 2025-12-04T08:53:16.1396052Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T08:53:16.1396472Z layer_outputs = encoder_layer( 2025-12-04T08:53:16.1396853Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1397244Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1397698Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 392, in forward 2025-12-04T08:53:16.1398167Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:53:16.1398436Z 2025-12-04T08:53:16.1398568Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1398961Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1399314Z res = mod(**inputs) 2025-12-04T08:53:16.1399720Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1400146Z outputs = self.model( 2025-12-04T08:53:16.1400545Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T08:53:16.1401001Z encoder_outputs = self.encoder( 2025-12-04T08:53:16.1401429Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T08:53:16.1401826Z layer_outputs = encoder_layer( 2025-12-04T08:53:16.1402186Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1402562Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1402971Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 394, in forward 2025-12-04T08:53:16.1403379Z hidden_states = self.fc2(hidden_states) 2025-12-04T08:53:16.1403532Z 2025-12-04T08:53:16.1403640Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1404013Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1404354Z res = mod(**inputs) 2025-12-04T08:53:16.1404736Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1405134Z outputs = self.model( 2025-12-04T08:53:16.1405513Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T08:53:16.1405911Z encoder_outputs = self.encoder( 2025-12-04T08:53:16.1406307Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T08:53:16.1406705Z layer_outputs = encoder_layer( 2025-12-04T08:53:16.1407067Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1407438Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1407844Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 396, in forward 2025-12-04T08:53:16.1408258Z hidden_states = residual + hidden_states 2025-12-04T08:53:16.1408399Z 2025-12-04T08:53:16.1408506Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1408884Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1409221Z res = mod(**inputs) 2025-12-04T08:53:16.1409596Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1409988Z outputs = self.model( 2025-12-04T08:53:16.1410397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T08:53:16.1410805Z encoder_outputs = self.encoder( 2025-12-04T08:53:16.1411195Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T08:53:16.1411606Z layer_outputs = encoder_layer( 2025-12-04T08:53:16.1411987Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1412363Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1412758Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T08:53:16.1413201Z hidden_states, attn_weights = self.self_attn( 2025-12-04T08:53:16.1413616Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1414008Z return func(*args, **kwargs) 2025-12-04T08:53:16.1414391Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-12-04T08:53:16.1414872Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T08:53:16.1415102Z 2025-12-04T08:53:16.1415215Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1415588Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1415933Z res = mod(**inputs) 2025-12-04T08:53:16.1416297Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1416685Z outputs = self.model( 2025-12-04T08:53:16.1417053Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T08:53:16.1417454Z encoder_outputs = self.encoder( 2025-12-04T08:53:16.1417851Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T08:53:16.1418256Z layer_outputs = encoder_layer( 2025-12-04T08:53:16.1418614Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1418981Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1419375Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T08:53:16.1419777Z hidden_states, attn_weights = self.self_attn( 2025-12-04T08:53:16.1420178Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1420559Z return func(*args, **kwargs) 2025-12-04T08:53:16.1421167Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-12-04T08:53:16.1421580Z key_states = self.k_proj(current_states) 2025-12-04T08:53:16.1421734Z 2025-12-04T08:53:16.1421843Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1422225Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1422570Z res = mod(**inputs) 2025-12-04T08:53:16.1422950Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1423350Z outputs = self.model( 2025-12-04T08:53:16.1423730Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T08:53:16.1424128Z encoder_outputs = self.encoder( 2025-12-04T08:53:16.1424520Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T08:53:16.1424959Z layer_outputs = encoder_layer( 2025-12-04T08:53:16.1425314Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1425671Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1426072Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T08:53:16.1426522Z hidden_states, attn_weights = self.self_attn( 2025-12-04T08:53:16.1426918Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1427337Z return func(*args, **kwargs) 2025-12-04T08:53:16.1427717Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 304, in forward 2025-12-04T08:53:16.1428120Z value_states = self.v_proj(current_states) 2025-12-04T08:53:16.1428263Z 2025-12-04T08:53:16.1428347Z cudagraph partition due to non gpu ops 2025-12-04T08:53:16.1428591Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1428961Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1429282Z res = mod(**inputs) 2025-12-04T08:53:16.1429680Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1430069Z outputs = self.model( 2025-12-04T08:53:16.1430441Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T08:53:16.1430829Z encoder_outputs = self.encoder( 2025-12-04T08:53:16.1431216Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T08:53:16.1431604Z layer_outputs = encoder_layer( 2025-12-04T08:53:16.1431951Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1432319Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1432710Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T08:53:16.1433118Z hidden_states, attn_weights = self.self_attn( 2025-12-04T08:53:16.1433498Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1433866Z return func(*args, **kwargs) 2025-12-04T08:53:16.1434239Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 322, in forward 2025-12-04T08:53:16.1434656Z attn_output, attn_weights = attention_interface( 2025-12-04T08:53:16.1435103Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T08:53:16.1435598Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T08:53:16.1435788Z 2025-12-04T08:53:16.1435906Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1436299Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1436649Z res = mod(**inputs) 2025-12-04T08:53:16.1437045Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1437465Z outputs = self.model( 2025-12-04T08:53:16.1437861Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T08:53:16.1438294Z encoder_outputs = self.encoder( 2025-12-04T08:53:16.1438789Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T08:53:16.1439259Z layer_outputs = encoder_layer( 2025-12-04T08:53:16.1439654Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1440071Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1440478Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T08:53:16.1440920Z hidden_states, attn_weights = self.self_attn( 2025-12-04T08:53:16.1441339Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1441769Z return func(*args, **kwargs) 2025-12-04T08:53:16.1442174Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 336, in forward 2025-12-04T08:53:16.1442585Z attn_output = self.out_proj(attn_output) 2025-12-04T08:53:16.1442737Z 2025-12-04T08:53:16.1442846Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1443218Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1443554Z res = mod(**inputs) 2025-12-04T08:53:16.1443934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1444344Z outputs = self.model( 2025-12-04T08:53:16.1444726Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T08:53:16.1445119Z encoder_outputs = self.encoder( 2025-12-04T08:53:16.1445522Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T08:53:16.1445925Z layer_outputs = encoder_layer( 2025-12-04T08:53:16.1446291Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1446672Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1447100Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 392, in forward 2025-12-04T08:53:16.1447582Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:53:16.1447763Z 2025-12-04T08:53:16.1447874Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1448252Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1448593Z res = mod(**inputs) 2025-12-04T08:53:16.1448980Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1449368Z outputs = self.model( 2025-12-04T08:53:16.1449744Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T08:53:16.1450149Z encoder_outputs = self.encoder( 2025-12-04T08:53:16.1450543Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T08:53:16.1450939Z layer_outputs = encoder_layer( 2025-12-04T08:53:16.1451307Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1451688Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1452087Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 392, in forward 2025-12-04T08:53:16.1452542Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:53:16.1452727Z 2025-12-04T08:53:16.1452838Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1453213Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1453567Z res = mod(**inputs) 2025-12-04T08:53:16.1453946Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1454346Z outputs = self.model( 2025-12-04T08:53:16.1454716Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T08:53:16.1455142Z encoder_outputs = self.encoder( 2025-12-04T08:53:16.1455539Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T08:53:16.1455940Z layer_outputs = encoder_layer( 2025-12-04T08:53:16.1456332Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1456712Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1457123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 394, in forward 2025-12-04T08:53:16.1457540Z hidden_states = self.fc2(hidden_states) 2025-12-04T08:53:16.1457681Z 2025-12-04T08:53:16.1457787Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1458157Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1458514Z res = mod(**inputs) 2025-12-04T08:53:16.1458884Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1459290Z outputs = self.model( 2025-12-04T08:53:16.1459673Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T08:53:16.1460076Z encoder_outputs = self.encoder( 2025-12-04T08:53:16.1460461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T08:53:16.1460860Z layer_outputs = encoder_layer( 2025-12-04T08:53:16.1461221Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1461589Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1461989Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T08:53:16.1462416Z hidden_states, attn_weights = self.self_attn( 2025-12-04T08:53:16.1462823Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1463208Z return func(*args, **kwargs) 2025-12-04T08:53:16.1463598Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-12-04T08:53:16.1464081Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T08:53:16.1464286Z 2025-12-04T08:53:16.1464398Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1464753Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1465078Z res = mod(**inputs) 2025-12-04T08:53:16.1465444Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1465828Z outputs = self.model( 2025-12-04T08:53:16.1466210Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T08:53:16.1466598Z encoder_outputs = self.encoder( 2025-12-04T08:53:16.1466982Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T08:53:16.1467361Z layer_outputs = encoder_layer( 2025-12-04T08:53:16.1467731Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1468104Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1468500Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T08:53:16.1468902Z hidden_states, attn_weights = self.self_attn( 2025-12-04T08:53:16.1469318Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1469705Z return func(*args, **kwargs) 2025-12-04T08:53:16.1470077Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-12-04T08:53:16.1470493Z key_states = self.k_proj(current_states) 2025-12-04T08:53:16.1470640Z 2025-12-04T08:53:16.1470748Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1471122Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1471454Z res = mod(**inputs) 2025-12-04T08:53:16.1471829Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1472229Z outputs = self.model( 2025-12-04T08:53:16.1472618Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T08:53:16.1473015Z encoder_outputs = self.encoder( 2025-12-04T08:53:16.1473404Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T08:53:16.1473803Z layer_outputs = encoder_layer( 2025-12-04T08:53:16.1474156Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1474532Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1474933Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T08:53:16.1475347Z hidden_states, attn_weights = self.self_attn( 2025-12-04T08:53:16.1475765Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1476174Z return func(*args, **kwargs) 2025-12-04T08:53:16.1476583Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 304, in forward 2025-12-04T08:53:16.1477009Z value_states = self.v_proj(current_states) 2025-12-04T08:53:16.1477170Z 2025-12-04T08:53:16.1477261Z cudagraph partition due to non gpu ops 2025-12-04T08:53:16.1477521Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1477912Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1478253Z res = mod(**inputs) 2025-12-04T08:53:16.1478721Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1479149Z outputs = self.model( 2025-12-04T08:53:16.1479559Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T08:53:16.1480011Z encoder_outputs = self.encoder( 2025-12-04T08:53:16.1480431Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T08:53:16.1480858Z layer_outputs = encoder_layer( 2025-12-04T08:53:16.1481238Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1481640Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1482070Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T08:53:16.1482540Z hidden_states, attn_weights = self.self_attn( 2025-12-04T08:53:16.1482968Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1483382Z return func(*args, **kwargs) 2025-12-04T08:53:16.1483816Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 322, in forward 2025-12-04T08:53:16.1484258Z attn_output, attn_weights = attention_interface( 2025-12-04T08:53:16.1484745Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T08:53:16.1485294Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T08:53:16.1485498Z 2025-12-04T08:53:16.1485621Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1486017Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1486383Z res = mod(**inputs) 2025-12-04T08:53:16.1486794Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1487224Z outputs = self.model( 2025-12-04T08:53:16.1487634Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T08:53:16.1488055Z encoder_outputs = self.encoder( 2025-12-04T08:53:16.1488469Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T08:53:16.1488852Z layer_outputs = encoder_layer( 2025-12-04T08:53:16.1489202Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1489574Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1489966Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T08:53:16.1490362Z hidden_states, attn_weights = self.self_attn( 2025-12-04T08:53:16.1490757Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1491134Z return func(*args, **kwargs) 2025-12-04T08:53:16.1491501Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 336, in forward 2025-12-04T08:53:16.1491897Z attn_output = self.out_proj(attn_output) 2025-12-04T08:53:16.1492042Z 2025-12-04T08:53:16.1492149Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1492507Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1492827Z res = mod(**inputs) 2025-12-04T08:53:16.1493193Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1493579Z outputs = self.model( 2025-12-04T08:53:16.1493942Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T08:53:16.1494336Z encoder_outputs = self.encoder( 2025-12-04T08:53:16.1494722Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T08:53:16.1495108Z layer_outputs = encoder_layer( 2025-12-04T08:53:16.1495456Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1495822Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1496212Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 392, in forward 2025-12-04T08:53:16.1496663Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:53:16.1496838Z 2025-12-04T08:53:16.1496948Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1497317Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1497652Z res = mod(**inputs) 2025-12-04T08:53:16.1498067Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1498454Z outputs = self.model( 2025-12-04T08:53:16.1498823Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T08:53:16.1499231Z encoder_outputs = self.encoder( 2025-12-04T08:53:16.1499605Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T08:53:16.1499995Z layer_outputs = encoder_layer( 2025-12-04T08:53:16.1500349Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1500712Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1501097Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 392, in forward 2025-12-04T08:53:16.1501549Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:53:16.1501727Z 2025-12-04T08:53:16.1501842Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1502203Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1502537Z res = mod(**inputs) 2025-12-04T08:53:16.1502905Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1503300Z outputs = self.model( 2025-12-04T08:53:16.1503675Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T08:53:16.1504060Z encoder_outputs = self.encoder( 2025-12-04T08:53:16.1504438Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T08:53:16.1504818Z layer_outputs = encoder_layer( 2025-12-04T08:53:16.1505175Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1505540Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1505926Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 394, in forward 2025-12-04T08:53:16.1506311Z hidden_states = self.fc2(hidden_states) 2025-12-04T08:53:16.1506456Z 2025-12-04T08:53:16.1506559Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1506917Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1507246Z res = mod(**inputs) 2025-12-04T08:53:16.1507599Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1507982Z outputs = self.model( 2025-12-04T08:53:16.1508353Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T08:53:16.1508734Z encoder_outputs = self.encoder( 2025-12-04T08:53:16.1509115Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T08:53:16.1509504Z layer_outputs = encoder_layer( 2025-12-04T08:53:16.1509854Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1510212Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1510624Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 396, in forward 2025-12-04T08:53:16.1511018Z hidden_states = residual + hidden_states 2025-12-04T08:53:16.1511155Z 2025-12-04T08:53:16.1511266Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1511622Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1511959Z res = mod(**inputs) 2025-12-04T08:53:16.1512317Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1512693Z outputs = self.model( 2025-12-04T08:53:16.1513056Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T08:53:16.1513425Z encoder_outputs = self.encoder( 2025-12-04T08:53:16.1513794Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T08:53:16.1514156Z layer_outputs = encoder_layer( 2025-12-04T08:53:16.1514497Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1514852Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1515246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T08:53:16.1515640Z hidden_states, attn_weights = self.self_attn( 2025-12-04T08:53:16.1516028Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1516401Z return func(*args, **kwargs) 2025-12-04T08:53:16.1516763Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-12-04T08:53:16.1517220Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T08:53:16.1517420Z 2025-12-04T08:53:16.1517529Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1517883Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1518194Z res = mod(**inputs) 2025-12-04T08:53:16.1518612Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1519020Z outputs = self.model( 2025-12-04T08:53:16.1519422Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T08:53:16.1519869Z encoder_outputs = self.encoder( 2025-12-04T08:53:16.1520289Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T08:53:16.1520851Z layer_outputs = encoder_layer( 2025-12-04T08:53:16.1521208Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1521572Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1521959Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T08:53:16.1522381Z hidden_states, attn_weights = self.self_attn( 2025-12-04T08:53:16.1522801Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1523192Z return func(*args, **kwargs) 2025-12-04T08:53:16.1523584Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-12-04T08:53:16.1523995Z key_states = self.k_proj(current_states) 2025-12-04T08:53:16.1524140Z 2025-12-04T08:53:16.1524243Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1524671Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1525001Z res = mod(**inputs) 2025-12-04T08:53:16.1525364Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1525756Z outputs = self.model( 2025-12-04T08:53:16.1526151Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T08:53:16.1526538Z encoder_outputs = self.encoder( 2025-12-04T08:53:16.1526923Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T08:53:16.1527347Z layer_outputs = encoder_layer( 2025-12-04T08:53:16.1527716Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1528078Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1528469Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T08:53:16.1528867Z hidden_states, attn_weights = self.self_attn( 2025-12-04T08:53:16.1529244Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1529637Z return func(*args, **kwargs) 2025-12-04T08:53:16.1530011Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 304, in forward 2025-12-04T08:53:16.1530418Z value_states = self.v_proj(current_states) 2025-12-04T08:53:16.1530558Z 2025-12-04T08:53:16.1530639Z cudagraph partition due to non gpu ops 2025-12-04T08:53:16.1530877Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1531229Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1531549Z res = mod(**inputs) 2025-12-04T08:53:16.1531895Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1532274Z outputs = self.model( 2025-12-04T08:53:16.1532634Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T08:53:16.1533013Z encoder_outputs = self.encoder( 2025-12-04T08:53:16.1533388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T08:53:16.1533773Z layer_outputs = encoder_layer( 2025-12-04T08:53:16.1534125Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1534488Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1534869Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T08:53:16.1535265Z hidden_states, attn_weights = self.self_attn( 2025-12-04T08:53:16.1535656Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1536036Z return func(*args, **kwargs) 2025-12-04T08:53:16.1536413Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 322, in forward 2025-12-04T08:53:16.1536836Z attn_output, attn_weights = attention_interface( 2025-12-04T08:53:16.1537289Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T08:53:16.1537792Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T08:53:16.1537989Z 2025-12-04T08:53:16.1538095Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1538505Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1538831Z res = mod(**inputs) 2025-12-04T08:53:16.1539206Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1539610Z outputs = self.model( 2025-12-04T08:53:16.1540010Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T08:53:16.1540417Z encoder_outputs = self.encoder( 2025-12-04T08:53:16.1540816Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T08:53:16.1541230Z layer_outputs = encoder_layer( 2025-12-04T08:53:16.1541583Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1541961Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1542361Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T08:53:16.1542776Z hidden_states, attn_weights = self.self_attn( 2025-12-04T08:53:16.1543175Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1543585Z return func(*args, **kwargs) 2025-12-04T08:53:16.1543975Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 336, in forward 2025-12-04T08:53:16.1544378Z attn_output = self.out_proj(attn_output) 2025-12-04T08:53:16.1544530Z 2025-12-04T08:53:16.1544637Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1545009Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1545344Z res = mod(**inputs) 2025-12-04T08:53:16.1545716Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1546121Z outputs = self.model( 2025-12-04T08:53:16.1546504Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T08:53:16.1546917Z encoder_outputs = self.encoder( 2025-12-04T08:53:16.1547318Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T08:53:16.1547717Z layer_outputs = encoder_layer( 2025-12-04T08:53:16.1548078Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1548447Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1548851Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 392, in forward 2025-12-04T08:53:16.1549304Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:53:16.1549482Z 2025-12-04T08:53:16.1549596Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1549960Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1550297Z res = mod(**inputs) 2025-12-04T08:53:16.1550674Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1551066Z outputs = self.model( 2025-12-04T08:53:16.1551444Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T08:53:16.1551850Z encoder_outputs = self.encoder( 2025-12-04T08:53:16.1552226Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T08:53:16.1552623Z layer_outputs = encoder_layer( 2025-12-04T08:53:16.1552971Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1553330Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1553707Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 392, in forward 2025-12-04T08:53:16.1554153Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:53:16.1554327Z 2025-12-04T08:53:16.1554427Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1554778Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1555107Z res = mod(**inputs) 2025-12-04T08:53:16.1555461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1555836Z outputs = self.model( 2025-12-04T08:53:16.1556097Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T08:53:16.1556178Z encoder_outputs = self.encoder( 2025-12-04T08:53:16.1556426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T08:53:16.1556518Z layer_outputs = encoder_layer( 2025-12-04T08:53:16.1556744Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1556822Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1557073Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 394, in forward 2025-12-04T08:53:16.1557153Z hidden_states = self.fc2(hidden_states) 2025-12-04T08:53:16.1557156Z 2025-12-04T08:53:16.1557257Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1557454Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1557516Z res = mod(**inputs) 2025-12-04T08:53:16.1557762Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1557838Z outputs = self.model( 2025-12-04T08:53:16.1558086Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T08:53:16.1558165Z encoder_outputs = self.encoder( 2025-12-04T08:53:16.1558604Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T08:53:16.1558684Z layer_outputs = encoder_layer( 2025-12-04T08:53:16.1558911Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1558989Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1559238Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T08:53:16.1559336Z hidden_states, attn_weights = self.self_attn( 2025-12-04T08:53:16.1559579Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1559661Z return func(*args, **kwargs) 2025-12-04T08:53:16.1559912Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-12-04T08:53:16.1560069Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T08:53:16.1560074Z 2025-12-04T08:53:16.1560190Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1560394Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1560469Z res = mod(**inputs) 2025-12-04T08:53:16.1560755Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1560828Z outputs = self.model( 2025-12-04T08:53:16.1561100Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T08:53:16.1561178Z encoder_outputs = self.encoder( 2025-12-04T08:53:16.1561459Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T08:53:16.1561542Z layer_outputs = encoder_layer( 2025-12-04T08:53:16.1561790Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1561889Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1562139Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T08:53:16.1562228Z hidden_states, attn_weights = self.self_attn( 2025-12-04T08:53:16.1562478Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1562547Z return func(*args, **kwargs) 2025-12-04T08:53:16.1562824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-12-04T08:53:16.1562904Z key_states = self.k_proj(current_states) 2025-12-04T08:53:16.1562908Z 2025-12-04T08:53:16.1563009Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1563214Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1563278Z res = mod(**inputs) 2025-12-04T08:53:16.1563528Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1563605Z outputs = self.model( 2025-12-04T08:53:16.1563854Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T08:53:16.1563933Z encoder_outputs = self.encoder( 2025-12-04T08:53:16.1564182Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T08:53:16.1564255Z layer_outputs = encoder_layer( 2025-12-04T08:53:16.1564480Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1564558Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1564812Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T08:53:16.1564903Z hidden_states, attn_weights = self.self_attn( 2025-12-04T08:53:16.1565145Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1565221Z return func(*args, **kwargs) 2025-12-04T08:53:16.1565469Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 304, in forward 2025-12-04T08:53:16.1565555Z value_states = self.v_proj(current_states) 2025-12-04T08:53:16.1565558Z 2025-12-04T08:53:16.1565649Z cudagraph partition due to non gpu ops 2025-12-04T08:53:16.1565749Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1565950Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1566015Z res = mod(**inputs) 2025-12-04T08:53:16.1566266Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1566340Z outputs = self.model( 2025-12-04T08:53:16.1566610Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T08:53:16.1566682Z encoder_outputs = self.encoder( 2025-12-04T08:53:16.1566941Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T08:53:16.1567015Z layer_outputs = encoder_layer( 2025-12-04T08:53:16.1567268Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1567348Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1567606Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T08:53:16.1567719Z hidden_states, attn_weights = self.self_attn( 2025-12-04T08:53:16.1567965Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1568043Z return func(*args, **kwargs) 2025-12-04T08:53:16.1568293Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 322, in forward 2025-12-04T08:53:16.1568393Z attn_output, attn_weights = attention_interface( 2025-12-04T08:53:16.1568721Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T08:53:16.1568856Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T08:53:16.1568860Z 2025-12-04T08:53:16.1568963Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1569173Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1569237Z res = mod(**inputs) 2025-12-04T08:53:16.1569501Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1569570Z outputs = self.model( 2025-12-04T08:53:16.1569826Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T08:53:16.1569908Z encoder_outputs = self.encoder( 2025-12-04T08:53:16.1570162Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T08:53:16.1570244Z layer_outputs = encoder_layer( 2025-12-04T08:53:16.1570470Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1570548Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1570810Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 381, in forward 2025-12-04T08:53:16.1570898Z hidden_states, attn_weights = self.self_attn( 2025-12-04T08:53:16.1571145Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1571221Z return func(*args, **kwargs) 2025-12-04T08:53:16.1571473Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 336, in forward 2025-12-04T08:53:16.1571565Z attn_output = self.out_proj(attn_output) 2025-12-04T08:53:16.1571568Z 2025-12-04T08:53:16.1571670Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1571867Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1571939Z res = mod(**inputs) 2025-12-04T08:53:16.1572197Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1572271Z outputs = self.model( 2025-12-04T08:53:16.1572539Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T08:53:16.1572614Z encoder_outputs = self.encoder( 2025-12-04T08:53:16.1572873Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T08:53:16.1572945Z layer_outputs = encoder_layer( 2025-12-04T08:53:16.1573168Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1573269Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1573526Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 392, in forward 2025-12-04T08:53:16.1573668Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:53:16.1573671Z 2025-12-04T08:53:16.1573774Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1573974Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1574048Z res = mod(**inputs) 2025-12-04T08:53:16.1574305Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1574375Z outputs = self.model( 2025-12-04T08:53:16.1574634Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T08:53:16.1574726Z encoder_outputs = self.encoder( 2025-12-04T08:53:16.1574990Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T08:53:16.1575064Z layer_outputs = encoder_layer( 2025-12-04T08:53:16.1575289Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1575375Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1575631Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 392, in forward 2025-12-04T08:53:16.1575755Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:53:16.1575758Z 2025-12-04T08:53:16.1575861Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1576062Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1576135Z res = mod(**inputs) 2025-12-04T08:53:16.1576396Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1576465Z outputs = self.model( 2025-12-04T08:53:16.1576734Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T08:53:16.1576810Z encoder_outputs = self.encoder( 2025-12-04T08:53:16.1577078Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T08:53:16.1577151Z layer_outputs = encoder_layer( 2025-12-04T08:53:16.1577375Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1577460Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1577719Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 394, in forward 2025-12-04T08:53:16.1577809Z hidden_states = self.fc2(hidden_states) 2025-12-04T08:53:16.1577812Z 2025-12-04T08:53:16.1577914Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1578116Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1578189Z res = mod(**inputs) 2025-12-04T08:53:16.1578449Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1578533Z outputs = self.model( 2025-12-04T08:53:16.1578800Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1258, in forward 2025-12-04T08:53:16.1578873Z encoder_outputs = self.encoder( 2025-12-04T08:53:16.1579134Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 883, in forward 2025-12-04T08:53:16.1579222Z layer_outputs = encoder_layer( 2025-12-04T08:53:16.1579447Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1579532Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1579800Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 396, in forward 2025-12-04T08:53:16.1579881Z hidden_states = residual + hidden_states 2025-12-04T08:53:16.1579892Z 2025-12-04T08:53:16.1579994Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1580191Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1580271Z res = mod(**inputs) 2025-12-04T08:53:16.1580526Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1580611Z outputs = self.model( 2025-12-04T08:53:16.1580891Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.1580964Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.1581229Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1098, in forward 2025-12-04T08:53:16.1581398Z positions = self.embed_positions(input_ids, inputs_embeds, past_key_values_length) 2025-12-04T08:53:16.1581641Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context 2025-12-04T08:53:16.1581720Z return func(*args, **kwargs) 2025-12-04T08:53:16.1581979Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 149, in forward 2025-12-04T08:53:16.1582204Z position_ids = create_position_ids_from_input_ids(input_ids, self.padding_idx, past_key_values_length).to( 2025-12-04T08:53:16.1582530Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 82, in create_position_ids_from_input_ids 2025-12-04T08:53:16.1582725Z incremental_indices = (torch.cumsum(mask, dim=1).type_as(mask) + past_key_values_length) * mask 2025-12-04T08:53:16.1582730Z 2025-12-04T08:53:16.1582840Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1583042Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1583109Z res = mod(**inputs) 2025-12-04T08:53:16.1583390Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1583459Z outputs = self.model( 2025-12-04T08:53:16.1583739Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.1583814Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.1584073Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1098, in forward 2025-12-04T08:53:16.1584247Z positions = self.embed_positions(input_ids, inputs_embeds, past_key_values_length) 2025-12-04T08:53:16.1584489Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context 2025-12-04T08:53:16.1584569Z return func(*args, **kwargs) 2025-12-04T08:53:16.1584845Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 149, in forward 2025-12-04T08:53:16.1585058Z position_ids = create_position_ids_from_input_ids(input_ids, self.padding_idx, past_key_values_length).to( 2025-12-04T08:53:16.1585381Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 82, in create_position_ids_from_input_ids 2025-12-04T08:53:16.1585584Z incremental_indices = (torch.cumsum(mask, dim=1).type_as(mask) + past_key_values_length) * mask 2025-12-04T08:53:16.1585589Z 2025-12-04T08:53:16.1585701Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1585914Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1585978Z res = mod(**inputs) 2025-12-04T08:53:16.1586239Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1586308Z outputs = self.model( 2025-12-04T08:53:16.1586564Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.1586644Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.1586896Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.1586997Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.1587221Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1587301Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1587556Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1587626Z return func(*args, **kwargs) 2025-12-04T08:53:16.1587889Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T08:53:16.1587991Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:53:16.1588234Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1588312Z return func(*args, **kwargs) 2025-12-04T08:53:16.1588568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-12-04T08:53:16.1588719Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T08:53:16.1588730Z 2025-12-04T08:53:16.1588832Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1589030Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1589102Z res = mod(**inputs) 2025-12-04T08:53:16.1589359Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1589429Z outputs = self.model( 2025-12-04T08:53:16.1589689Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.1589763Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.1590040Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.1590114Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.1590338Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1590430Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1590674Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1590747Z return func(*args, **kwargs) 2025-12-04T08:53:16.1591036Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T08:53:16.1591143Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:53:16.1591405Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1591477Z return func(*args, **kwargs) 2025-12-04T08:53:16.1591756Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-12-04T08:53:16.1591850Z key_states = self.k_proj(current_states) 2025-12-04T08:53:16.1591883Z 2025-12-04T08:53:16.1591988Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1592201Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1592267Z res = mod(**inputs) 2025-12-04T08:53:16.1592532Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1592611Z outputs = self.model( 2025-12-04T08:53:16.1592876Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.1592967Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.1593239Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.1593314Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.1593555Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1593638Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1593896Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1593976Z return func(*args, **kwargs) 2025-12-04T08:53:16.1594239Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T08:53:16.1594342Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:53:16.1594614Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1594692Z return func(*args, **kwargs) 2025-12-04T08:53:16.1594974Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 304, in forward 2025-12-04T08:53:16.1595070Z value_states = self.v_proj(current_states) 2025-12-04T08:53:16.1595075Z 2025-12-04T08:53:16.1595163Z cudagraph partition due to non gpu ops 2025-12-04T08:53:16.1595284Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1595498Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1595576Z res = mod(**inputs) 2025-12-04T08:53:16.1595855Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1595928Z outputs = self.model( 2025-12-04T08:53:16.1596268Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.1596350Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.1596629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.1596717Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.1596960Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1597052Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1597338Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1597412Z return func(*args, **kwargs) 2025-12-04T08:53:16.1597698Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T08:53:16.1597805Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:53:16.1598091Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1598175Z return func(*args, **kwargs) 2025-12-04T08:53:16.1598528Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 322, in forward 2025-12-04T08:53:16.1598670Z attn_output, attn_weights = attention_interface( 2025-12-04T08:53:16.1598999Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T08:53:16.1599150Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T08:53:16.1599155Z 2025-12-04T08:53:16.1599281Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1599514Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1599613Z res = mod(**inputs) 2025-12-04T08:53:16.1599903Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1599979Z outputs = self.model( 2025-12-04T08:53:16.1600265Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.1600342Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.1600610Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.1600695Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.1600927Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1601018Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1601269Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1601342Z return func(*args, **kwargs) 2025-12-04T08:53:16.1601615Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T08:53:16.1601714Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:53:16.1601969Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1602040Z return func(*args, **kwargs) 2025-12-04T08:53:16.1602301Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 336, in forward 2025-12-04T08:53:16.1602394Z attn_output = self.out_proj(attn_output) 2025-12-04T08:53:16.1602397Z 2025-12-04T08:53:16.1602501Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1602702Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1602776Z res = mod(**inputs) 2025-12-04T08:53:16.1603043Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1603117Z outputs = self.model( 2025-12-04T08:53:16.1603379Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.1603454Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.1603719Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.1603815Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.1604043Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1604131Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1604379Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1604474Z return func(*args, **kwargs) 2025-12-04T08:53:16.1604737Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T08:53:16.1604850Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T08:53:16.1605122Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1605191Z return func(*args, **kwargs) 2025-12-04T08:53:16.1605459Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-12-04T08:53:16.1605615Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T08:53:16.1605619Z 2025-12-04T08:53:16.1605723Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1605955Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1606022Z res = mod(**inputs) 2025-12-04T08:53:16.1606284Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1606361Z outputs = self.model( 2025-12-04T08:53:16.1606631Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.1606713Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.1606977Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.1607050Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.1607292Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1607372Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1607635Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1607706Z return func(*args, **kwargs) 2025-12-04T08:53:16.1607967Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T08:53:16.1608091Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T08:53:16.1608339Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1608408Z return func(*args, **kwargs) 2025-12-04T08:53:16.1608681Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-12-04T08:53:16.1608762Z key_states = self.k_proj(current_states) 2025-12-04T08:53:16.1608766Z 2025-12-04T08:53:16.1608879Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1609087Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1609152Z res = mod(**inputs) 2025-12-04T08:53:16.1609433Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1609505Z outputs = self.model( 2025-12-04T08:53:16.1609777Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.1609853Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.1610135Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.1610218Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.1610447Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1610529Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1610807Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1610878Z return func(*args, **kwargs) 2025-12-04T08:53:16.1611144Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T08:53:16.1611272Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T08:53:16.1611519Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1611598Z return func(*args, **kwargs) 2025-12-04T08:53:16.1611859Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 304, in forward 2025-12-04T08:53:16.1611945Z value_states = self.v_proj(current_states) 2025-12-04T08:53:16.1611972Z 2025-12-04T08:53:16.1612060Z cudagraph partition due to non gpu ops 2025-12-04T08:53:16.1612168Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1612379Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1612445Z res = mod(**inputs) 2025-12-04T08:53:16.1612708Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1612787Z outputs = self.model( 2025-12-04T08:53:16.1613049Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.1613132Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.1613397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.1613471Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.1613709Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1613790Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1614042Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1614120Z return func(*args, **kwargs) 2025-12-04T08:53:16.1614391Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T08:53:16.1614509Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T08:53:16.1614763Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1614833Z return func(*args, **kwargs) 2025-12-04T08:53:16.1615107Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 322, in forward 2025-12-04T08:53:16.1615209Z attn_output, attn_weights = attention_interface( 2025-12-04T08:53:16.1615521Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T08:53:16.1615655Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T08:53:16.1615660Z 2025-12-04T08:53:16.1615766Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1615972Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1616036Z res = mod(**inputs) 2025-12-04T08:53:16.1616314Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1616390Z outputs = self.model( 2025-12-04T08:53:16.1616645Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.1616726Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.1617006Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.1617079Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.1617312Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1617407Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1617655Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1617723Z return func(*args, **kwargs) 2025-12-04T08:53:16.1617976Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T08:53:16.1618088Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T08:53:16.1618327Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1618412Z return func(*args, **kwargs) 2025-12-04T08:53:16.1618677Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 336, in forward 2025-12-04T08:53:16.1618761Z attn_output = self.out_proj(attn_output) 2025-12-04T08:53:16.1618766Z 2025-12-04T08:53:16.1618873Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1619074Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1619137Z res = mod(**inputs) 2025-12-04T08:53:16.1619408Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1619475Z outputs = self.model( 2025-12-04T08:53:16.1619736Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.1619819Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.1620084Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.1620162Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.1620406Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1620484Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1620939Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1621016Z return func(*args, **kwargs) 2025-12-04T08:53:16.1621283Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 508, in forward 2025-12-04T08:53:16.1621404Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:53:16.1621409Z 2025-12-04T08:53:16.1621516Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1621729Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1621795Z res = mod(**inputs) 2025-12-04T08:53:16.1622050Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1622128Z outputs = self.model( 2025-12-04T08:53:16.1622383Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.1622508Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.1622768Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.1622840Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.1623070Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1623173Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1623425Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1623493Z return func(*args, **kwargs) 2025-12-04T08:53:16.1623773Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 508, in forward 2025-12-04T08:53:16.1623901Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:53:16.1623905Z 2025-12-04T08:53:16.1624008Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1624206Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1624280Z res = mod(**inputs) 2025-12-04T08:53:16.1624535Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1624633Z outputs = self.model( 2025-12-04T08:53:16.1624892Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.1624966Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.1625230Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.1625305Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.1625526Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1625615Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1625858Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1625933Z return func(*args, **kwargs) 2025-12-04T08:53:16.1626189Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 510, in forward 2025-12-04T08:53:16.1626272Z hidden_states = self.fc2(hidden_states) 2025-12-04T08:53:16.1626276Z 2025-12-04T08:53:16.1626386Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1626583Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1626656Z res = mod(**inputs) 2025-12-04T08:53:16.1626912Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1626980Z outputs = self.model( 2025-12-04T08:53:16.1627239Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.1627311Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.1627565Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.1627647Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.1627868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1627953Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1628194Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1628263Z return func(*args, **kwargs) 2025-12-04T08:53:16.1628565Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T08:53:16.1628666Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:53:16.1628914Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1628984Z return func(*args, **kwargs) 2025-12-04T08:53:16.1629262Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-12-04T08:53:16.1629423Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T08:53:16.1629426Z 2025-12-04T08:53:16.1629529Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1629744Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1629813Z res = mod(**inputs) 2025-12-04T08:53:16.1630070Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1630143Z outputs = self.model( 2025-12-04T08:53:16.1630399Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.1630471Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.1630755Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.1630827Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.1631056Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1631137Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1631379Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1631453Z return func(*args, **kwargs) 2025-12-04T08:53:16.1631709Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T08:53:16.1631808Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:53:16.1632057Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1632127Z return func(*args, **kwargs) 2025-12-04T08:53:16.1632389Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-12-04T08:53:16.1632469Z key_states = self.k_proj(current_states) 2025-12-04T08:53:16.1632473Z 2025-12-04T08:53:16.1632575Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1632781Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1632845Z res = mod(**inputs) 2025-12-04T08:53:16.1633101Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1633176Z outputs = self.model( 2025-12-04T08:53:16.1633428Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.1633507Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.1633760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.1633831Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.1634063Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1634143Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1634407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1634475Z return func(*args, **kwargs) 2025-12-04T08:53:16.1634747Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T08:53:16.1634856Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:53:16.1635104Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1635175Z return func(*args, **kwargs) 2025-12-04T08:53:16.1635454Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 304, in forward 2025-12-04T08:53:16.1635545Z value_states = self.v_proj(current_states) 2025-12-04T08:53:16.1635563Z 2025-12-04T08:53:16.1635654Z cudagraph partition due to non gpu ops 2025-12-04T08:53:16.1635758Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1635958Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1636034Z res = mod(**inputs) 2025-12-04T08:53:16.1636293Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1636361Z outputs = self.model( 2025-12-04T08:53:16.1636628Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.1636722Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.1636990Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.1637063Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.1637294Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1637383Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1637633Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1637708Z return func(*args, **kwargs) 2025-12-04T08:53:16.1637970Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T08:53:16.1638071Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:53:16.1638331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1638458Z return func(*args, **kwargs) 2025-12-04T08:53:16.1638729Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 322, in forward 2025-12-04T08:53:16.1638846Z attn_output, attn_weights = attention_interface( 2025-12-04T08:53:16.1639164Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T08:53:16.1639320Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T08:53:16.1639324Z 2025-12-04T08:53:16.1639435Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1639659Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1639741Z res = mod(**inputs) 2025-12-04T08:53:16.1640025Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1640107Z outputs = self.model( 2025-12-04T08:53:16.1640388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.1640466Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.1640807Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.1640883Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.1641147Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1641238Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1641488Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1641567Z return func(*args, **kwargs) 2025-12-04T08:53:16.1641915Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T08:53:16.1642018Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:53:16.1642290Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1642360Z return func(*args, **kwargs) 2025-12-04T08:53:16.1642632Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 336, in forward 2025-12-04T08:53:16.1642717Z attn_output = self.out_proj(attn_output) 2025-12-04T08:53:16.1642720Z 2025-12-04T08:53:16.1642826Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1643041Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1643123Z res = mod(**inputs) 2025-12-04T08:53:16.1643398Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1643474Z outputs = self.model( 2025-12-04T08:53:16.1643739Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.1643824Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.1644087Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.1644164Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.1644429Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1644513Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1644764Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1644846Z return func(*args, **kwargs) 2025-12-04T08:53:16.1645112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 486, in forward 2025-12-04T08:53:16.1645202Z hidden_states = residual + hidden_states 2025-12-04T08:53:16.1645207Z 2025-12-04T08:53:16.1645313Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1645518Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1645593Z res = mod(**inputs) 2025-12-04T08:53:16.1645859Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1645935Z outputs = self.model( 2025-12-04T08:53:16.1646198Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.1646275Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.1646548Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.1646622Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.1646852Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1646940Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1647191Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1647289Z return func(*args, **kwargs) 2025-12-04T08:53:16.1647555Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T08:53:16.1647666Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T08:53:16.1647929Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1648016Z return func(*args, **kwargs) 2025-12-04T08:53:16.1648285Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-12-04T08:53:16.1648456Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T08:53:16.1648460Z 2025-12-04T08:53:16.1648564Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1648775Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1648841Z res = mod(**inputs) 2025-12-04T08:53:16.1649106Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1649183Z outputs = self.model( 2025-12-04T08:53:16.1649447Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.1649547Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.1649812Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.1649885Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.1650120Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1650207Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1650451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1650517Z return func(*args, **kwargs) 2025-12-04T08:53:16.1650763Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T08:53:16.1650875Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T08:53:16.1651111Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1651177Z return func(*args, **kwargs) 2025-12-04T08:53:16.1651430Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-12-04T08:53:16.1651510Z key_states = self.k_proj(current_states) 2025-12-04T08:53:16.1651514Z 2025-12-04T08:53:16.1651619Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1651813Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1651875Z res = mod(**inputs) 2025-12-04T08:53:16.1652129Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1652193Z outputs = self.model( 2025-12-04T08:53:16.1652441Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.1652530Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.1653241Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.1653320Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.1653531Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1653606Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1653861Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1653928Z return func(*args, **kwargs) 2025-12-04T08:53:16.1654174Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T08:53:16.1654277Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T08:53:16.1654522Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1654596Z return func(*args, **kwargs) 2025-12-04T08:53:16.1654838Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 304, in forward 2025-12-04T08:53:16.1654939Z value_states = self.v_proj(current_states) 2025-12-04T08:53:16.1654950Z 2025-12-04T08:53:16.1655029Z cudagraph partition due to non gpu ops 2025-12-04T08:53:16.1655127Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1655322Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1655385Z res = mod(**inputs) 2025-12-04T08:53:16.1655629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1655717Z outputs = self.model( 2025-12-04T08:53:16.1655965Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.1656043Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.1656289Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.1656361Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.1656586Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1656663Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1656898Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1656971Z return func(*args, **kwargs) 2025-12-04T08:53:16.1657217Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T08:53:16.1657328Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T08:53:16.1657564Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1657631Z return func(*args, **kwargs) 2025-12-04T08:53:16.1657885Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 322, in forward 2025-12-04T08:53:16.1657979Z attn_output, attn_weights = attention_interface( 2025-12-04T08:53:16.1658269Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T08:53:16.1658395Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T08:53:16.1658399Z 2025-12-04T08:53:16.1658497Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1658696Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1658757Z res = mod(**inputs) 2025-12-04T08:53:16.1659008Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1659081Z outputs = self.model( 2025-12-04T08:53:16.1659330Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.1659407Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.1659671Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.1659741Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.1659961Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1660038Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1660286Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1660351Z return func(*args, **kwargs) 2025-12-04T08:53:16.1660596Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T08:53:16.1660721Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T08:53:16.1660950Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1661015Z return func(*args, **kwargs) 2025-12-04T08:53:16.1661261Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 336, in forward 2025-12-04T08:53:16.1661340Z attn_output = self.out_proj(attn_output) 2025-12-04T08:53:16.1661343Z 2025-12-04T08:53:16.1661472Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1661664Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1661728Z res = mod(**inputs) 2025-12-04T08:53:16.1661990Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1662057Z outputs = self.model( 2025-12-04T08:53:16.1662309Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.1662388Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.1662643Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.1662721Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.1662952Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1663027Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1663271Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1663337Z return func(*args, **kwargs) 2025-12-04T08:53:16.1663594Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 508, in forward 2025-12-04T08:53:16.1663714Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:53:16.1663717Z 2025-12-04T08:53:16.1663816Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1664021Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1664083Z res = mod(**inputs) 2025-12-04T08:53:16.1664337Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1664409Z outputs = self.model( 2025-12-04T08:53:16.1664663Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.1664742Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.1664999Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.1665071Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.1665299Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1665390Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1665640Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1665705Z return func(*args, **kwargs) 2025-12-04T08:53:16.1665955Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 508, in forward 2025-12-04T08:53:16.1666093Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:53:16.1666097Z 2025-12-04T08:53:16.1666198Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1666390Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1666474Z res = mod(**inputs) 2025-12-04T08:53:16.1666723Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1666796Z outputs = self.model( 2025-12-04T08:53:16.1667051Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.1667124Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.1667385Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.1667475Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.1667700Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1667786Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1668031Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1668106Z return func(*args, **kwargs) 2025-12-04T08:53:16.1668363Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 510, in forward 2025-12-04T08:53:16.1668444Z hidden_states = self.fc2(hidden_states) 2025-12-04T08:53:16.1668447Z 2025-12-04T08:53:16.1668553Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1668746Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1668817Z res = mod(**inputs) 2025-12-04T08:53:16.1669065Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1669131Z outputs = self.model( 2025-12-04T08:53:16.1669385Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.1669458Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.1669704Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.1669784Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.1670004Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1670089Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1670327Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1670396Z return func(*args, **kwargs) 2025-12-04T08:53:16.1670653Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T08:53:16.1670752Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:53:16.1670996Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1671063Z return func(*args, **kwargs) 2025-12-04T08:53:16.1671323Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-12-04T08:53:16.1671477Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T08:53:16.1671480Z 2025-12-04T08:53:16.1671579Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1671768Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1671839Z res = mod(**inputs) 2025-12-04T08:53:16.1672102Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1672176Z outputs = self.model( 2025-12-04T08:53:16.1672445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.1672516Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.1672771Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.1672841Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.1673061Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1673137Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1673390Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1673463Z return func(*args, **kwargs) 2025-12-04T08:53:16.1673708Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T08:53:16.1673805Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:53:16.1674046Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1674112Z return func(*args, **kwargs) 2025-12-04T08:53:16.1674366Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-12-04T08:53:16.1674444Z key_states = self.k_proj(current_states) 2025-12-04T08:53:16.1674448Z 2025-12-04T08:53:16.1674546Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1674747Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1674814Z res = mod(**inputs) 2025-12-04T08:53:16.1675062Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1675138Z outputs = self.model( 2025-12-04T08:53:16.1675385Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.1675464Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.1675712Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.1675783Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.1676011Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1676090Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1676341Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1676408Z return func(*args, **kwargs) 2025-12-04T08:53:16.1676663Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T08:53:16.1676770Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:53:16.1677010Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1677077Z return func(*args, **kwargs) 2025-12-04T08:53:16.1677351Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 304, in forward 2025-12-04T08:53:16.1677438Z value_states = self.v_proj(current_states) 2025-12-04T08:53:16.1677442Z 2025-12-04T08:53:16.1677529Z cudagraph partition due to non gpu ops 2025-12-04T08:53:16.1677630Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1677840Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1677911Z res = mod(**inputs) 2025-12-04T08:53:16.1678169Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1678261Z outputs = self.model( 2025-12-04T08:53:16.1678628Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.1678720Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.1679014Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.1679096Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.1679354Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1679475Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1679755Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1679831Z return func(*args, **kwargs) 2025-12-04T08:53:16.1680095Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T08:53:16.1680198Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:53:16.1680457Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1680527Z return func(*args, **kwargs) 2025-12-04T08:53:16.1680788Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 322, in forward 2025-12-04T08:53:16.1680896Z attn_output, attn_weights = attention_interface( 2025-12-04T08:53:16.1681203Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T08:53:16.1681346Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T08:53:16.1681350Z 2025-12-04T08:53:16.1681455Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1681659Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1681733Z res = mod(**inputs) 2025-12-04T08:53:16.1682009Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1682085Z outputs = self.model( 2025-12-04T08:53:16.1682348Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.1682425Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.1682696Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.1682770Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.1682999Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1683086Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1683334Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1683411Z return func(*args, **kwargs) 2025-12-04T08:53:16.1683685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T08:53:16.1683785Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:53:16.1684049Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1684124Z return func(*args, **kwargs) 2025-12-04T08:53:16.1684422Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 336, in forward 2025-12-04T08:53:16.1684512Z attn_output = self.out_proj(attn_output) 2025-12-04T08:53:16.1684541Z 2025-12-04T08:53:16.1684658Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1684869Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1684935Z res = mod(**inputs) 2025-12-04T08:53:16.1685201Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1685282Z outputs = self.model( 2025-12-04T08:53:16.1685572Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.1685676Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.1685965Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.1686039Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.1686281Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1686361Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1686618Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1686699Z return func(*args, **kwargs) 2025-12-04T08:53:16.1686965Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T08:53:16.1687083Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T08:53:16.1687338Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1687411Z return func(*args, **kwargs) 2025-12-04T08:53:16.1687685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-12-04T08:53:16.1687838Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T08:53:16.1687843Z 2025-12-04T08:53:16.1687954Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1688165Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1688233Z res = mod(**inputs) 2025-12-04T08:53:16.1688511Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1688580Z outputs = self.model( 2025-12-04T08:53:16.1688846Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.1688929Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.1689196Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.1689277Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.1689512Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1689591Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1689875Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1689947Z return func(*args, **kwargs) 2025-12-04T08:53:16.1690225Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T08:53:16.1690339Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T08:53:16.1690611Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1690691Z return func(*args, **kwargs) 2025-12-04T08:53:16.1690950Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-12-04T08:53:16.1691051Z key_states = self.k_proj(current_states) 2025-12-04T08:53:16.1691062Z 2025-12-04T08:53:16.1691166Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1691369Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1691442Z res = mod(**inputs) 2025-12-04T08:53:16.1691704Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1691775Z outputs = self.model( 2025-12-04T08:53:16.1692058Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.1692134Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.1692399Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.1692475Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.1692703Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1692788Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1693035Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1693105Z return func(*args, **kwargs) 2025-12-04T08:53:16.1693370Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T08:53:16.1693481Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T08:53:16.1693739Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1693808Z return func(*args, **kwargs) 2025-12-04T08:53:16.1694068Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 304, in forward 2025-12-04T08:53:16.1694164Z value_states = self.v_proj(current_states) 2025-12-04T08:53:16.1694169Z 2025-12-04T08:53:16.1694252Z cudagraph partition due to non gpu ops 2025-12-04T08:53:16.1694355Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1694565Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1694631Z res = mod(**inputs) 2025-12-04T08:53:16.1694898Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1694970Z outputs = self.model( 2025-12-04T08:53:16.1695230Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.1695311Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.1695569Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.1695650Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.1695874Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1695972Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1696231Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1696300Z return func(*args, **kwargs) 2025-12-04T08:53:16.1696567Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T08:53:16.1696699Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T08:53:16.1696948Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1697041Z return func(*args, **kwargs) 2025-12-04T08:53:16.1697299Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 322, in forward 2025-12-04T08:53:16.1697398Z attn_output, attn_weights = attention_interface( 2025-12-04T08:53:16.1697708Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T08:53:16.1697843Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T08:53:16.1697847Z 2025-12-04T08:53:16.1697957Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1698178Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1698244Z res = mod(**inputs) 2025-12-04T08:53:16.1698516Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1698586Z outputs = self.model( 2025-12-04T08:53:16.1698849Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.1698931Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.1699194Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.1699274Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.1699504Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1699587Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1699851Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1699923Z return func(*args, **kwargs) 2025-12-04T08:53:16.1700175Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T08:53:16.1700290Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T08:53:16.1700537Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1700614Z return func(*args, **kwargs) 2025-12-04T08:53:16.1700859Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 336, in forward 2025-12-04T08:53:16.1700941Z attn_output = self.out_proj(attn_output) 2025-12-04T08:53:16.1700944Z 2025-12-04T08:53:16.1701056Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1701263Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1701334Z res = mod(**inputs) 2025-12-04T08:53:16.1701592Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1701661Z outputs = self.model( 2025-12-04T08:53:16.1701922Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.1701994Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.1702264Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.1702346Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.1702569Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1702661Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1702929Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1702997Z return func(*args, **kwargs) 2025-12-04T08:53:16.1703252Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 503, in forward 2025-12-04T08:53:16.1703345Z hidden_states = residual + hidden_states 2025-12-04T08:53:16.1703348Z 2025-12-04T08:53:16.1703458Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1703654Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1703716Z res = mod(**inputs) 2025-12-04T08:53:16.1703974Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1704056Z outputs = self.model( 2025-12-04T08:53:16.1704309Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.1704388Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.1704636Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.1704715Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.1704932Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1705008Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1705263Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1705331Z return func(*args, **kwargs) 2025-12-04T08:53:16.1705579Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 508, in forward 2025-12-04T08:53:16.1705705Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:53:16.1705708Z 2025-12-04T08:53:16.1705808Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1706011Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1706075Z res = mod(**inputs) 2025-12-04T08:53:16.1706327Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1706402Z outputs = self.model( 2025-12-04T08:53:16.1706652Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.1706730Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.1706979Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.1707051Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.1707277Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1707353Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1707594Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1707668Z return func(*args, **kwargs) 2025-12-04T08:53:16.1707915Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 508, in forward 2025-12-04T08:53:16.1708055Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:53:16.1708059Z 2025-12-04T08:53:16.1708160Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1708355Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1708426Z res = mod(**inputs) 2025-12-04T08:53:16.1708696Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1708769Z outputs = self.model( 2025-12-04T08:53:16.1709019Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.1709105Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.1709358Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.1709427Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.1709642Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1709726Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1709963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1710051Z return func(*args, **kwargs) 2025-12-04T08:53:16.1710304Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 510, in forward 2025-12-04T08:53:16.1710381Z hidden_states = self.fc2(hidden_states) 2025-12-04T08:53:16.1710386Z 2025-12-04T08:53:16.1710494Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1710691Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1710757Z res = mod(**inputs) 2025-12-04T08:53:16.1711013Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1711080Z outputs = self.model( 2025-12-04T08:53:16.1711351Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.1711425Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.1711680Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.1711756Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.1711976Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1712062Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1712304Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1712370Z return func(*args, **kwargs) 2025-12-04T08:53:16.1712631Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T08:53:16.1712726Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:53:16.1712968Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1713044Z return func(*args, **kwargs) 2025-12-04T08:53:16.1713298Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-12-04T08:53:16.1713451Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T08:53:16.1713456Z 2025-12-04T08:53:16.1713554Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1713759Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1713828Z res = mod(**inputs) 2025-12-04T08:53:16.1714091Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1714164Z outputs = self.model( 2025-12-04T08:53:16.1714405Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.1714491Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.1714740Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.1714811Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.1715042Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1715127Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1715366Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1715439Z return func(*args, **kwargs) 2025-12-04T08:53:16.1715685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T08:53:16.1715781Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:53:16.1716045Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1716112Z return func(*args, **kwargs) 2025-12-04T08:53:16.1716367Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-12-04T08:53:16.1716447Z key_states = self.k_proj(current_states) 2025-12-04T08:53:16.1716450Z 2025-12-04T08:53:16.1716549Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1716748Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1716811Z res = mod(**inputs) 2025-12-04T08:53:16.1717060Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1717135Z outputs = self.model( 2025-12-04T08:53:16.1717389Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.1717471Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.1717722Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.1717795Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.1718023Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1718101Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1718398Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1718497Z return func(*args, **kwargs) 2025-12-04T08:53:16.1718785Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T08:53:16.1718903Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:53:16.1719179Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1719255Z return func(*args, **kwargs) 2025-12-04T08:53:16.1719555Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 304, in forward 2025-12-04T08:53:16.1719652Z value_states = self.v_proj(current_states) 2025-12-04T08:53:16.1719656Z 2025-12-04T08:53:16.1719754Z cudagraph partition due to non gpu ops 2025-12-04T08:53:16.1719864Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1720112Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1720185Z res = mod(**inputs) 2025-12-04T08:53:16.1720436Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1720505Z outputs = self.model( 2025-12-04T08:53:16.1720985Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.1721063Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.1721322Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.1721416Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.1721634Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1721721Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1721956Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1722030Z return func(*args, **kwargs) 2025-12-04T08:53:16.1722276Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T08:53:16.1722426Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:53:16.1722672Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1722739Z return func(*args, **kwargs) 2025-12-04T08:53:16.1722988Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 322, in forward 2025-12-04T08:53:16.1723093Z attn_output, attn_weights = attention_interface( 2025-12-04T08:53:16.1723383Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T08:53:16.1723520Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T08:53:16.1723523Z 2025-12-04T08:53:16.1723621Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1723816Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1723889Z res = mod(**inputs) 2025-12-04T08:53:16.1724142Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1724215Z outputs = self.model( 2025-12-04T08:53:16.1724464Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.1724534Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.1724789Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.1724859Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.1725079Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1725164Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1725403Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1725478Z return func(*args, **kwargs) 2025-12-04T08:53:16.1725728Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T08:53:16.1725825Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:53:16.1726071Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1726137Z return func(*args, **kwargs) 2025-12-04T08:53:16.1726408Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 336, in forward 2025-12-04T08:53:16.1726497Z attn_output = self.out_proj(attn_output) 2025-12-04T08:53:16.1726500Z 2025-12-04T08:53:16.1726599Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1726813Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1726880Z res = mod(**inputs) 2025-12-04T08:53:16.1727129Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1727218Z outputs = self.model( 2025-12-04T08:53:16.1727467Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.1727544Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.1727795Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.1727866Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.1728101Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1728192Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1728429Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1728505Z return func(*args, **kwargs) 2025-12-04T08:53:16.1728750Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T08:53:16.1728864Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T08:53:16.1729111Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1729181Z return func(*args, **kwargs) 2025-12-04T08:53:16.1729441Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-12-04T08:53:16.1729589Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T08:53:16.1729595Z 2025-12-04T08:53:16.1729704Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1729900Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1729965Z res = mod(**inputs) 2025-12-04T08:53:16.1730227Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1730297Z outputs = self.model( 2025-12-04T08:53:16.1730552Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.1730635Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.1730895Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.1730973Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.1731190Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1731270Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1731519Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1731587Z return func(*args, **kwargs) 2025-12-04T08:53:16.1731842Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T08:53:16.1731961Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T08:53:16.1732228Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1732306Z return func(*args, **kwargs) 2025-12-04T08:53:16.1732560Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-12-04T08:53:16.1732642Z key_states = self.k_proj(current_states) 2025-12-04T08:53:16.1732646Z 2025-12-04T08:53:16.1732769Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1732968Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1733038Z res = mod(**inputs) 2025-12-04T08:53:16.1733311Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1733379Z outputs = self.model( 2025-12-04T08:53:16.1733640Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.1733714Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.1733969Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.1734049Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.1734288Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1734374Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1734617Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1734685Z return func(*args, **kwargs) 2025-12-04T08:53:16.1734946Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T08:53:16.1735054Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T08:53:16.1735304Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1735373Z return func(*args, **kwargs) 2025-12-04T08:53:16.1735627Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 304, in forward 2025-12-04T08:53:16.1735721Z value_states = self.v_proj(current_states) 2025-12-04T08:53:16.1735724Z 2025-12-04T08:53:16.1735807Z cudagraph partition due to non gpu ops 2025-12-04T08:53:16.1735909Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1736114Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1736179Z res = mod(**inputs) 2025-12-04T08:53:16.1736441Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1736509Z outputs = self.model( 2025-12-04T08:53:16.1736762Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.1736842Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.1737099Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.1737171Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.1737402Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1737479Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1737727Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1737797Z return func(*args, **kwargs) 2025-12-04T08:53:16.1738048Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T08:53:16.1738180Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T08:53:16.1738426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1738501Z return func(*args, **kwargs) 2025-12-04T08:53:16.1738755Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 322, in forward 2025-12-04T08:53:16.1738870Z attn_output, attn_weights = attention_interface( 2025-12-04T08:53:16.1739172Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T08:53:16.1739319Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T08:53:16.1739323Z 2025-12-04T08:53:16.1739431Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1739630Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1739695Z res = mod(**inputs) 2025-12-04T08:53:16.1739956Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1740022Z outputs = self.model( 2025-12-04T08:53:16.1740290Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.1740372Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.1740631Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.1740712Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.1740938Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1741018Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1741273Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1741342Z return func(*args, **kwargs) 2025-12-04T08:53:16.1741610Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T08:53:16.1741726Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T08:53:16.1741968Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1742041Z return func(*args, **kwargs) 2025-12-04T08:53:16.1742297Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 336, in forward 2025-12-04T08:53:16.1742379Z attn_output = self.out_proj(attn_output) 2025-12-04T08:53:16.1742383Z 2025-12-04T08:53:16.1742490Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1742690Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1742760Z res = mod(**inputs) 2025-12-04T08:53:16.1743025Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1743092Z outputs = self.model( 2025-12-04T08:53:16.1743354Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.1743427Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.1743679Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.1743761Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.1743983Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1744069Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1744324Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1744393Z return func(*args, **kwargs) 2025-12-04T08:53:16.1744655Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 508, in forward 2025-12-04T08:53:16.1744775Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:53:16.1744793Z 2025-12-04T08:53:16.1744903Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1745101Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1745179Z res = mod(**inputs) 2025-12-04T08:53:16.1745443Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1745511Z outputs = self.model( 2025-12-04T08:53:16.1745764Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.1745843Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.1746095Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.1746190Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.1746415Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1746492Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1746742Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1746811Z return func(*args, **kwargs) 2025-12-04T08:53:16.1747063Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 508, in forward 2025-12-04T08:53:16.1747189Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:53:16.1747193Z 2025-12-04T08:53:16.1747296Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1747497Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1747564Z res = mod(**inputs) 2025-12-04T08:53:16.1747821Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1747895Z outputs = self.model( 2025-12-04T08:53:16.1748148Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.1748230Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.1748483Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.1748555Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.1748784Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1748862Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1749104Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1749181Z return func(*args, **kwargs) 2025-12-04T08:53:16.1749435Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 510, in forward 2025-12-04T08:53:16.1749522Z hidden_states = self.fc2(hidden_states) 2025-12-04T08:53:16.1749525Z 2025-12-04T08:53:16.1749628Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1749824Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1749897Z res = mod(**inputs) 2025-12-04T08:53:16.1750170Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1750239Z outputs = self.model( 2025-12-04T08:53:16.1750511Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.1750583Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.1750855Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.1750927Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.1751143Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1751246Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1751482Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1751555Z return func(*args, **kwargs) 2025-12-04T08:53:16.1751808Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 512, in forward 2025-12-04T08:53:16.1751886Z hidden_states = residual + hidden_states 2025-12-04T08:53:16.1751890Z 2025-12-04T08:53:16.1751995Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1752217Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1752280Z res = mod(**inputs) 2025-12-04T08:53:16.1752536Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1752604Z outputs = self.model( 2025-12-04T08:53:16.1752860Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.1752931Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.1753176Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.1753253Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.1753468Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1753552Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1753796Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1753865Z return func(*args, **kwargs) 2025-12-04T08:53:16.1754122Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T08:53:16.1754222Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:53:16.1754465Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1754540Z return func(*args, **kwargs) 2025-12-04T08:53:16.1754794Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-12-04T08:53:16.1754952Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T08:53:16.1754958Z 2025-12-04T08:53:16.1755060Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1755262Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1755334Z res = mod(**inputs) 2025-12-04T08:53:16.1755597Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1755673Z outputs = self.model( 2025-12-04T08:53:16.1755933Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.1756025Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.1756294Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.1756368Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.1756594Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1756697Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1756953Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1757029Z return func(*args, **kwargs) 2025-12-04T08:53:16.1757303Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T08:53:16.1757403Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:53:16.1757662Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1757732Z return func(*args, **kwargs) 2025-12-04T08:53:16.1757989Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-12-04T08:53:16.1758096Z key_states = self.k_proj(current_states) 2025-12-04T08:53:16.1758100Z 2025-12-04T08:53:16.1758206Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1758486Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1758558Z res = mod(**inputs) 2025-12-04T08:53:16.1758821Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1758903Z outputs = self.model( 2025-12-04T08:53:16.1759180Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.1759272Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.1759556Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.1759646Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.1759881Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1759964Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1760213Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1760293Z return func(*args, **kwargs) 2025-12-04T08:53:16.1760552Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T08:53:16.1760659Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:53:16.1760910Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1760981Z return func(*args, **kwargs) 2025-12-04T08:53:16.1761257Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 304, in forward 2025-12-04T08:53:16.1761345Z value_states = self.v_proj(current_states) 2025-12-04T08:53:16.1761349Z 2025-12-04T08:53:16.1761438Z cudagraph partition due to non gpu ops 2025-12-04T08:53:16.1761542Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1761739Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1761813Z res = mod(**inputs) 2025-12-04T08:53:16.1762065Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1762132Z outputs = self.model( 2025-12-04T08:53:16.1762435Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.1762509Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.1762767Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.1762839Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.1763071Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1763156Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1763397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1763480Z return func(*args, **kwargs) 2025-12-04T08:53:16.1763747Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T08:53:16.1763847Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:53:16.1764094Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1764161Z return func(*args, **kwargs) 2025-12-04T08:53:16.1764413Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 322, in forward 2025-12-04T08:53:16.1764536Z attn_output, attn_weights = attention_interface( 2025-12-04T08:53:16.1764829Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T08:53:16.1764974Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T08:53:16.1764977Z 2025-12-04T08:53:16.1765078Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1765277Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1765350Z res = mod(**inputs) 2025-12-04T08:53:16.1765608Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1765676Z outputs = self.model( 2025-12-04T08:53:16.1765943Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.1766022Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.1766287Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.1766363Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.1766592Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1766681Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1766935Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1767013Z return func(*args, **kwargs) 2025-12-04T08:53:16.1767274Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T08:53:16.1767376Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:53:16.1767644Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1767713Z return func(*args, **kwargs) 2025-12-04T08:53:16.1767976Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 336, in forward 2025-12-04T08:53:16.1768067Z attn_output = self.out_proj(attn_output) 2025-12-04T08:53:16.1768070Z 2025-12-04T08:53:16.1768173Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1768403Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1768470Z res = mod(**inputs) 2025-12-04T08:53:16.1768730Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1768808Z outputs = self.model( 2025-12-04T08:53:16.1769082Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.1769158Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.1769426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.1769517Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.1769753Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1769831Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1770085Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1770160Z return func(*args, **kwargs) 2025-12-04T08:53:16.1770424Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T08:53:16.1770558Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T08:53:16.1770809Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1770879Z return func(*args, **kwargs) 2025-12-04T08:53:16.1771143Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-12-04T08:53:16.1771301Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T08:53:16.1771305Z 2025-12-04T08:53:16.1771415Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1771619Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1771684Z res = mod(**inputs) 2025-12-04T08:53:16.1771953Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1772023Z outputs = self.model( 2025-12-04T08:53:16.1772286Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.1772369Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.1772627Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.1772710Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.1772940Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1773021Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1773275Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1773344Z return func(*args, **kwargs) 2025-12-04T08:53:16.1773600Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T08:53:16.1773718Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T08:53:16.1773969Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1774047Z return func(*args, **kwargs) 2025-12-04T08:53:16.1774308Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-12-04T08:53:16.1774391Z key_states = self.k_proj(current_states) 2025-12-04T08:53:16.1774394Z 2025-12-04T08:53:16.1774523Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1774730Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1774803Z res = mod(**inputs) 2025-12-04T08:53:16.1775065Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1775135Z outputs = self.model( 2025-12-04T08:53:16.1775418Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.1775494Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.1775783Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.1775868Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.1776096Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1776184Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1776432Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1776502Z return func(*args, **kwargs) 2025-12-04T08:53:16.1776789Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T08:53:16.1776902Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T08:53:16.1777153Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1777233Z return func(*args, **kwargs) 2025-12-04T08:53:16.1777494Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 304, in forward 2025-12-04T08:53:16.1777590Z value_states = self.v_proj(current_states) 2025-12-04T08:53:16.1777593Z 2025-12-04T08:53:16.1777679Z cudagraph partition due to non gpu ops 2025-12-04T08:53:16.1777785Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1777996Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1778063Z res = mod(**inputs) 2025-12-04T08:53:16.1778332Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1778401Z outputs = self.model( 2025-12-04T08:53:16.1778662Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.1778746Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.1779010Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.1779084Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.1779321Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1779401Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1779655Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1779726Z return func(*args, **kwargs) 2025-12-04T08:53:16.1779992Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T08:53:16.1780109Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T08:53:16.1780360Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1780437Z return func(*args, **kwargs) 2025-12-04T08:53:16.1780696Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 322, in forward 2025-12-04T08:53:16.1780817Z attn_output, attn_weights = attention_interface( 2025-12-04T08:53:16.1781128Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T08:53:16.1781263Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T08:53:16.1781268Z 2025-12-04T08:53:16.1781387Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1781598Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1781665Z res = mod(**inputs) 2025-12-04T08:53:16.1781960Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1782038Z outputs = self.model( 2025-12-04T08:53:16.1782293Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.1782374Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.1782640Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.1782722Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.1782959Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1783040Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1783301Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1783371Z return func(*args, **kwargs) 2025-12-04T08:53:16.1783629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T08:53:16.1783747Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T08:53:16.1783995Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1784071Z return func(*args, **kwargs) 2025-12-04T08:53:16.1784329Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 336, in forward 2025-12-04T08:53:16.1784415Z attn_output = self.out_proj(attn_output) 2025-12-04T08:53:16.1784420Z 2025-12-04T08:53:16.1784532Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1784742Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1784806Z res = mod(**inputs) 2025-12-04T08:53:16.1785076Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1785143Z outputs = self.model( 2025-12-04T08:53:16.1785407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.1785480Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.1785736Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.1785815Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.1786042Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1786126Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1786369Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1786438Z return func(*args, **kwargs) 2025-12-04T08:53:16.1786701Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 508, in forward 2025-12-04T08:53:16.1786821Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:53:16.1786844Z 2025-12-04T08:53:16.1786947Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1787153Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1787218Z res = mod(**inputs) 2025-12-04T08:53:16.1787503Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1787571Z outputs = self.model( 2025-12-04T08:53:16.1787827Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.1787929Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.1788187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.1788267Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.1788495Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1788573Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1788828Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1788914Z return func(*args, **kwargs) 2025-12-04T08:53:16.1789168Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 508, in forward 2025-12-04T08:53:16.1789296Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:53:16.1789299Z 2025-12-04T08:53:16.1789404Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1789610Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1789674Z res = mod(**inputs) 2025-12-04T08:53:16.1789935Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1790011Z outputs = self.model( 2025-12-04T08:53:16.1790267Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.1790343Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.1790607Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.1790678Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.1790904Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1790983Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1791225Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1791300Z return func(*args, **kwargs) 2025-12-04T08:53:16.1791556Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 510, in forward 2025-12-04T08:53:16.1791644Z hidden_states = self.fc2(hidden_states) 2025-12-04T08:53:16.1791649Z 2025-12-04T08:53:16.1791752Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1791948Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1792021Z res = mod(**inputs) 2025-12-04T08:53:16.1792275Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1792344Z outputs = self.model( 2025-12-04T08:53:16.1792605Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.1792678Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.1792957Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.1793032Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.1793255Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1793342Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1793604Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1793681Z return func(*args, **kwargs) 2025-12-04T08:53:16.1793940Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T08:53:16.1794054Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:53:16.1794305Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1794373Z return func(*args, **kwargs) 2025-12-04T08:53:16.1794624Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-12-04T08:53:16.1794780Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T08:53:16.1794799Z 2025-12-04T08:53:16.1794904Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1795117Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1795182Z res = mod(**inputs) 2025-12-04T08:53:16.1795446Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1795525Z outputs = self.model( 2025-12-04T08:53:16.1795788Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.1795871Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.1796134Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.1796209Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.1796444Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1796527Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1796777Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1796855Z return func(*args, **kwargs) 2025-12-04T08:53:16.1797116Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T08:53:16.1797223Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:53:16.1797471Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1797542Z return func(*args, **kwargs) 2025-12-04T08:53:16.1797808Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-12-04T08:53:16.1797891Z key_states = self.k_proj(current_states) 2025-12-04T08:53:16.1797896Z 2025-12-04T08:53:16.1797999Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1798210Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1798274Z res = mod(**inputs) 2025-12-04T08:53:16.1798611Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1798692Z outputs = self.model( 2025-12-04T08:53:16.1798950Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.1799057Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.1799341Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.1799428Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.1799688Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1799799Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1800073Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1800146Z return func(*args, **kwargs) 2025-12-04T08:53:16.1800439Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T08:53:16.1800553Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:53:16.1800818Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1800900Z return func(*args, **kwargs) 2025-12-04T08:53:16.1801183Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 304, in forward 2025-12-04T08:53:16.1801288Z value_states = self.v_proj(current_states) 2025-12-04T08:53:16.1801292Z 2025-12-04T08:53:16.1801385Z cudagraph partition due to non gpu ops 2025-12-04T08:53:16.1801490Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1801698Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1801765Z res = mod(**inputs) 2025-12-04T08:53:16.1802031Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1802108Z outputs = self.model( 2025-12-04T08:53:16.1802372Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.1802446Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.1802714Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.1802789Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.1803027Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1803109Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1803358Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1803438Z return func(*args, **kwargs) 2025-12-04T08:53:16.1803708Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T08:53:16.1803809Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:53:16.1804067Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1804136Z return func(*args, **kwargs) 2025-12-04T08:53:16.1804403Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 322, in forward 2025-12-04T08:53:16.1804507Z attn_output, attn_weights = attention_interface( 2025-12-04T08:53:16.1804808Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T08:53:16.1804952Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T08:53:16.1804955Z 2025-12-04T08:53:16.1805058Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1805268Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1805352Z res = mod(**inputs) 2025-12-04T08:53:16.1805617Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1805694Z outputs = self.model( 2025-12-04T08:53:16.1805958Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.1806062Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.1806330Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.1806403Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.1806654Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1806734Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1806987Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1807065Z return func(*args, **kwargs) 2025-12-04T08:53:16.1807329Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T08:53:16.1807434Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:53:16.1807701Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1807771Z return func(*args, **kwargs) 2025-12-04T08:53:16.1808037Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 336, in forward 2025-12-04T08:53:16.1808121Z attn_output = self.out_proj(attn_output) 2025-12-04T08:53:16.1808124Z 2025-12-04T08:53:16.1808227Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1808438Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1808504Z res = mod(**inputs) 2025-12-04T08:53:16.1808771Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1808840Z outputs = self.model( 2025-12-04T08:53:16.1809102Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.1809186Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.1809447Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.1809523Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.1809758Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1809840Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1810097Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1810167Z return func(*args, **kwargs) 2025-12-04T08:53:16.1810425Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 486, in forward 2025-12-04T08:53:16.1810517Z hidden_states = residual + hidden_states 2025-12-04T08:53:16.1810521Z 2025-12-04T08:53:16.1810625Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1810845Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1810906Z res = mod(**inputs) 2025-12-04T08:53:16.1811159Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1811234Z outputs = self.model( 2025-12-04T08:53:16.1811500Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.1811572Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.1811830Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.1811901Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.1812150Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1812228Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1812472Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1812562Z return func(*args, **kwargs) 2025-12-04T08:53:16.1812816Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T08:53:16.1812932Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T08:53:16.1813176Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1813245Z return func(*args, **kwargs) 2025-12-04T08:53:16.1813504Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-12-04T08:53:16.1813672Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T08:53:16.1813676Z 2025-12-04T08:53:16.1813777Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1813982Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1814049Z res = mod(**inputs) 2025-12-04T08:53:16.1814313Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1814379Z outputs = self.model( 2025-12-04T08:53:16.1814634Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.1814715Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.1814968Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.1815047Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.1815270Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1815347Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1815592Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1815661Z return func(*args, **kwargs) 2025-12-04T08:53:16.1815912Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T08:53:16.1816028Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T08:53:16.1816271Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1816344Z return func(*args, **kwargs) 2025-12-04T08:53:16.1816594Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-12-04T08:53:16.1816676Z key_states = self.k_proj(current_states) 2025-12-04T08:53:16.1816679Z 2025-12-04T08:53:16.1816787Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1816984Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1817051Z res = mod(**inputs) 2025-12-04T08:53:16.1817312Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1817379Z outputs = self.model( 2025-12-04T08:53:16.1817656Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.1817730Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.1817987Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.1818068Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.1818305Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1818392Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1818678Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1818745Z return func(*args, **kwargs) 2025-12-04T08:53:16.1819009Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T08:53:16.1819116Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T08:53:16.1819362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1819438Z return func(*args, **kwargs) 2025-12-04T08:53:16.1819708Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 304, in forward 2025-12-04T08:53:16.1819801Z value_states = self.v_proj(current_states) 2025-12-04T08:53:16.1819804Z 2025-12-04T08:53:16.1819885Z cudagraph partition due to non gpu ops 2025-12-04T08:53:16.1819988Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1820195Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1820259Z res = mod(**inputs) 2025-12-04T08:53:16.1820522Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1820600Z outputs = self.model( 2025-12-04T08:53:16.1821023Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.1821111Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.1821370Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.1821444Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.1821674Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1821757Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1822008Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1822077Z return func(*args, **kwargs) 2025-12-04T08:53:16.1822330Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T08:53:16.1822446Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T08:53:16.1822689Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1822760Z return func(*args, **kwargs) 2025-12-04T08:53:16.1823024Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 322, in forward 2025-12-04T08:53:16.1823123Z attn_output, attn_weights = attention_interface( 2025-12-04T08:53:16.1823424Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T08:53:16.1823566Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T08:53:16.1823570Z 2025-12-04T08:53:16.1823717Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1823923Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1823987Z res = mod(**inputs) 2025-12-04T08:53:16.1824264Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1824333Z outputs = self.model( 2025-12-04T08:53:16.1824609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.1824688Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.1824957Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.1825030Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.1825254Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1825332Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1825577Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1825644Z return func(*args, **kwargs) 2025-12-04T08:53:16.1825913Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T08:53:16.1826027Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T08:53:16.1826262Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1826336Z return func(*args, **kwargs) 2025-12-04T08:53:16.1826594Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 336, in forward 2025-12-04T08:53:16.1826674Z attn_output = self.out_proj(attn_output) 2025-12-04T08:53:16.1826677Z 2025-12-04T08:53:16.1826788Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1826980Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1827042Z res = mod(**inputs) 2025-12-04T08:53:16.1827340Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1827410Z outputs = self.model( 2025-12-04T08:53:16.1827669Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.1827742Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.1827996Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.1828074Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.1828295Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1828373Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1828622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1828692Z return func(*args, **kwargs) 2025-12-04T08:53:16.1828951Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 508, in forward 2025-12-04T08:53:16.1829068Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:53:16.1829072Z 2025-12-04T08:53:16.1829173Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1829379Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1829443Z res = mod(**inputs) 2025-12-04T08:53:16.1829730Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1829801Z outputs = self.model( 2025-12-04T08:53:16.1830062Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.1830143Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.1830424Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.1830499Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.1830731Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1830826Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1831081Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1831150Z return func(*args, **kwargs) 2025-12-04T08:53:16.1831410Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 508, in forward 2025-12-04T08:53:16.1831540Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:53:16.1831544Z 2025-12-04T08:53:16.1831646Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1831873Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1831941Z res = mod(**inputs) 2025-12-04T08:53:16.1832204Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1832280Z outputs = self.model( 2025-12-04T08:53:16.1832546Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.1832621Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.1832893Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.1832966Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.1833212Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1833293Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1833544Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1833621Z return func(*args, **kwargs) 2025-12-04T08:53:16.1833884Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 510, in forward 2025-12-04T08:53:16.1833970Z hidden_states = self.fc2(hidden_states) 2025-12-04T08:53:16.1833981Z 2025-12-04T08:53:16.1834085Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1834291Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1834365Z res = mod(**inputs) 2025-12-04T08:53:16.1834630Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1834703Z outputs = self.model( 2025-12-04T08:53:16.1834993Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.1835073Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.1835359Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.1835438Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.1835678Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1835770Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1836069Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1836144Z return func(*args, **kwargs) 2025-12-04T08:53:16.1836425Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T08:53:16.1836536Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:53:16.1836820Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1836896Z return func(*args, **kwargs) 2025-12-04T08:53:16.1837173Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-12-04T08:53:16.1837366Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T08:53:16.1837370Z 2025-12-04T08:53:16.1837481Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1837705Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1837774Z res = mod(**inputs) 2025-12-04T08:53:16.1838052Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1838151Z outputs = self.model( 2025-12-04T08:53:16.1838489Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.1838576Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.1838868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.1838949Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.1839203Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1839291Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1839565Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1839661Z return func(*args, **kwargs) 2025-12-04T08:53:16.1839935Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T08:53:16.1840056Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:53:16.1840323Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1840399Z return func(*args, **kwargs) 2025-12-04T08:53:16.1840690Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-12-04T08:53:16.1840779Z key_states = self.k_proj(current_states) 2025-12-04T08:53:16.1840783Z 2025-12-04T08:53:16.1860970Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1861336Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1861412Z res = mod(**inputs) 2025-12-04T08:53:16.1861703Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1861779Z outputs = self.model( 2025-12-04T08:53:16.1862050Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.1862140Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.1862388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.1862460Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.1862684Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1862861Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1863116Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1863186Z return func(*args, **kwargs) 2025-12-04T08:53:16.1863439Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T08:53:16.1863579Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:53:16.1863820Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1863924Z return func(*args, **kwargs) 2025-12-04T08:53:16.1864181Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 304, in forward 2025-12-04T08:53:16.1864273Z value_states = self.v_proj(current_states) 2025-12-04T08:53:16.1864280Z 2025-12-04T08:53:16.1864378Z cudagraph partition due to non gpu ops 2025-12-04T08:53:16.1864493Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1864698Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1864775Z res = mod(**inputs) 2025-12-04T08:53:16.1865072Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1865150Z outputs = self.model( 2025-12-04T08:53:16.1865399Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.1865470Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.1865728Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.1865801Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.1866031Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1866108Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1866349Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1866429Z return func(*args, **kwargs) 2025-12-04T08:53:16.1866678Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T08:53:16.1866778Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:53:16.1867025Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1867095Z return func(*args, **kwargs) 2025-12-04T08:53:16.1867346Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 322, in forward 2025-12-04T08:53:16.1867447Z attn_output, attn_weights = attention_interface( 2025-12-04T08:53:16.1867736Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T08:53:16.1867880Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T08:53:16.1867886Z 2025-12-04T08:53:16.1867992Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1868211Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1868275Z res = mod(**inputs) 2025-12-04T08:53:16.1868521Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1868603Z outputs = self.model( 2025-12-04T08:53:16.1868843Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.1868927Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.1869182Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.1869252Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.1869478Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1869572Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1869808Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1869879Z return func(*args, **kwargs) 2025-12-04T08:53:16.1870144Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T08:53:16.1870250Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:53:16.1870488Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1870556Z return func(*args, **kwargs) 2025-12-04T08:53:16.1870812Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 336, in forward 2025-12-04T08:53:16.1870909Z attn_output = self.out_proj(attn_output) 2025-12-04T08:53:16.1870913Z 2025-12-04T08:53:16.1871018Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1871224Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1871288Z res = mod(**inputs) 2025-12-04T08:53:16.1871547Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1871612Z outputs = self.model( 2025-12-04T08:53:16.1871859Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.1871938Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.1872192Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.1872264Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.1872498Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1872579Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1872828Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1872911Z return func(*args, **kwargs) 2025-12-04T08:53:16.1873161Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T08:53:16.1873270Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T08:53:16.1873519Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1873596Z return func(*args, **kwargs) 2025-12-04T08:53:16.1873844Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-12-04T08:53:16.1873998Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T08:53:16.1874004Z 2025-12-04T08:53:16.1874118Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1874321Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1874388Z res = mod(**inputs) 2025-12-04T08:53:16.1874663Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1874734Z outputs = self.model( 2025-12-04T08:53:16.1875019Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.1875097Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.1875355Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.1875440Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.1875689Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1875781Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1876046Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1876137Z return func(*args, **kwargs) 2025-12-04T08:53:16.1876422Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T08:53:16.1876547Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T08:53:16.1876814Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1876899Z return func(*args, **kwargs) 2025-12-04T08:53:16.1877175Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-12-04T08:53:16.1877288Z key_states = self.k_proj(current_states) 2025-12-04T08:53:16.1877292Z 2025-12-04T08:53:16.1877403Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1877619Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1877700Z res = mod(**inputs) 2025-12-04T08:53:16.1877976Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1878060Z outputs = self.model( 2025-12-04T08:53:16.1878336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.1878526Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.1878832Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.1878915Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.1879164Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1879260Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1879526Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1879613Z return func(*args, **kwargs) 2025-12-04T08:53:16.1879894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T08:53:16.1880008Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T08:53:16.1880271Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1880344Z return func(*args, **kwargs) 2025-12-04T08:53:16.1880615Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 304, in forward 2025-12-04T08:53:16.1880714Z value_states = self.v_proj(current_states) 2025-12-04T08:53:16.1880718Z 2025-12-04T08:53:16.1880804Z cudagraph partition due to non gpu ops 2025-12-04T08:53:16.1880921Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1881130Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1881199Z res = mod(**inputs) 2025-12-04T08:53:16.1881498Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1881574Z outputs = self.model( 2025-12-04T08:53:16.1881861Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.1881943Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.1882240Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.1882327Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.1882570Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1882672Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1882947Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1883022Z return func(*args, **kwargs) 2025-12-04T08:53:16.1883309Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T08:53:16.1883434Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T08:53:16.1883685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1883781Z return func(*args, **kwargs) 2025-12-04T08:53:16.1884051Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 322, in forward 2025-12-04T08:53:16.1884152Z attn_output, attn_weights = attention_interface( 2025-12-04T08:53:16.1884457Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T08:53:16.1884593Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T08:53:16.1884597Z 2025-12-04T08:53:16.1884712Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1884910Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1884976Z res = mod(**inputs) 2025-12-04T08:53:16.1885247Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1885324Z outputs = self.model( 2025-12-04T08:53:16.1885609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.1885691Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.1885967Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.1886056Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.1886307Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1886393Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1886665Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1886739Z return func(*args, **kwargs) 2025-12-04T08:53:16.1887023Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T08:53:16.1887141Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T08:53:16.1887406Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1887490Z return func(*args, **kwargs) 2025-12-04T08:53:16.1887763Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 336, in forward 2025-12-04T08:53:16.1887860Z attn_output = self.out_proj(attn_output) 2025-12-04T08:53:16.1887864Z 2025-12-04T08:53:16.1887994Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1888210Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1888291Z res = mod(**inputs) 2025-12-04T08:53:16.1888569Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1888665Z outputs = self.model( 2025-12-04T08:53:16.1888954Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.1889033Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.1889344Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.1889420Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.1889650Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1889740Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1889991Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1890071Z return func(*args, **kwargs) 2025-12-04T08:53:16.1890347Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 503, in forward 2025-12-04T08:53:16.1890430Z hidden_states = residual + hidden_states 2025-12-04T08:53:16.1890434Z 2025-12-04T08:53:16.1890548Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1890754Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1890820Z res = mod(**inputs) 2025-12-04T08:53:16.1891090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1891163Z outputs = self.model( 2025-12-04T08:53:16.1891435Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.1891510Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.1891769Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.1891855Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.1892086Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1892166Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1892424Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1892495Z return func(*args, **kwargs) 2025-12-04T08:53:16.1892765Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 508, in forward 2025-12-04T08:53:16.1892904Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:53:16.1892908Z 2025-12-04T08:53:16.1893021Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1893239Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1893307Z res = mod(**inputs) 2025-12-04T08:53:16.1893583Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1893656Z outputs = self.model( 2025-12-04T08:53:16.1893913Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.1893994Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.1894268Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.1894342Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.1894588Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1894664Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1894931Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1895005Z return func(*args, **kwargs) 2025-12-04T08:53:16.1895262Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 508, in forward 2025-12-04T08:53:16.1895408Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:53:16.1895412Z 2025-12-04T08:53:16.1895516Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1895724Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1895799Z res = mod(**inputs) 2025-12-04T08:53:16.1896071Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1896137Z outputs = self.model( 2025-12-04T08:53:16.1896416Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.1896523Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.1896792Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.1896867Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.1897109Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1897190Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1897434Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1897508Z return func(*args, **kwargs) 2025-12-04T08:53:16.1897760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 510, in forward 2025-12-04T08:53:16.1897840Z hidden_states = self.fc2(hidden_states) 2025-12-04T08:53:16.1897844Z 2025-12-04T08:53:16.1897954Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1898151Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1898214Z res = mod(**inputs) 2025-12-04T08:53:16.1898476Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1898541Z outputs = self.model( 2025-12-04T08:53:16.1898803Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.1898877Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.1899130Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.1899210Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.1899435Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1899520Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1899761Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1899831Z return func(*args, **kwargs) 2025-12-04T08:53:16.1900093Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T08:53:16.1900194Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:53:16.1900456Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1900531Z return func(*args, **kwargs) 2025-12-04T08:53:16.1900792Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-12-04T08:53:16.1900956Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T08:53:16.1900977Z 2025-12-04T08:53:16.1901083Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1901286Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1901375Z res = mod(**inputs) 2025-12-04T08:53:16.1901635Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1901715Z outputs = self.model( 2025-12-04T08:53:16.1901974Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.1902048Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.1902311Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.1902400Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.1902639Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1902724Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1902966Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1903043Z return func(*args, **kwargs) 2025-12-04T08:53:16.1903299Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T08:53:16.1903400Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:53:16.1903652Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1903722Z return func(*args, **kwargs) 2025-12-04T08:53:16.1903984Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-12-04T08:53:16.1904078Z key_states = self.k_proj(current_states) 2025-12-04T08:53:16.1904081Z 2025-12-04T08:53:16.1904182Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1904392Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1904460Z res = mod(**inputs) 2025-12-04T08:53:16.1904720Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1904797Z outputs = self.model( 2025-12-04T08:53:16.1905057Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.1905134Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.1905398Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.1905473Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.1905708Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1905788Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1906036Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1906113Z return func(*args, **kwargs) 2025-12-04T08:53:16.1906373Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T08:53:16.1906495Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:53:16.1906745Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1906814Z return func(*args, **kwargs) 2025-12-04T08:53:16.1907081Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 304, in forward 2025-12-04T08:53:16.1907189Z value_states = self.v_proj(current_states) 2025-12-04T08:53:16.1907194Z 2025-12-04T08:53:16.1907287Z cudagraph partition due to non gpu ops 2025-12-04T08:53:16.1907389Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1907613Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1907682Z res = mod(**inputs) 2025-12-04T08:53:16.1907942Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1908011Z outputs = self.model( 2025-12-04T08:53:16.1908295Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.1908370Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.1908673Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.1908752Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.1908992Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1909086Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1909352Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1909435Z return func(*args, **kwargs) 2025-12-04T08:53:16.1909701Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T08:53:16.1909801Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:53:16.1910057Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1910129Z return func(*args, **kwargs) 2025-12-04T08:53:16.1910392Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 322, in forward 2025-12-04T08:53:16.1910501Z attn_output, attn_weights = attention_interface( 2025-12-04T08:53:16.1910803Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T08:53:16.1910949Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T08:53:16.1910953Z 2025-12-04T08:53:16.1911053Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1911259Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1911333Z res = mod(**inputs) 2025-12-04T08:53:16.1911597Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1911668Z outputs = self.model( 2025-12-04T08:53:16.1911939Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.1912010Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.1912277Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.1912350Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.1912578Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1912680Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1912927Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1913004Z return func(*args, **kwargs) 2025-12-04T08:53:16.1913263Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T08:53:16.1913387Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:53:16.1913642Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1913728Z return func(*args, **kwargs) 2025-12-04T08:53:16.1913990Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 336, in forward 2025-12-04T08:53:16.1914080Z attn_output = self.out_proj(attn_output) 2025-12-04T08:53:16.1914084Z 2025-12-04T08:53:16.1914189Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1914393Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1914457Z res = mod(**inputs) 2025-12-04T08:53:16.1914725Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1914819Z outputs = self.model( 2025-12-04T08:53:16.1915083Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.1915159Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.1915428Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.1915501Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.1915737Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1915816Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1916065Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1916144Z return func(*args, **kwargs) 2025-12-04T08:53:16.1916408Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T08:53:16.1916528Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T08:53:16.1916788Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1916861Z return func(*args, **kwargs) 2025-12-04T08:53:16.1917147Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-12-04T08:53:16.1917311Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T08:53:16.1917316Z 2025-12-04T08:53:16.1917434Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1917653Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1917723Z res = mod(**inputs) 2025-12-04T08:53:16.1918017Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1918090Z outputs = self.model( 2025-12-04T08:53:16.1918449Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.1918552Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.1918833Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.1918921Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.1919187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1919274Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1919543Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1919616Z return func(*args, **kwargs) 2025-12-04T08:53:16.1919908Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T08:53:16.1920038Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T08:53:16.1920302Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1920404Z return func(*args, **kwargs) 2025-12-04T08:53:16.1920948Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-12-04T08:53:16.1921051Z key_states = self.k_proj(current_states) 2025-12-04T08:53:16.1921055Z 2025-12-04T08:53:16.1921176Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1921392Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1921528Z res = mod(**inputs) 2025-12-04T08:53:16.1921808Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1921882Z outputs = self.model( 2025-12-04T08:53:16.1922167Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.1922245Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.1922518Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.1922605Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.1922844Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1922935Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1923199Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1923274Z return func(*args, **kwargs) 2025-12-04T08:53:16.1923553Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T08:53:16.1923669Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T08:53:16.1923930Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1924011Z return func(*args, **kwargs) 2025-12-04T08:53:16.1924295Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 304, in forward 2025-12-04T08:53:16.1924397Z value_states = self.v_proj(current_states) 2025-12-04T08:53:16.1924401Z 2025-12-04T08:53:16.1924491Z cudagraph partition due to non gpu ops 2025-12-04T08:53:16.1924601Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1924822Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1924892Z res = mod(**inputs) 2025-12-04T08:53:16.1925175Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1925249Z outputs = self.model( 2025-12-04T08:53:16.1925518Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.1925602Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.1925901Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.1925981Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.1926229Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1926313Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1926606Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1926681Z return func(*args, **kwargs) 2025-12-04T08:53:16.1926953Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T08:53:16.1927113Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T08:53:16.1927373Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1927447Z return func(*args, **kwargs) 2025-12-04T08:53:16.1927722Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 322, in forward 2025-12-04T08:53:16.1927830Z attn_output, attn_weights = attention_interface( 2025-12-04T08:53:16.1928146Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T08:53:16.1928312Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T08:53:16.1928316Z 2025-12-04T08:53:16.1928426Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1928653Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1928723Z res = mod(**inputs) 2025-12-04T08:53:16.1929007Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1929081Z outputs = self.model( 2025-12-04T08:53:16.1929360Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.1929446Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.1929722Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.1929801Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.1930049Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1930133Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1930403Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1930480Z return func(*args, **kwargs) 2025-12-04T08:53:16.1930751Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T08:53:16.1930874Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T08:53:16.1931136Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1931215Z return func(*args, **kwargs) 2025-12-04T08:53:16.1931494Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 336, in forward 2025-12-04T08:53:16.1931576Z attn_output = self.out_proj(attn_output) 2025-12-04T08:53:16.1931579Z 2025-12-04T08:53:16.1931691Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1931894Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1931960Z res = mod(**inputs) 2025-12-04T08:53:16.1932229Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1932315Z outputs = self.model( 2025-12-04T08:53:16.1932586Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.1932657Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.1932918Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.1933019Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.1933246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1933333Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1933600Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1933670Z return func(*args, **kwargs) 2025-12-04T08:53:16.1933950Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 508, in forward 2025-12-04T08:53:16.1934074Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:53:16.1934078Z 2025-12-04T08:53:16.1934184Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1934408Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1934489Z res = mod(**inputs) 2025-12-04T08:53:16.1934758Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1934827Z outputs = self.model( 2025-12-04T08:53:16.1935089Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.1935172Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.1935432Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.1935507Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.1935747Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1935825Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1936082Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1936152Z return func(*args, **kwargs) 2025-12-04T08:53:16.1936415Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 508, in forward 2025-12-04T08:53:16.1936550Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:53:16.1936554Z 2025-12-04T08:53:16.1936664Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1936884Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1936962Z res = mod(**inputs) 2025-12-04T08:53:16.1937223Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1937298Z outputs = self.model( 2025-12-04T08:53:16.1937557Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.1937633Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.1937901Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.1937974Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.1938218Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1938304Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1938597Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1938682Z return func(*args, **kwargs) 2025-12-04T08:53:16.1938975Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 510, in forward 2025-12-04T08:53:16.1939061Z hidden_states = self.fc2(hidden_states) 2025-12-04T08:53:16.1939070Z 2025-12-04T08:53:16.1939180Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1939439Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1939511Z res = mod(**inputs) 2025-12-04T08:53:16.1939789Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1939877Z outputs = self.model( 2025-12-04T08:53:16.1940160Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.1940238Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.1940522Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.1940599Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.1940857Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1940944Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1941207Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1941284Z return func(*args, **kwargs) 2025-12-04T08:53:16.1941573Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 512, in forward 2025-12-04T08:53:16.1941651Z hidden_states = residual + hidden_states 2025-12-04T08:53:16.1941654Z 2025-12-04T08:53:16.1941768Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1941971Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1942040Z res = mod(**inputs) 2025-12-04T08:53:16.1942337Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1942412Z outputs = self.model( 2025-12-04T08:53:16.1942698Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.1942774Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.1943050Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.1943134Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.1943381Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1943465Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1943734Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1943808Z return func(*args, **kwargs) 2025-12-04T08:53:16.1944092Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T08:53:16.1944197Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:53:16.1944470Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1944552Z return func(*args, **kwargs) 2025-12-04T08:53:16.1944840Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-12-04T08:53:16.1945004Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T08:53:16.1945030Z 2025-12-04T08:53:16.1945150Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1945353Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1945424Z res = mod(**inputs) 2025-12-04T08:53:16.1945708Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1945778Z outputs = self.model( 2025-12-04T08:53:16.1946048Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.1946137Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.1946403Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.1946476Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.1946703Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1946790Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1947035Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1947123Z return func(*args, **kwargs) 2025-12-04T08:53:16.1947399Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T08:53:16.1947498Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:53:16.1947755Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1947827Z return func(*args, **kwargs) 2025-12-04T08:53:16.1948089Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-12-04T08:53:16.1948175Z key_states = self.k_proj(current_states) 2025-12-04T08:53:16.1948179Z 2025-12-04T08:53:16.1948282Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1948489Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1948557Z res = mod(**inputs) 2025-12-04T08:53:16.1948821Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1948894Z outputs = self.model( 2025-12-04T08:53:16.1949155Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.1949229Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.1949498Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.1949569Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.1949814Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1949892Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1950143Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1950222Z return func(*args, **kwargs) 2025-12-04T08:53:16.1950494Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T08:53:16.1950592Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:53:16.1950849Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1950918Z return func(*args, **kwargs) 2025-12-04T08:53:16.1951184Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 304, in forward 2025-12-04T08:53:16.1951285Z value_states = self.v_proj(current_states) 2025-12-04T08:53:16.1951289Z 2025-12-04T08:53:16.1951373Z cudagraph partition due to non gpu ops 2025-12-04T08:53:16.1951485Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1951687Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1951761Z res = mod(**inputs) 2025-12-04T08:53:16.1952034Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1952104Z outputs = self.model( 2025-12-04T08:53:16.1952387Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.1952461Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.1952723Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.1952803Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.1953033Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1953123Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1953405Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1953479Z return func(*args, **kwargs) 2025-12-04T08:53:16.1953764Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T08:53:16.1953868Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:53:16.1954136Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1954206Z return func(*args, **kwargs) 2025-12-04T08:53:16.1954483Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 322, in forward 2025-12-04T08:53:16.1954591Z attn_output, attn_weights = attention_interface( 2025-12-04T08:53:16.1954913Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T08:53:16.1955058Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T08:53:16.1955062Z 2025-12-04T08:53:16.1955178Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1955394Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1955471Z res = mod(**inputs) 2025-12-04T08:53:16.1955750Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1955823Z outputs = self.model( 2025-12-04T08:53:16.1956109Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.1956187Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.1956471Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.1956551Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.1956793Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1956885Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1957147Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1957223Z return func(*args, **kwargs) 2025-12-04T08:53:16.1957507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T08:53:16.1957630Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:53:16.1957901Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1957974Z return func(*args, **kwargs) 2025-12-04T08:53:16.1958252Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 336, in forward 2025-12-04T08:53:16.1958441Z attn_output = self.out_proj(attn_output) 2025-12-04T08:53:16.1958448Z 2025-12-04T08:53:16.1958568Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1958794Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1958895Z res = mod(**inputs) 2025-12-04T08:53:16.1959184Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1959280Z outputs = self.model( 2025-12-04T08:53:16.1959560Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.1959640Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.1959939Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.1960043Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.1960286Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1960367Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1960620Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1960699Z return func(*args, **kwargs) 2025-12-04T08:53:16.1960966Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T08:53:16.1961080Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T08:53:16.1961343Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1961414Z return func(*args, **kwargs) 2025-12-04T08:53:16.1961685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-12-04T08:53:16.1961838Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T08:53:16.1961842Z 2025-12-04T08:53:16.1961946Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1962160Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1962226Z res = mod(**inputs) 2025-12-04T08:53:16.1962500Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1962570Z outputs = self.model( 2025-12-04T08:53:16.1962831Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.1962915Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.1963177Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.1963250Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.1963484Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1963565Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1963820Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1963890Z return func(*args, **kwargs) 2025-12-04T08:53:16.1964165Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T08:53:16.1964284Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T08:53:16.1964545Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1964617Z return func(*args, **kwargs) 2025-12-04T08:53:16.1964902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-12-04T08:53:16.1964986Z key_states = self.k_proj(current_states) 2025-12-04T08:53:16.1964989Z 2025-12-04T08:53:16.1965116Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1965330Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1965404Z res = mod(**inputs) 2025-12-04T08:53:16.1965668Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1965737Z outputs = self.model( 2025-12-04T08:53:16.1966009Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.1966099Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.1966367Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.1966440Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.1966668Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1966754Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1967003Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1967072Z return func(*args, **kwargs) 2025-12-04T08:53:16.1967340Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T08:53:16.1967450Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T08:53:16.1967706Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1967778Z return func(*args, **kwargs) 2025-12-04T08:53:16.1968039Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 304, in forward 2025-12-04T08:53:16.1968133Z value_states = self.v_proj(current_states) 2025-12-04T08:53:16.1968138Z 2025-12-04T08:53:16.1968221Z cudagraph partition due to non gpu ops 2025-12-04T08:53:16.1968326Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1968536Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1968603Z res = mod(**inputs) 2025-12-04T08:53:16.1968871Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1968940Z outputs = self.model( 2025-12-04T08:53:16.1969199Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.1969285Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.1969545Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.1969625Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.1969854Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1969931Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1970199Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1970270Z return func(*args, **kwargs) 2025-12-04T08:53:16.1970532Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T08:53:16.1970647Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T08:53:16.1970917Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1970994Z return func(*args, **kwargs) 2025-12-04T08:53:16.1971256Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 322, in forward 2025-12-04T08:53:16.1971373Z attn_output, attn_weights = attention_interface( 2025-12-04T08:53:16.1971677Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T08:53:16.1971818Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T08:53:16.1971822Z 2025-12-04T08:53:16.1971931Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1972135Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1972218Z res = mod(**inputs) 2025-12-04T08:53:16.1972488Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1972558Z outputs = self.model( 2025-12-04T08:53:16.1972819Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.1972903Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.1973162Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.1973242Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.1973473Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1973554Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1973809Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1973880Z return func(*args, **kwargs) 2025-12-04T08:53:16.1974138Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T08:53:16.1974253Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T08:53:16.1974501Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1974576Z return func(*args, **kwargs) 2025-12-04T08:53:16.1974837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 336, in forward 2025-12-04T08:53:16.1974918Z attn_output = self.out_proj(attn_output) 2025-12-04T08:53:16.1974922Z 2025-12-04T08:53:16.1975035Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1975236Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1975311Z res = mod(**inputs) 2025-12-04T08:53:16.1975576Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1975646Z outputs = self.model( 2025-12-04T08:53:16.1975915Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.1975990Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.1976249Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.1976347Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.1976576Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1976662Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1976909Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1976996Z return func(*args, **kwargs) 2025-12-04T08:53:16.1977265Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 508, in forward 2025-12-04T08:53:16.1977387Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:53:16.1977408Z 2025-12-04T08:53:16.1977521Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1977725Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1977791Z res = mod(**inputs) 2025-12-04T08:53:16.1978066Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1978135Z outputs = self.model( 2025-12-04T08:53:16.1978395Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.1978493Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.1978757Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.1978838Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.1979069Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1979148Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1979411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1979481Z return func(*args, **kwargs) 2025-12-04T08:53:16.1979749Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 508, in forward 2025-12-04T08:53:16.1979869Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:53:16.1979874Z 2025-12-04T08:53:16.1979978Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1980188Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1980253Z res = mod(**inputs) 2025-12-04T08:53:16.1980518Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1980598Z outputs = self.model( 2025-12-04T08:53:16.1980857Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.1980937Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.1981197Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.1981269Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.1981503Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1981585Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1981835Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1981910Z return func(*args, **kwargs) 2025-12-04T08:53:16.1982172Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 510, in forward 2025-12-04T08:53:16.1982261Z hidden_states = self.fc2(hidden_states) 2025-12-04T08:53:16.1982265Z 2025-12-04T08:53:16.1982385Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1982589Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1982663Z res = mod(**inputs) 2025-12-04T08:53:16.1982926Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1983003Z outputs = self.model( 2025-12-04T08:53:16.1983291Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.1983366Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.1983638Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.1983736Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.1983964Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1984052Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1984313Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1984395Z return func(*args, **kwargs) 2025-12-04T08:53:16.1984688Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T08:53:16.1984807Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:53:16.1985062Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1985133Z return func(*args, **kwargs) 2025-12-04T08:53:16.1985404Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-12-04T08:53:16.1985567Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T08:53:16.1985571Z 2025-12-04T08:53:16.1985682Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1985905Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1985973Z res = mod(**inputs) 2025-12-04T08:53:16.1986255Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1986336Z outputs = self.model( 2025-12-04T08:53:16.1986613Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.1986698Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.1986974Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.1987049Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.1987300Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1987383Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1987646Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1987726Z return func(*args, **kwargs) 2025-12-04T08:53:16.1988004Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T08:53:16.1988116Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:53:16.1988377Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1988451Z return func(*args, **kwargs) 2025-12-04T08:53:16.1988730Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-12-04T08:53:16.1988831Z key_states = self.k_proj(current_states) 2025-12-04T08:53:16.1988835Z 2025-12-04T08:53:16.1988950Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1989163Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1989231Z res = mod(**inputs) 2025-12-04T08:53:16.1989531Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1989606Z outputs = self.model( 2025-12-04T08:53:16.1989885Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.1989993Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.1990269Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.1990358Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.1990606Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1990694Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1990972Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1991063Z return func(*args, **kwargs) 2025-12-04T08:53:16.1991350Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T08:53:16.1991455Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:53:16.1991719Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1991801Z return func(*args, **kwargs) 2025-12-04T08:53:16.1992076Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 304, in forward 2025-12-04T08:53:16.1992172Z value_states = self.v_proj(current_states) 2025-12-04T08:53:16.1992176Z 2025-12-04T08:53:16.1992271Z cudagraph partition due to non gpu ops 2025-12-04T08:53:16.1992382Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1992604Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1992675Z res = mod(**inputs) 2025-12-04T08:53:16.1992953Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1993034Z outputs = self.model( 2025-12-04T08:53:16.1993309Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.1993389Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.1993671Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.1993749Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.1993999Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1994083Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1994348Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1994431Z return func(*args, **kwargs) 2025-12-04T08:53:16.1994704Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T08:53:16.1994817Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:53:16.1995081Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1995153Z return func(*args, **kwargs) 2025-12-04T08:53:16.1995454Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 322, in forward 2025-12-04T08:53:16.1995559Z attn_output, attn_weights = attention_interface( 2025-12-04T08:53:16.1995877Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T08:53:16.1996031Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T08:53:16.1996052Z 2025-12-04T08:53:16.1996163Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1996387Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1996473Z res = mod(**inputs) 2025-12-04T08:53:16.1996755Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.1996838Z outputs = self.model( 2025-12-04T08:53:16.1997116Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.1997202Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.1997482Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.1997580Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.1997837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.1997923Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.1998201Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1998282Z return func(*args, **kwargs) 2025-12-04T08:53:16.1998630Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T08:53:16.1998750Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:53:16.1999016Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.1999089Z return func(*args, **kwargs) 2025-12-04T08:53:16.1999375Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 336, in forward 2025-12-04T08:53:16.1999465Z attn_output = self.out_proj(attn_output) 2025-12-04T08:53:16.1999470Z 2025-12-04T08:53:16.1999587Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.1999801Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.1999871Z res = mod(**inputs) 2025-12-04T08:53:16.2000152Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.2000227Z outputs = self.model( 2025-12-04T08:53:16.2000503Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.2000590Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.2000864Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.2000950Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.2001195Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.2001279Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.2001542Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.2001611Z return func(*args, **kwargs) 2025-12-04T08:53:16.2001862Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 486, in forward 2025-12-04T08:53:16.2001968Z hidden_states = residual + hidden_states 2025-12-04T08:53:16.2001972Z 2025-12-04T08:53:16.2002074Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.2002281Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.2002346Z res = mod(**inputs) 2025-12-04T08:53:16.2002617Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.2002694Z outputs = self.model( 2025-12-04T08:53:16.2002950Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.2003047Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.2003304Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.2003376Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.2003609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.2003685Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.2003938Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.2004033Z return func(*args, **kwargs) 2025-12-04T08:53:16.2004295Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T08:53:16.2004412Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T08:53:16.2004664Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.2004734Z return func(*args, **kwargs) 2025-12-04T08:53:16.2005002Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-12-04T08:53:16.2005155Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T08:53:16.2005159Z 2025-12-04T08:53:16.2005269Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.2005475Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.2005542Z res = mod(**inputs) 2025-12-04T08:53:16.2005813Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.2005882Z outputs = self.model( 2025-12-04T08:53:16.2006146Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.2006237Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.2006496Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.2006572Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.2006796Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.2006872Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.2007125Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.2007194Z return func(*args, **kwargs) 2025-12-04T08:53:16.2007446Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T08:53:16.2007564Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T08:53:16.2007813Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.2007892Z return func(*args, **kwargs) 2025-12-04T08:53:16.2008166Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-12-04T08:53:16.2008249Z key_states = self.k_proj(current_states) 2025-12-04T08:53:16.2008253Z 2025-12-04T08:53:16.2008365Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.2008570Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.2008670Z res = mod(**inputs) 2025-12-04T08:53:16.2008932Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.2009021Z outputs = self.model( 2025-12-04T08:53:16.2009291Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.2009366Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.2009628Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.2009710Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.2009946Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.2010045Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.2010287Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.2010355Z return func(*args, **kwargs) 2025-12-04T08:53:16.2010612Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T08:53:16.2010720Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T08:53:16.2010966Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.2011033Z return func(*args, **kwargs) 2025-12-04T08:53:16.2011282Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 304, in forward 2025-12-04T08:53:16.2011374Z value_states = self.v_proj(current_states) 2025-12-04T08:53:16.2011377Z 2025-12-04T08:53:16.2011459Z cudagraph partition due to non gpu ops 2025-12-04T08:53:16.2011559Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.2011767Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.2011830Z res = mod(**inputs) 2025-12-04T08:53:16.2012103Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.2012171Z outputs = self.model( 2025-12-04T08:53:16.2012436Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.2012516Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.2012767Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.2012838Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.2013067Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.2013147Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.2013395Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.2013462Z return func(*args, **kwargs) 2025-12-04T08:53:16.2013716Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T08:53:16.2013827Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T08:53:16.2014084Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.2014160Z return func(*args, **kwargs) 2025-12-04T08:53:16.2014409Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 322, in forward 2025-12-04T08:53:16.2014508Z attn_output, attn_weights = attention_interface( 2025-12-04T08:53:16.2014822Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T08:53:16.2014956Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T08:53:16.2014960Z 2025-12-04T08:53:16.2015084Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.2015282Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.2015345Z res = mod(**inputs) 2025-12-04T08:53:16.2015609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.2015678Z outputs = self.model( 2025-12-04T08:53:16.2015930Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.2016025Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.2016283Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.2016360Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.2016587Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.2016665Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.2016918Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.2016985Z return func(*args, **kwargs) 2025-12-04T08:53:16.2017243Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T08:53:16.2017356Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T08:53:16.2017602Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.2017677Z return func(*args, **kwargs) 2025-12-04T08:53:16.2017937Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 336, in forward 2025-12-04T08:53:16.2018017Z attn_output = self.out_proj(attn_output) 2025-12-04T08:53:16.2018022Z 2025-12-04T08:53:16.2018130Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.2018330Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.2018400Z res = mod(**inputs) 2025-12-04T08:53:16.2018660Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.2018728Z outputs = self.model( 2025-12-04T08:53:16.2018996Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.2019070Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.2019331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.2019408Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.2019633Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.2019717Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.2019964Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.2020047Z return func(*args, **kwargs) 2025-12-04T08:53:16.2020310Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 508, in forward 2025-12-04T08:53:16.2020430Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:53:16.2020435Z 2025-12-04T08:53:16.2020544Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.2020962Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.2021033Z res = mod(**inputs) 2025-12-04T08:53:16.2021300Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.2021402Z outputs = self.model( 2025-12-04T08:53:16.2021664Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.2021746Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.2022007Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.2022086Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.2022312Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.2022413Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.2022664Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.2022732Z return func(*args, **kwargs) 2025-12-04T08:53:16.2022984Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 508, in forward 2025-12-04T08:53:16.2023109Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:53:16.2023113Z 2025-12-04T08:53:16.2023215Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.2023421Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.2023487Z res = mod(**inputs) 2025-12-04T08:53:16.2023744Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.2023820Z outputs = self.model( 2025-12-04T08:53:16.2024069Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.2024148Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.2024393Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.2024466Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.2024691Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.2024767Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.2025002Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.2025078Z return func(*args, **kwargs) 2025-12-04T08:53:16.2025322Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 510, in forward 2025-12-04T08:53:16.2025411Z hidden_states = self.fc2(hidden_states) 2025-12-04T08:53:16.2025414Z 2025-12-04T08:53:16.2025512Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.2025702Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.2025773Z res = mod(**inputs) 2025-12-04T08:53:16.2026020Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.2026093Z outputs = self.model( 2025-12-04T08:53:16.2026358Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.2026430Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.2026682Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.2026754Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.2026989Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.2027074Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.2027326Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.2027400Z return func(*args, **kwargs) 2025-12-04T08:53:16.2027650Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T08:53:16.2027752Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:53:16.2028000Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.2028070Z return func(*args, **kwargs) 2025-12-04T08:53:16.2028337Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-12-04T08:53:16.2028491Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T08:53:16.2028494Z 2025-12-04T08:53:16.2028592Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.2028794Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.2028856Z res = mod(**inputs) 2025-12-04T08:53:16.2029104Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.2029180Z outputs = self.model( 2025-12-04T08:53:16.2029427Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.2029503Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.2029751Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.2029821Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.2030047Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.2030126Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.2030369Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.2030443Z return func(*args, **kwargs) 2025-12-04T08:53:16.2030694Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T08:53:16.2030799Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:53:16.2031042Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.2031110Z return func(*args, **kwargs) 2025-12-04T08:53:16.2031376Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-12-04T08:53:16.2031457Z key_states = self.k_proj(current_states) 2025-12-04T08:53:16.2031460Z 2025-12-04T08:53:16.2031570Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.2031778Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.2031852Z res = mod(**inputs) 2025-12-04T08:53:16.2032139Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.2032209Z outputs = self.model( 2025-12-04T08:53:16.2032466Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.2032548Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.2032821Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.2032898Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.2033119Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.2033213Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.2033462Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.2033530Z return func(*args, **kwargs) 2025-12-04T08:53:16.2033783Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T08:53:16.2033886Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:53:16.2034125Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.2034216Z return func(*args, **kwargs) 2025-12-04T08:53:16.2034479Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 304, in forward 2025-12-04T08:53:16.2034566Z value_states = self.v_proj(current_states) 2025-12-04T08:53:16.2034571Z 2025-12-04T08:53:16.2034660Z cudagraph partition due to non gpu ops 2025-12-04T08:53:16.2034762Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.2034973Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.2035037Z res = mod(**inputs) 2025-12-04T08:53:16.2035304Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.2035380Z outputs = self.model( 2025-12-04T08:53:16.2035647Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.2035723Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.2035997Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.2036071Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.2036313Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.2036391Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.2036644Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.2036723Z return func(*args, **kwargs) 2025-12-04T08:53:16.2036991Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T08:53:16.2037099Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:53:16.2037368Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.2037442Z return func(*args, **kwargs) 2025-12-04T08:53:16.2037731Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 322, in forward 2025-12-04T08:53:16.2037839Z attn_output, attn_weights = attention_interface( 2025-12-04T08:53:16.2038166Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T08:53:16.2038425Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T08:53:16.2038434Z 2025-12-04T08:53:16.2038553Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.2038779Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.2038852Z res = mod(**inputs) 2025-12-04T08:53:16.2039155Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.2039244Z outputs = self.model( 2025-12-04T08:53:16.2039537Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.2039645Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.2039941Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.2040021Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.2040288Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.2040377Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.2040668Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.2040773Z return func(*args, **kwargs) 2025-12-04T08:53:16.2041052Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T08:53:16.2041167Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:53:16.2041433Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.2041510Z return func(*args, **kwargs) 2025-12-04T08:53:16.2041792Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 336, in forward 2025-12-04T08:53:16.2041882Z attn_output = self.out_proj(attn_output) 2025-12-04T08:53:16.2041886Z 2025-12-04T08:53:16.2041996Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.2042218Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.2042290Z res = mod(**inputs) 2025-12-04T08:53:16.2042586Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.2042659Z outputs = self.model( 2025-12-04T08:53:16.2042933Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.2043020Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.2043293Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.2043378Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.2043620Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.2043705Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.2043976Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.2044051Z return func(*args, **kwargs) 2025-12-04T08:53:16.2044323Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T08:53:16.2044450Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T08:53:16.2044716Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.2044795Z return func(*args, **kwargs) 2025-12-04T08:53:16.2045084Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-12-04T08:53:16.2045249Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T08:53:16.2045253Z 2025-12-04T08:53:16.2045372Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.2045589Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.2045669Z res = mod(**inputs) 2025-12-04T08:53:16.2045963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.2046038Z outputs = self.model( 2025-12-04T08:53:16.2046339Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.2046417Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.2046693Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.2046780Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.2047018Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.2047108Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.2047388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.2047464Z return func(*args, **kwargs) 2025-12-04T08:53:16.2047744Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T08:53:16.2047862Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T08:53:16.2048124Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.2048205Z return func(*args, **kwargs) 2025-12-04T08:53:16.2048478Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-12-04T08:53:16.2048572Z key_states = self.k_proj(current_states) 2025-12-04T08:53:16.2048576Z 2025-12-04T08:53:16.2048684Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.2048900Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.2048977Z res = mod(**inputs) 2025-12-04T08:53:16.2049253Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.2049331Z outputs = self.model( 2025-12-04T08:53:16.2049606Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.2049683Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.2049977Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.2050054Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.2050300Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.2050382Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.2050620Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.2050692Z return func(*args, **kwargs) 2025-12-04T08:53:16.2050935Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T08:53:16.2051041Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T08:53:16.2051293Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.2051359Z return func(*args, **kwargs) 2025-12-04T08:53:16.2051626Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 304, in forward 2025-12-04T08:53:16.2051710Z value_states = self.v_proj(current_states) 2025-12-04T08:53:16.2051714Z 2025-12-04T08:53:16.2051792Z cudagraph partition due to non gpu ops 2025-12-04T08:53:16.2051901Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.2052104Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.2052168Z res = mod(**inputs) 2025-12-04T08:53:16.2052425Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.2052507Z outputs = self.model( 2025-12-04T08:53:16.2052763Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.2052835Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.2053083Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.2053161Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.2053378Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.2053481Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.2053723Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.2053790Z return func(*args, **kwargs) 2025-12-04T08:53:16.2054045Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T08:53:16.2054148Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T08:53:16.2054384Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.2054458Z return func(*args, **kwargs) 2025-12-04T08:53:16.2054704Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 322, in forward 2025-12-04T08:53:16.2054807Z attn_output, attn_weights = attention_interface( 2025-12-04T08:53:16.2055095Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T08:53:16.2055225Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T08:53:16.2055228Z 2025-12-04T08:53:16.2055335Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.2055530Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.2055594Z res = mod(**inputs) 2025-12-04T08:53:16.2055860Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.2055926Z outputs = self.model( 2025-12-04T08:53:16.2056183Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.2056254Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.2056505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.2056581Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.2056802Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.2056886Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.2057120Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.2057187Z return func(*args, **kwargs) 2025-12-04T08:53:16.2057462Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T08:53:16.2057569Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T08:53:16.2057806Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.2057882Z return func(*args, **kwargs) 2025-12-04T08:53:16.2058143Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 336, in forward 2025-12-04T08:53:16.2058230Z attn_output = self.out_proj(attn_output) 2025-12-04T08:53:16.2058247Z 2025-12-04T08:53:16.2058347Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.2058543Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.2058614Z res = mod(**inputs) 2025-12-04T08:53:16.2058866Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.2058938Z outputs = self.model( 2025-12-04T08:53:16.2059186Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.2059276Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.2059529Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.2059600Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.2059816Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.2059900Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.2060137Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.2060211Z return func(*args, **kwargs) 2025-12-04T08:53:16.2060458Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 503, in forward 2025-12-04T08:53:16.2060535Z hidden_states = residual + hidden_states 2025-12-04T08:53:16.2060538Z 2025-12-04T08:53:16.2060646Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.2060840Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.2060904Z res = mod(**inputs) 2025-12-04T08:53:16.2061158Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.2061226Z outputs = self.model( 2025-12-04T08:53:16.2061478Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.2061548Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.2061797Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.2061874Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.2062088Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.2062172Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.2062409Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.2062476Z return func(*args, **kwargs) 2025-12-04T08:53:16.2062725Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 508, in forward 2025-12-04T08:53:16.2062842Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:53:16.2062846Z 2025-12-04T08:53:16.2062944Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.2063159Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.2063223Z res = mod(**inputs) 2025-12-04T08:53:16.2063480Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.2063545Z outputs = self.model( 2025-12-04T08:53:16.2063810Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.2063889Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.2064135Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.2064228Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.2064443Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.2064518Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.2064761Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.2064827Z return func(*args, **kwargs) 2025-12-04T08:53:16.2065072Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 508, in forward 2025-12-04T08:53:16.2065210Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:53:16.2065214Z 2025-12-04T08:53:16.2065313Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.2065520Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.2065584Z res = mod(**inputs) 2025-12-04T08:53:16.2065831Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.2065904Z outputs = self.model( 2025-12-04T08:53:16.2066154Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.2066223Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.2066479Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.2066551Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.2066777Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.2066852Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.2067090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.2067165Z return func(*args, **kwargs) 2025-12-04T08:53:16.2067411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 510, in forward 2025-12-04T08:53:16.2067499Z hidden_states = self.fc2(hidden_states) 2025-12-04T08:53:16.2067503Z 2025-12-04T08:53:16.2067602Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.2067796Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.2067869Z res = mod(**inputs) 2025-12-04T08:53:16.2068119Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.2068184Z outputs = self.model( 2025-12-04T08:53:16.2068437Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.2068509Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.2068763Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.2068833Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.2069067Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.2069151Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.2069387Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.2069466Z return func(*args, **kwargs) 2025-12-04T08:53:16.2069724Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T08:53:16.2069822Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:53:16.2070081Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.2070147Z return func(*args, **kwargs) 2025-12-04T08:53:16.2070397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-12-04T08:53:16.2070552Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T08:53:16.2070556Z 2025-12-04T08:53:16.2070653Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.2070852Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.2070929Z res = mod(**inputs) 2025-12-04T08:53:16.2071177Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.2071252Z outputs = self.model( 2025-12-04T08:53:16.2071495Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.2071574Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.2071824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.2071896Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.2072123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.2072200Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.2072441Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.2072520Z return func(*args, **kwargs) 2025-12-04T08:53:16.2072770Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T08:53:16.2072874Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:53:16.2073122Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.2073189Z return func(*args, **kwargs) 2025-12-04T08:53:16.2073449Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-12-04T08:53:16.2073529Z key_states = self.k_proj(current_states) 2025-12-04T08:53:16.2073532Z 2025-12-04T08:53:16.2073639Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.2073837Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.2073903Z res = mod(**inputs) 2025-12-04T08:53:16.2074162Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.2074229Z outputs = self.model( 2025-12-04T08:53:16.2074483Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.2074563Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.2074827Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.2074908Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.2075130Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.2075207Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.2075483Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.2075554Z return func(*args, **kwargs) 2025-12-04T08:53:16.2075814Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T08:53:16.2075948Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:53:16.2076199Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.2076276Z return func(*args, **kwargs) 2025-12-04T08:53:16.2076538Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 304, in forward 2025-12-04T08:53:16.2076627Z value_states = self.v_proj(current_states) 2025-12-04T08:53:16.2076630Z 2025-12-04T08:53:16.2076721Z cudagraph partition due to non gpu ops 2025-12-04T08:53:16.2076842Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.2077053Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.2077118Z res = mod(**inputs) 2025-12-04T08:53:16.2077381Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.2077459Z outputs = self.model( 2025-12-04T08:53:16.2077717Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.2077790Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.2078057Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.2078129Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.2078422Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.2078512Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.2078763Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.2078843Z return func(*args, **kwargs) 2025-12-04T08:53:16.2079127Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T08:53:16.2079234Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:53:16.2079514Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.2079589Z return func(*args, **kwargs) 2025-12-04T08:53:16.2079888Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 322, in forward 2025-12-04T08:53:16.2079996Z attn_output, attn_weights = attention_interface( 2025-12-04T08:53:16.2080335Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T08:53:16.2080489Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T08:53:16.2080493Z 2025-12-04T08:53:16.2080601Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.2080831Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.2080897Z res = mod(**inputs) 2025-12-04T08:53:16.2081177Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.2081256Z outputs = self.model( 2025-12-04T08:53:16.2081518Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.2081594Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.2081878Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.2081954Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.2082191Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.2082288Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.2082537Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.2082614Z return func(*args, **kwargs) 2025-12-04T08:53:16.2082891Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 477, in forward 2025-12-04T08:53:16.2083003Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:53:16.2083266Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.2083361Z return func(*args, **kwargs) 2025-12-04T08:53:16.2083655Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 336, in forward 2025-12-04T08:53:16.2083743Z attn_output = self.out_proj(attn_output) 2025-12-04T08:53:16.2083747Z 2025-12-04T08:53:16.2083858Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.2084085Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.2084153Z res = mod(**inputs) 2025-12-04T08:53:16.2084441Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.2084515Z outputs = self.model( 2025-12-04T08:53:16.2084796Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.2084886Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.2085167Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.2085243Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.2085495Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.2085581Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.2085856Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.2085930Z return func(*args, **kwargs) 2025-12-04T08:53:16.2086209Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T08:53:16.2086337Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T08:53:16.2086602Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.2086684Z return func(*args, **kwargs) 2025-12-04T08:53:16.2086964Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 283, in forward 2025-12-04T08:53:16.2087128Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T08:53:16.2087134Z 2025-12-04T08:53:16.2087252Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.2087466Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.2087534Z res = mod(**inputs) 2025-12-04T08:53:16.2087838Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.2087912Z outputs = self.model( 2025-12-04T08:53:16.2088203Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.2088283Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.2088576Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.2088664Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.2088926Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.2089017Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.2089282Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.2089358Z return func(*args, **kwargs) 2025-12-04T08:53:16.2089643Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T08:53:16.2089758Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T08:53:16.2090038Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.2090119Z return func(*args, **kwargs) 2025-12-04T08:53:16.2090393Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 303, in forward 2025-12-04T08:53:16.2090488Z key_states = self.k_proj(current_states) 2025-12-04T08:53:16.2090491Z 2025-12-04T08:53:16.2090601Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.2090816Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.2090894Z res = mod(**inputs) 2025-12-04T08:53:16.2091172Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.2091255Z outputs = self.model( 2025-12-04T08:53:16.2091529Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.2091610Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.2091892Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.2091969Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.2092222Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.2092314Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.2092579Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.2092659Z return func(*args, **kwargs) 2025-12-04T08:53:16.2092937Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T08:53:16.2093054Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T08:53:16.2093335Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.2093404Z return func(*args, **kwargs) 2025-12-04T08:53:16.2093662Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 304, in forward 2025-12-04T08:53:16.2093760Z value_states = self.v_proj(current_states) 2025-12-04T08:53:16.2093764Z 2025-12-04T08:53:16.2093847Z cudagraph partition due to non gpu ops 2025-12-04T08:53:16.2093959Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.2094182Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.2094248Z res = mod(**inputs) 2025-12-04T08:53:16.2094526Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.2094601Z outputs = self.model( 2025-12-04T08:53:16.2094896Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.2094977Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.2095255Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.2095354Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.2095584Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.2095663Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.2095922Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.2095992Z return func(*args, **kwargs) 2025-12-04T08:53:16.2096261Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T08:53:16.2096392Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T08:53:16.2096643Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.2096720Z return func(*args, **kwargs) 2025-12-04T08:53:16.2096984Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 322, in forward 2025-12-04T08:53:16.2097091Z attn_output, attn_weights = attention_interface( 2025-12-04T08:53:16.2097393Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T08:53:16.2097529Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T08:53:16.2097533Z 2025-12-04T08:53:16.2097643Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.2097849Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.2097917Z res = mod(**inputs) 2025-12-04T08:53:16.2098188Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.2098257Z outputs = self.model( 2025-12-04T08:53:16.2098526Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.2098599Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.2098858Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.2098936Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.2099163Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.2099244Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.2099504Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.2099575Z return func(*args, **kwargs) 2025-12-04T08:53:16.2099843Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 494, in forward 2025-12-04T08:53:16.2099954Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-12-04T08:53:16.2100203Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.2100279Z return func(*args, **kwargs) 2025-12-04T08:53:16.2100556Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 336, in forward 2025-12-04T08:53:16.2100648Z attn_output = self.out_proj(attn_output) 2025-12-04T08:53:16.2100651Z 2025-12-04T08:53:16.2100757Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.2100989Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.2101065Z res = mod(**inputs) 2025-12-04T08:53:16.2101329Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.2101415Z outputs = self.model( 2025-12-04T08:53:16.2101685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.2101758Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.2102032Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.2102106Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.2102336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.2102439Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.2102701Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.2102779Z return func(*args, **kwargs) 2025-12-04T08:53:16.2103046Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 508, in forward 2025-12-04T08:53:16.2103166Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:53:16.2103170Z 2025-12-04T08:53:16.2103278Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.2103480Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.2103544Z res = mod(**inputs) 2025-12-04T08:53:16.2103807Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.2103874Z outputs = self.model( 2025-12-04T08:53:16.2104134Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.2104206Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.2104459Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.2104539Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.2104759Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.2104843Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.2105087Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.2105155Z return func(*args, **kwargs) 2025-12-04T08:53:16.2105414Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 508, in forward 2025-12-04T08:53:16.2105532Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:53:16.2105537Z 2025-12-04T08:53:16.2105639Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.2105845Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.2105911Z res = mod(**inputs) 2025-12-04T08:53:16.2106170Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.2106237Z outputs = self.model( 2025-12-04T08:53:16.2106505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.2106587Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.2106847Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.2106918Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.2107161Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.2107238Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.2107479Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.2107563Z return func(*args, **kwargs) 2025-12-04T08:53:16.2107809Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 510, in forward 2025-12-04T08:53:16.2107897Z hidden_states = self.fc2(hidden_states) 2025-12-04T08:53:16.2107901Z 2025-12-04T08:53:16.2108001Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.2108201Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.2108280Z res = mod(**inputs) 2025-12-04T08:53:16.2108536Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1401, in forward 2025-12-04T08:53:16.2108611Z outputs = self.model( 2025-12-04T08:53:16.2108865Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1276, in forward 2025-12-04T08:53:16.2108937Z decoder_outputs = self.decoder( 2025-12-04T08:53:16.2109198Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1130, in forward 2025-12-04T08:53:16.2109269Z layer_outputs = decoder_layer( 2025-12-04T08:53:16.2109498Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:16.2109575Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:16.2109818Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:16.2109892Z return func(*args, **kwargs) 2025-12-04T08:53:16.2110143Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 512, in forward 2025-12-04T08:53:16.2110229Z hidden_states = residual + hidden_states 2025-12-04T08:53:16.2110232Z 2025-12-04T08:53:16.2110334Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.2110529Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.2110600Z res = mod(**inputs) 2025-12-04T08:53:16.2110854Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1419, in forward 2025-12-04T08:53:16.2110933Z lm_logits = self.lm_head(outputs[0]) 2025-12-04T08:53:16.2110936Z 2025-12-04T08:53:16.2111044Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:16.2111240Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:16.2111312Z res = mod(**inputs) 2025-12-04T08:53:16.2111570Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1426, in forward 2025-12-04T08:53:16.2111739Z masked_lm_loss = loss_fct(lm_logits.view(-1, self.config.vocab_size), labels.view(-1)) 2025-12-04T08:53:16.2111744Z 2025-12-04T08:53:28.7400828Z Compilation time (from dynamo_timed): 27.308629702 2025-12-04T08:53:28.7504057Z pass 2025-12-04T08:53:28.7504656Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T08:53:28.7506322Z TIMING: _recursive_pre_grad_passes:0.01322 _recursive_joint_graph_passes:1.12891 _recursive_post_grad_passes:0.12711 async_compile.wait:1.16531 code_gen:11.81677 inductor_compile:14.90925 backend_compile:21.58162 gc:0.00064 entire_frame_compile:27.30863 total_wall_time:27.30863 2025-12-04T08:53:28.7508040Z STATS: call_* op count: 906 | FakeTensorMode.__torch_dispatch__:18114 | FakeTensor.__torch_dispatch__:10222 | ProxyTorchDispatchMode.__torch_dispatch__:4990 2025-12-04T08:53:28.7509027Z Dynamo produced 1 graphs covering 906 ops with 0 graph breaks (0 unique) 2025-12-04T08:53:31.7637423Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/__init__.py:65: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-12-04T08:53:31.7639509Z import pynvml # type: ignore[import] 2025-12-04T08:53:35.2350931Z 2025-12-04T08:53:38.5216825Z loading model: 0it [00:00, ?it/s] 2025-12-04T08:53:38.5217155Z loading model: 0it [00:03, ?it/s] 2025-12-04T08:53:38.5230887Z cpu eval MBartForCausalLM 2025-12-04T08:53:43.2655081Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T08:53:45.3652045Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T08:53:47.4153581Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T08:53:54.8149746Z cudagraph partition due to non gpu ops 2025-12-04T08:53:54.8150126Z cudagraph partition due to non gpu ops 2025-12-04T08:53:54.8150401Z cudagraph partition due to non gpu ops 2025-12-04T08:53:54.8150635Z cudagraph partition due to non gpu ops 2025-12-04T08:53:54.8150858Z cudagraph partition due to non gpu ops 2025-12-04T08:53:54.8151135Z cudagraph partition due to non gpu ops 2025-12-04T08:53:54.8151378Z cudagraph partition due to non gpu ops 2025-12-04T08:53:54.8151609Z cudagraph partition due to non gpu ops 2025-12-04T08:53:54.8151838Z cudagraph partition due to non gpu ops 2025-12-04T08:53:54.8152071Z cudagraph partition due to non gpu ops 2025-12-04T08:53:54.8152291Z cudagraph partition due to non gpu ops 2025-12-04T08:53:54.8152524Z cudagraph partition due to non gpu ops 2025-12-04T08:53:54.8152807Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:54.8153244Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:54.8153665Z res = mod(**inputs) 2025-12-04T08:53:54.8154123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T08:53:54.8154585Z outputs = self.model.decoder( 2025-12-04T08:53:54.8155023Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T08:53:54.8155468Z layer_outputs = decoder_layer( 2025-12-04T08:53:54.8155881Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:54.8156302Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:54.8156739Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8157181Z return func(*args, **kwargs) 2025-12-04T08:53:54.8157615Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T08:53:54.8158135Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:53:54.8158886Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8159314Z return func(*args, **kwargs) 2025-12-04T08:53:54.8160118Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 228, in forward 2025-12-04T08:53:54.8160643Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T08:53:54.8160872Z 2025-12-04T08:53:54.8161002Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:54.8161401Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:54.8161780Z res = mod(**inputs) 2025-12-04T08:53:54.8162260Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T08:53:54.8162703Z outputs = self.model.decoder( 2025-12-04T08:53:54.8163195Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T08:53:54.8163645Z layer_outputs = decoder_layer( 2025-12-04T08:53:54.8164065Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:54.8164465Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:54.8164894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8165302Z return func(*args, **kwargs) 2025-12-04T08:53:54.8165765Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T08:53:54.8166216Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:53:54.8166672Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8167086Z return func(*args, **kwargs) 2025-12-04T08:53:54.8167488Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 248, in forward 2025-12-04T08:53:54.8167927Z key_states = self.k_proj(current_states) 2025-12-04T08:53:54.8168081Z 2025-12-04T08:53:54.8168201Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:54.8168597Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:54.8168947Z res = mod(**inputs) 2025-12-04T08:53:54.8169348Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T08:53:54.8169782Z outputs = self.model.decoder( 2025-12-04T08:53:54.8170202Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T08:53:54.8170626Z layer_outputs = decoder_layer( 2025-12-04T08:53:54.8171025Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:54.8171425Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:54.8171847Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8172258Z return func(*args, **kwargs) 2025-12-04T08:53:54.8172667Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T08:53:54.8173123Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:53:54.8173551Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8173952Z return func(*args, **kwargs) 2025-12-04T08:53:54.8174362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 249, in forward 2025-12-04T08:53:54.8174793Z value_states = self.v_proj(current_states) 2025-12-04T08:53:54.8174957Z 2025-12-04T08:53:54.8175045Z cudagraph partition due to non gpu ops 2025-12-04T08:53:54.8175309Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:54.8175724Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:54.8176073Z res = mod(**inputs) 2025-12-04T08:53:54.8176476Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T08:53:54.8176917Z outputs = self.model.decoder( 2025-12-04T08:53:54.8177354Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T08:53:54.8177800Z layer_outputs = decoder_layer( 2025-12-04T08:53:54.8178187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:54.8178604Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:54.8179011Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8179421Z return func(*args, **kwargs) 2025-12-04T08:53:54.8179835Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T08:53:54.8180259Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:53:54.8180668Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8181124Z return func(*args, **kwargs) 2025-12-04T08:53:54.8181512Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 267, in forward 2025-12-04T08:53:54.8181933Z attn_output, attn_weights = attention_interface( 2025-12-04T08:53:54.8182403Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T08:53:54.8182909Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T08:53:54.8183101Z 2025-12-04T08:53:54.8183218Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:54.8183587Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:54.8183922Z res = mod(**inputs) 2025-12-04T08:53:54.8184305Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T08:53:54.8184719Z outputs = self.model.decoder( 2025-12-04T08:53:54.8185107Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T08:53:54.8185512Z layer_outputs = decoder_layer( 2025-12-04T08:53:54.8185876Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:54.8186247Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:54.8186641Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8187029Z return func(*args, **kwargs) 2025-12-04T08:53:54.8187420Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T08:53:54.8187849Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:53:54.8188269Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8188655Z return func(*args, **kwargs) 2025-12-04T08:53:54.8189036Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 281, in forward 2025-12-04T08:53:54.8189449Z attn_output = self.out_proj(attn_output) 2025-12-04T08:53:54.8189599Z 2025-12-04T08:53:54.8189708Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:54.8190083Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:54.8190419Z res = mod(**inputs) 2025-12-04T08:53:54.8190828Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T08:53:54.8191271Z outputs = self.model.decoder( 2025-12-04T08:53:54.8191693Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T08:53:54.8192123Z layer_outputs = decoder_layer( 2025-12-04T08:53:54.8192519Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:54.8193159Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:54.8193586Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8193998Z return func(*args, **kwargs) 2025-12-04T08:53:54.8194415Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 451, in forward 2025-12-04T08:53:54.8194896Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:53:54.8195090Z 2025-12-04T08:53:54.8195207Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:54.8195598Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:54.8195973Z res = mod(**inputs) 2025-12-04T08:53:54.8196385Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T08:53:54.8196805Z outputs = self.model.decoder( 2025-12-04T08:53:54.8197243Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T08:53:54.8197677Z layer_outputs = decoder_layer( 2025-12-04T08:53:54.8198058Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:54.8198637Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:54.8199057Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8199465Z return func(*args, **kwargs) 2025-12-04T08:53:54.8199870Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 451, in forward 2025-12-04T08:53:54.8200350Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:53:54.8200784Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T08:53:54.8201168Z return self.act(input) 2025-12-04T08:53:54.8201293Z 2025-12-04T08:53:54.8201408Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:54.8201801Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:54.8202157Z res = mod(**inputs) 2025-12-04T08:53:54.8202550Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T08:53:54.8202982Z outputs = self.model.decoder( 2025-12-04T08:53:54.8203404Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T08:53:54.8203834Z layer_outputs = decoder_layer( 2025-12-04T08:53:54.8204212Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:54.8204612Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:54.8205032Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8205441Z return func(*args, **kwargs) 2025-12-04T08:53:54.8205901Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 453, in forward 2025-12-04T08:53:54.8206368Z hidden_states = self.fc2(hidden_states) 2025-12-04T08:53:54.8206521Z 2025-12-04T08:53:54.8206642Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:54.8207026Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:54.8207381Z res = mod(**inputs) 2025-12-04T08:53:54.8207794Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T08:53:54.8208216Z outputs = self.model.decoder( 2025-12-04T08:53:54.8208636Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T08:53:54.8209132Z layer_outputs = decoder_layer( 2025-12-04T08:53:54.8209517Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:54.8209905Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:54.8210300Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8210687Z return func(*args, **kwargs) 2025-12-04T08:53:54.8211083Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T08:53:54.8211558Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:53:54.8212010Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8212434Z return func(*args, **kwargs) 2025-12-04T08:53:54.8212846Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 228, in forward 2025-12-04T08:53:54.8213365Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T08:53:54.8213600Z 2025-12-04T08:53:54.8213712Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:54.8214089Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:54.8214420Z res = mod(**inputs) 2025-12-04T08:53:54.8214803Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T08:53:54.8215217Z outputs = self.model.decoder( 2025-12-04T08:53:54.8215613Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T08:53:54.8216022Z layer_outputs = decoder_layer( 2025-12-04T08:53:54.8216388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:54.8216768Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:54.8217161Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8217555Z return func(*args, **kwargs) 2025-12-04T08:53:54.8217949Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T08:53:54.8218382Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:53:54.8218801Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8219197Z return func(*args, **kwargs) 2025-12-04T08:53:54.8219591Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 248, in forward 2025-12-04T08:53:54.8220000Z key_states = self.k_proj(current_states) 2025-12-04T08:53:54.8220153Z 2025-12-04T08:53:54.8220264Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:54.8220642Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:54.8221220Z res = mod(**inputs) 2025-12-04T08:53:54.8221649Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T08:53:54.8222059Z outputs = self.model.decoder( 2025-12-04T08:53:54.8222459Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T08:53:54.8222863Z layer_outputs = decoder_layer( 2025-12-04T08:53:54.8223250Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:54.8223633Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:54.8224060Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8224439Z return func(*args, **kwargs) 2025-12-04T08:53:54.8224828Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T08:53:54.8225258Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:53:54.8225672Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8226050Z return func(*args, **kwargs) 2025-12-04T08:53:54.8226472Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 249, in forward 2025-12-04T08:53:54.8226888Z value_states = self.v_proj(current_states) 2025-12-04T08:53:54.8227034Z 2025-12-04T08:53:54.8227137Z cudagraph partition due to non gpu ops 2025-12-04T08:53:54.8227384Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:54.8227756Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:54.8228089Z res = mod(**inputs) 2025-12-04T08:53:54.8228457Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T08:53:54.8228883Z outputs = self.model.decoder( 2025-12-04T08:53:54.8229298Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T08:53:54.8229716Z layer_outputs = decoder_layer( 2025-12-04T08:53:54.8230099Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:54.8230502Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:54.8230894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8231293Z return func(*args, **kwargs) 2025-12-04T08:53:54.8231705Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T08:53:54.8232183Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:53:54.8232624Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8233037Z return func(*args, **kwargs) 2025-12-04T08:53:54.8233445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 267, in forward 2025-12-04T08:53:54.8233899Z attn_output, attn_weights = attention_interface( 2025-12-04T08:53:54.8234386Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T08:53:54.8234914Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T08:53:54.8235123Z 2025-12-04T08:53:54.8235237Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:54.8235627Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:54.8235971Z res = mod(**inputs) 2025-12-04T08:53:54.8236382Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T08:53:54.8236844Z outputs = self.model.decoder( 2025-12-04T08:53:54.8237286Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T08:53:54.8237722Z layer_outputs = decoder_layer( 2025-12-04T08:53:54.8238136Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:54.8238613Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:54.8239033Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8239485Z return func(*args, **kwargs) 2025-12-04T08:53:54.8239894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T08:53:54.8240445Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:53:54.8240882Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8241289Z return func(*args, **kwargs) 2025-12-04T08:53:54.8241734Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 281, in forward 2025-12-04T08:53:54.8242195Z attn_output = self.out_proj(attn_output) 2025-12-04T08:53:54.8242355Z 2025-12-04T08:53:54.8242469Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:54.8242861Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:54.8243214Z res = mod(**inputs) 2025-12-04T08:53:54.8243605Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T08:53:54.8244003Z outputs = self.model.decoder( 2025-12-04T08:53:54.8244388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T08:53:54.8244778Z layer_outputs = decoder_layer( 2025-12-04T08:53:54.8245123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:54.8245493Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:54.8245873Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8246237Z return func(*args, **kwargs) 2025-12-04T08:53:54.8246613Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 451, in forward 2025-12-04T08:53:54.8247049Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:53:54.8247221Z 2025-12-04T08:53:54.8247333Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:54.8247685Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:54.8248010Z res = mod(**inputs) 2025-12-04T08:53:54.8248371Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T08:53:54.8248759Z outputs = self.model.decoder( 2025-12-04T08:53:54.8249142Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T08:53:54.8249531Z layer_outputs = decoder_layer( 2025-12-04T08:53:54.8249882Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:54.8250239Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:54.8250621Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8251025Z return func(*args, **kwargs) 2025-12-04T08:53:54.8251485Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 451, in forward 2025-12-04T08:53:54.8251926Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:53:54.8252330Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T08:53:54.8252691Z return self.act(input) 2025-12-04T08:53:54.8252818Z 2025-12-04T08:53:54.8252927Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:54.8253312Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:54.8253654Z res = mod(**inputs) 2025-12-04T08:53:54.8254020Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T08:53:54.8254409Z outputs = self.model.decoder( 2025-12-04T08:53:54.8254799Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T08:53:54.8255193Z layer_outputs = decoder_layer( 2025-12-04T08:53:54.8255538Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:54.8255920Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:54.8256301Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8256674Z return func(*args, **kwargs) 2025-12-04T08:53:54.8257042Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 453, in forward 2025-12-04T08:53:54.8257440Z hidden_states = self.fc2(hidden_states) 2025-12-04T08:53:54.8257582Z 2025-12-04T08:53:54.8257699Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:54.8258068Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:54.8258394Z res = mod(**inputs) 2025-12-04T08:53:54.8258765Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T08:53:54.8259165Z outputs = self.model.decoder( 2025-12-04T08:53:54.8259551Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T08:53:54.8259963Z layer_outputs = decoder_layer( 2025-12-04T08:53:54.8260313Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:54.8260683Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:54.8261064Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8261449Z return func(*args, **kwargs) 2025-12-04T08:53:54.8261835Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 455, in forward 2025-12-04T08:53:54.8262234Z hidden_states = residual + hidden_states 2025-12-04T08:53:54.8262381Z 2025-12-04T08:53:54.8262488Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:54.8262860Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:54.8263193Z res = mod(**inputs) 2025-12-04T08:53:54.8263556Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T08:53:54.8263958Z outputs = self.model.decoder( 2025-12-04T08:53:54.8264350Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T08:53:54.8264740Z layer_outputs = decoder_layer( 2025-12-04T08:53:54.8265115Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:54.8265490Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:54.8265878Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8266251Z return func(*args, **kwargs) 2025-12-04T08:53:54.8266653Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T08:53:54.8267086Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:53:54.8267509Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8267904Z return func(*args, **kwargs) 2025-12-04T08:53:54.8268290Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 228, in forward 2025-12-04T08:53:54.8268773Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T08:53:54.8268980Z 2025-12-04T08:53:54.8269089Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:54.8269458Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:54.8269785Z res = mod(**inputs) 2025-12-04T08:53:54.8270179Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T08:53:54.8270576Z outputs = self.model.decoder( 2025-12-04T08:53:54.8270967Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T08:53:54.8271369Z layer_outputs = decoder_layer( 2025-12-04T08:53:54.8271731Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:54.8272101Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:54.8272491Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8272870Z return func(*args, **kwargs) 2025-12-04T08:53:54.8273246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T08:53:54.8273675Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:53:54.8274099Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8274512Z return func(*args, **kwargs) 2025-12-04T08:53:54.8274909Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 248, in forward 2025-12-04T08:53:54.8275546Z key_states = self.k_proj(current_states) 2025-12-04T08:53:54.8275697Z 2025-12-04T08:53:54.8275818Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:54.8276207Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:54.8276560Z res = mod(**inputs) 2025-12-04T08:53:54.8276976Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T08:53:54.8277410Z outputs = self.model.decoder( 2025-12-04T08:53:54.8277824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T08:53:54.8278338Z layer_outputs = decoder_layer( 2025-12-04T08:53:54.8278786Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:54.8279219Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:54.8279632Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8280057Z return func(*args, **kwargs) 2025-12-04T08:53:54.8280501Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T08:53:54.8280987Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:53:54.8281428Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8281838Z return func(*args, **kwargs) 2025-12-04T08:53:54.8282271Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 249, in forward 2025-12-04T08:53:54.8282705Z value_states = self.v_proj(current_states) 2025-12-04T08:53:54.8282888Z 2025-12-04T08:53:54.8282979Z cudagraph partition due to non gpu ops 2025-12-04T08:53:54.8283240Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:54.8283628Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:54.8283986Z res = mod(**inputs) 2025-12-04T08:53:54.8284386Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T08:53:54.8284817Z outputs = self.model.decoder( 2025-12-04T08:53:54.8285228Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T08:53:54.8285675Z layer_outputs = decoder_layer( 2025-12-04T08:53:54.8286058Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:54.8286447Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:54.8286868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8287290Z return func(*args, **kwargs) 2025-12-04T08:53:54.8287700Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T08:53:54.8288149Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:53:54.8288586Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8288991Z return func(*args, **kwargs) 2025-12-04T08:53:54.8289398Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 267, in forward 2025-12-04T08:53:54.8289839Z attn_output, attn_weights = attention_interface( 2025-12-04T08:53:54.8290328Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T08:53:54.8290863Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T08:53:54.8291064Z 2025-12-04T08:53:54.8291177Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:54.8291572Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:54.8291924Z res = mod(**inputs) 2025-12-04T08:53:54.8292318Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T08:53:54.8292745Z outputs = self.model.decoder( 2025-12-04T08:53:54.8293171Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T08:53:54.8293592Z layer_outputs = decoder_layer( 2025-12-04T08:53:54.8293978Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:54.8294370Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:54.8294882Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8295270Z return func(*args, **kwargs) 2025-12-04T08:53:54.8295678Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T08:53:54.8296112Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:53:54.8296530Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8296922Z return func(*args, **kwargs) 2025-12-04T08:53:54.8297338Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 281, in forward 2025-12-04T08:53:54.8297751Z attn_output = self.out_proj(attn_output) 2025-12-04T08:53:54.8297894Z 2025-12-04T08:53:54.8298011Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:54.8298408Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:54.8298755Z res = mod(**inputs) 2025-12-04T08:53:54.8299128Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T08:53:54.8299535Z outputs = self.model.decoder( 2025-12-04T08:53:54.8299919Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T08:53:54.8300325Z layer_outputs = decoder_layer( 2025-12-04T08:53:54.8300707Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:54.8301094Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:54.8301481Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8301863Z return func(*args, **kwargs) 2025-12-04T08:53:54.8302248Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 451, in forward 2025-12-04T08:53:54.8302685Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:53:54.8302871Z 2025-12-04T08:53:54.8302978Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:54.8303348Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:54.8303678Z res = mod(**inputs) 2025-12-04T08:53:54.8304046Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T08:53:54.8304450Z outputs = self.model.decoder( 2025-12-04T08:53:54.8304845Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T08:53:54.8305242Z layer_outputs = decoder_layer( 2025-12-04T08:53:54.8305602Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:54.8305976Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:54.8306372Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8306748Z return func(*args, **kwargs) 2025-12-04T08:53:54.8307140Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 451, in forward 2025-12-04T08:53:54.8307575Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:53:54.8307972Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T08:53:54.8308318Z return self.act(input) 2025-12-04T08:53:54.8308439Z 2025-12-04T08:53:54.8308544Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:54.8308918Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:54.8309245Z res = mod(**inputs) 2025-12-04T08:53:54.8309619Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T08:53:54.8310039Z outputs = self.model.decoder( 2025-12-04T08:53:54.8310441Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T08:53:54.8310828Z layer_outputs = decoder_layer( 2025-12-04T08:53:54.8311184Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:54.8311571Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:54.8311949Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8312345Z return func(*args, **kwargs) 2025-12-04T08:53:54.8312721Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 453, in forward 2025-12-04T08:53:54.8313129Z hidden_states = self.fc2(hidden_states) 2025-12-04T08:53:54.8313269Z 2025-12-04T08:53:54.8313375Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:54.8313739Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:54.8314064Z res = mod(**inputs) 2025-12-04T08:53:54.8314436Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T08:53:54.8314846Z outputs = self.model.decoder( 2025-12-04T08:53:54.8315231Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T08:53:54.8315695Z layer_outputs = decoder_layer( 2025-12-04T08:53:54.8316045Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:54.8316413Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:54.8316797Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8317175Z return func(*args, **kwargs) 2025-12-04T08:53:54.8317543Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T08:53:54.8317959Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:53:54.8318417Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8318835Z return func(*args, **kwargs) 2025-12-04T08:53:54.8319269Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 228, in forward 2025-12-04T08:53:54.8319792Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T08:53:54.8320014Z 2025-12-04T08:53:54.8320134Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:54.8320528Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:54.8321017Z res = mod(**inputs) 2025-12-04T08:53:54.8321400Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T08:53:54.8321806Z outputs = self.model.decoder( 2025-12-04T08:53:54.8322195Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T08:53:54.8322602Z layer_outputs = decoder_layer( 2025-12-04T08:53:54.8322963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:54.8323334Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:54.8323737Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8324124Z return func(*args, **kwargs) 2025-12-04T08:53:54.8324563Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T08:53:54.8324987Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:53:54.8325402Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8325785Z return func(*args, **kwargs) 2025-12-04T08:53:54.8326197Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 248, in forward 2025-12-04T08:53:54.8326617Z key_states = self.k_proj(current_states) 2025-12-04T08:53:54.8326768Z 2025-12-04T08:53:54.8326875Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:54.8327276Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:54.8327604Z res = mod(**inputs) 2025-12-04T08:53:54.8327977Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T08:53:54.8328382Z outputs = self.model.decoder( 2025-12-04T08:53:54.8328775Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T08:53:54.8329176Z layer_outputs = decoder_layer( 2025-12-04T08:53:54.8329559Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:54.8329935Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:54.8330322Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8330705Z return func(*args, **kwargs) 2025-12-04T08:53:54.8331089Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T08:53:54.8331511Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:53:54.8331916Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8332301Z return func(*args, **kwargs) 2025-12-04T08:53:54.8332685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 249, in forward 2025-12-04T08:53:54.8333092Z value_states = self.v_proj(current_states) 2025-12-04T08:53:54.8333245Z 2025-12-04T08:53:54.8333329Z cudagraph partition due to non gpu ops 2025-12-04T08:53:54.8333578Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:54.8333944Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:54.8334269Z res = mod(**inputs) 2025-12-04T08:53:54.8334641Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T08:53:54.8335043Z outputs = self.model.decoder( 2025-12-04T08:53:54.8335430Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T08:53:54.8335827Z layer_outputs = decoder_layer( 2025-12-04T08:53:54.8336186Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:54.8336558Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:54.8336942Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8337324Z return func(*args, **kwargs) 2025-12-04T08:53:54.8337703Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T08:53:54.8338128Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:53:54.8338531Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8338933Z return func(*args, **kwargs) 2025-12-04T08:53:54.8339324Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 267, in forward 2025-12-04T08:53:54.8339746Z attn_output, attn_weights = attention_interface( 2025-12-04T08:53:54.8340240Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T08:53:54.8340745Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T08:53:54.8340935Z 2025-12-04T08:53:54.8341050Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:54.8341434Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:54.8341766Z res = mod(**inputs) 2025-12-04T08:53:54.8342146Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T08:53:54.8342573Z outputs = self.model.decoder( 2025-12-04T08:53:54.8342982Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T08:53:54.8343387Z layer_outputs = decoder_layer( 2025-12-04T08:53:54.8343747Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:54.8344132Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:54.8344530Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8344918Z return func(*args, **kwargs) 2025-12-04T08:53:54.8345309Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T08:53:54.8345729Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:53:54.8346142Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8346529Z return func(*args, **kwargs) 2025-12-04T08:53:54.8346910Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 281, in forward 2025-12-04T08:53:54.8347321Z attn_output = self.out_proj(attn_output) 2025-12-04T08:53:54.8347469Z 2025-12-04T08:53:54.8347578Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:54.8347950Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:54.8348276Z res = mod(**inputs) 2025-12-04T08:53:54.8348655Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T08:53:54.8349069Z outputs = self.model.decoder( 2025-12-04T08:53:54.8349486Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T08:53:54.8349902Z layer_outputs = decoder_layer( 2025-12-04T08:53:54.8350262Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:54.8350639Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:54.8351028Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8351413Z return func(*args, **kwargs) 2025-12-04T08:53:54.8351797Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 451, in forward 2025-12-04T08:53:54.8352257Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:53:54.8352445Z 2025-12-04T08:53:54.8352557Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:54.8352948Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:54.8353329Z res = mod(**inputs) 2025-12-04T08:53:54.8353720Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T08:53:54.8354150Z outputs = self.model.decoder( 2025-12-04T08:53:54.8354566Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T08:53:54.8355007Z layer_outputs = decoder_layer( 2025-12-04T08:53:54.8355381Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:54.8355778Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:54.8356212Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8356615Z return func(*args, **kwargs) 2025-12-04T08:53:54.8357018Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 451, in forward 2025-12-04T08:53:54.8357487Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:53:54.8357911Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T08:53:54.8358354Z return self.act(input) 2025-12-04T08:53:54.8358514Z 2025-12-04T08:53:54.8358631Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:54.8359032Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:54.8359387Z res = mod(**inputs) 2025-12-04T08:53:54.8359779Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T08:53:54.8360212Z outputs = self.model.decoder( 2025-12-04T08:53:54.8360635Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T08:53:54.8361057Z layer_outputs = decoder_layer( 2025-12-04T08:53:54.8361442Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:54.8361843Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:54.8362259Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8362668Z return func(*args, **kwargs) 2025-12-04T08:53:54.8363079Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 453, in forward 2025-12-04T08:53:54.8363511Z hidden_states = self.fc2(hidden_states) 2025-12-04T08:53:54.8363663Z 2025-12-04T08:53:54.8363784Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:54.8364173Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:54.8364536Z res = mod(**inputs) 2025-12-04T08:53:54.8364943Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T08:53:54.8365382Z outputs = self.model.decoder( 2025-12-04T08:53:54.8365809Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T08:53:54.8366249Z layer_outputs = decoder_layer( 2025-12-04T08:53:54.8366647Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:54.8367050Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:54.8367479Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8367901Z return func(*args, **kwargs) 2025-12-04T08:53:54.8368308Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 455, in forward 2025-12-04T08:53:54.8368741Z hidden_states = residual + hidden_states 2025-12-04T08:53:54.8368889Z 2025-12-04T08:53:54.8368995Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:54.8369363Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:54.8369691Z res = mod(**inputs) 2025-12-04T08:53:54.8370084Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T08:53:54.8370500Z outputs = self.model.decoder( 2025-12-04T08:53:54.8370888Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T08:53:54.8371303Z layer_outputs = decoder_layer( 2025-12-04T08:53:54.8371661Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:54.8372034Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:54.8372418Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8372800Z return func(*args, **kwargs) 2025-12-04T08:53:54.8373188Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T08:53:54.8373636Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:53:54.8374058Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8374448Z return func(*args, **kwargs) 2025-12-04T08:53:54.8374838Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 228, in forward 2025-12-04T08:53:54.8375315Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T08:53:54.8375536Z 2025-12-04T08:53:54.8375643Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:54.8376019Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:54.8376351Z res = mod(**inputs) 2025-12-04T08:53:54.8376719Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T08:53:54.8377125Z outputs = self.model.decoder( 2025-12-04T08:53:54.8377525Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T08:53:54.8377928Z layer_outputs = decoder_layer( 2025-12-04T08:53:54.8378282Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:54.8378660Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:54.8379075Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8379455Z return func(*args, **kwargs) 2025-12-04T08:53:54.8379844Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T08:53:54.8380272Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:53:54.8380685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8381063Z return func(*args, **kwargs) 2025-12-04T08:53:54.8381454Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 248, in forward 2025-12-04T08:53:54.8381866Z key_states = self.k_proj(current_states) 2025-12-04T08:53:54.8382006Z 2025-12-04T08:53:54.8382120Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:54.8382483Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:54.8382818Z res = mod(**inputs) 2025-12-04T08:53:54.8383209Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T08:53:54.8383608Z outputs = self.model.decoder( 2025-12-04T08:53:54.8384003Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T08:53:54.8384410Z layer_outputs = decoder_layer( 2025-12-04T08:53:54.8384787Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:54.8385153Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:54.8385564Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8385954Z return func(*args, **kwargs) 2025-12-04T08:53:54.8386335Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T08:53:54.8386769Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:53:54.8387181Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8387567Z return func(*args, **kwargs) 2025-12-04T08:53:54.8387967Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 249, in forward 2025-12-04T08:53:54.8388386Z value_states = self.v_proj(current_states) 2025-12-04T08:53:54.8388531Z 2025-12-04T08:53:54.8388624Z cudagraph partition due to non gpu ops 2025-12-04T08:53:54.8388867Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:54.8389241Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:54.8389571Z res = mod(**inputs) 2025-12-04T08:53:54.8389941Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T08:53:54.8390343Z outputs = self.model.decoder( 2025-12-04T08:53:54.8390733Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T08:53:54.8391132Z layer_outputs = decoder_layer( 2025-12-04T08:53:54.8391484Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:54.8391862Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:54.8392254Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8392640Z return func(*args, **kwargs) 2025-12-04T08:53:54.8393014Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T08:53:54.8393440Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:53:54.8393854Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8394235Z return func(*args, **kwargs) 2025-12-04T08:53:54.8394609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 267, in forward 2025-12-04T08:53:54.8395036Z attn_output, attn_weights = attention_interface( 2025-12-04T08:53:54.8395493Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T08:53:54.8396000Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T08:53:54.8396207Z 2025-12-04T08:53:54.8396321Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:54.8396714Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:54.8397072Z res = mod(**inputs) 2025-12-04T08:53:54.8397487Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T08:53:54.8397930Z outputs = self.model.decoder( 2025-12-04T08:53:54.8398436Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T08:53:54.8398882Z layer_outputs = decoder_layer( 2025-12-04T08:53:54.8399293Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:54.8399707Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:54.8400133Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8400563Z return func(*args, **kwargs) 2025-12-04T08:53:54.8400975Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T08:53:54.8401437Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:53:54.8401878Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8402278Z return func(*args, **kwargs) 2025-12-04T08:53:54.8402689Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 281, in forward 2025-12-04T08:53:54.8403165Z attn_output = self.out_proj(attn_output) 2025-12-04T08:53:54.8403315Z 2025-12-04T08:53:54.8403435Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:54.8403826Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:54.8404193Z res = mod(**inputs) 2025-12-04T08:53:54.8404588Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T08:53:54.8405029Z outputs = self.model.decoder( 2025-12-04T08:53:54.8405445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T08:53:54.8405870Z layer_outputs = decoder_layer( 2025-12-04T08:53:54.8406249Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:54.8406643Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:54.8407061Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8407470Z return func(*args, **kwargs) 2025-12-04T08:53:54.8407873Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 451, in forward 2025-12-04T08:53:54.8408352Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:53:54.8408546Z 2025-12-04T08:53:54.8408658Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:54.8409051Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:54.8409395Z res = mod(**inputs) 2025-12-04T08:53:54.8409785Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T08:53:54.8410192Z outputs = self.model.decoder( 2025-12-04T08:53:54.8410578Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T08:53:54.8410977Z layer_outputs = decoder_layer( 2025-12-04T08:53:54.8411336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:54.8411745Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:54.8412158Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8412543Z return func(*args, **kwargs) 2025-12-04T08:53:54.8412951Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 451, in forward 2025-12-04T08:53:54.8413401Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:53:54.8413797Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T08:53:54.8414158Z return self.act(input) 2025-12-04T08:53:54.8414321Z 2025-12-04T08:53:54.8414441Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:54.8414811Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:54.8415200Z res = mod(**inputs) 2025-12-04T08:53:54.8415577Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T08:53:54.8415984Z outputs = self.model.decoder( 2025-12-04T08:53:54.8416383Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T08:53:54.8416810Z layer_outputs = decoder_layer( 2025-12-04T08:53:54.8417188Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:54.8417579Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:54.8417971Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8418359Z return func(*args, **kwargs) 2025-12-04T08:53:54.8418741Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 453, in forward 2025-12-04T08:53:54.8419144Z hidden_states = self.fc2(hidden_states) 2025-12-04T08:53:54.8419293Z 2025-12-04T08:53:54.8419399Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:54.8419771Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:54.8420098Z res = mod(**inputs) 2025-12-04T08:53:54.8420464Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T08:53:54.8420984Z outputs = self.model.decoder( 2025-12-04T08:53:54.8421389Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T08:53:54.8421784Z layer_outputs = decoder_layer( 2025-12-04T08:53:54.8422149Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:54.8422530Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:54.8422922Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8423303Z return func(*args, **kwargs) 2025-12-04T08:53:54.8423696Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T08:53:54.8424130Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:53:54.8424543Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8424958Z return func(*args, **kwargs) 2025-12-04T08:53:54.8425367Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 228, in forward 2025-12-04T08:53:54.8425878Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T08:53:54.8426090Z 2025-12-04T08:53:54.8426198Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:54.8426569Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:54.8426899Z res = mod(**inputs) 2025-12-04T08:53:54.8427336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T08:53:54.8427731Z outputs = self.model.decoder( 2025-12-04T08:53:54.8428125Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T08:53:54.8428554Z layer_outputs = decoder_layer( 2025-12-04T08:53:54.8428956Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:54.8429367Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:54.8429761Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8430193Z return func(*args, **kwargs) 2025-12-04T08:53:54.8430595Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T08:53:54.8431051Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:53:54.8431485Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8431909Z return func(*args, **kwargs) 2025-12-04T08:53:54.8432316Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 248, in forward 2025-12-04T08:53:54.8432790Z key_states = self.k_proj(current_states) 2025-12-04T08:53:54.8432942Z 2025-12-04T08:53:54.8433066Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:54.8433458Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:54.8433819Z res = mod(**inputs) 2025-12-04T08:53:54.8434221Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T08:53:54.8434677Z outputs = self.model.decoder( 2025-12-04T08:53:54.8435096Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T08:53:54.8435529Z layer_outputs = decoder_layer( 2025-12-04T08:53:54.8435915Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:54.8436317Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:54.8436755Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8437178Z return func(*args, **kwargs) 2025-12-04T08:53:54.8437605Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T08:53:54.8438072Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:53:54.8438609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8439034Z return func(*args, **kwargs) 2025-12-04T08:53:54.8439445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 249, in forward 2025-12-04T08:53:54.8439895Z value_states = self.v_proj(current_states) 2025-12-04T08:53:54.8440069Z 2025-12-04T08:53:54.8440158Z cudagraph partition due to non gpu ops 2025-12-04T08:53:54.8440423Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:54.8440814Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:54.8441163Z res = mod(**inputs) 2025-12-04T08:53:54.8441557Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T08:53:54.8441988Z outputs = self.model.decoder( 2025-12-04T08:53:54.8442400Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T08:53:54.8442847Z layer_outputs = decoder_layer( 2025-12-04T08:53:54.8443245Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:54.8443653Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:54.8444090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8444522Z return func(*args, **kwargs) 2025-12-04T08:53:54.8444941Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T08:53:54.8445386Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:53:54.8445847Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8446257Z return func(*args, **kwargs) 2025-12-04T08:53:54.8446658Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 267, in forward 2025-12-04T08:53:54.8447111Z attn_output, attn_weights = attention_interface( 2025-12-04T08:53:54.8447593Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T08:53:54.8448111Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T08:53:54.8448302Z 2025-12-04T08:53:54.8448416Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:54.8448790Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:54.8449124Z res = mod(**inputs) 2025-12-04T08:53:54.8449503Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T08:53:54.8449907Z outputs = self.model.decoder( 2025-12-04T08:53:54.8450302Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T08:53:54.8450703Z layer_outputs = decoder_layer( 2025-12-04T08:53:54.8451061Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:54.8451438Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:54.8451835Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8452224Z return func(*args, **kwargs) 2025-12-04T08:53:54.8452603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T08:53:54.8453038Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:53:54.8453451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8453835Z return func(*args, **kwargs) 2025-12-04T08:53:54.8454217Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 281, in forward 2025-12-04T08:53:54.8454628Z attn_output = self.out_proj(attn_output) 2025-12-04T08:53:54.8454768Z 2025-12-04T08:53:54.8454882Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:54.8455249Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:54.8455581Z res = mod(**inputs) 2025-12-04T08:53:54.8455953Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T08:53:54.8456357Z outputs = self.model.decoder( 2025-12-04T08:53:54.8456741Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T08:53:54.8457137Z layer_outputs = decoder_layer( 2025-12-04T08:53:54.8457525Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:54.8457894Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:54.8458285Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8458681Z return func(*args, **kwargs) 2025-12-04T08:53:54.8459100Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 451, in forward 2025-12-04T08:53:54.8459545Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:53:54.8459733Z 2025-12-04T08:53:54.8459871Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:54.8460245Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:54.8460577Z res = mod(**inputs) 2025-12-04T08:53:54.8460945Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T08:53:54.8461355Z outputs = self.model.decoder( 2025-12-04T08:53:54.8461751Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T08:53:54.8462163Z layer_outputs = decoder_layer( 2025-12-04T08:53:54.8462530Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:54.8462907Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:54.8463306Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8463696Z return func(*args, **kwargs) 2025-12-04T08:53:54.8464079Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 451, in forward 2025-12-04T08:53:54.8464523Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:53:54.8464931Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T08:53:54.8465280Z return self.act(input) 2025-12-04T08:53:54.8465401Z 2025-12-04T08:53:54.8465509Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:54.8465882Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:54.8466210Z res = mod(**inputs) 2025-12-04T08:53:54.8466586Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T08:53:54.8466990Z outputs = self.model.decoder( 2025-12-04T08:53:54.8467378Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T08:53:54.8467778Z layer_outputs = decoder_layer( 2025-12-04T08:53:54.8468144Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:54.8468521Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:54.8468908Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8469299Z return func(*args, **kwargs) 2025-12-04T08:53:54.8469705Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 453, in forward 2025-12-04T08:53:54.8470107Z hidden_states = self.fc2(hidden_states) 2025-12-04T08:53:54.8470246Z 2025-12-04T08:53:54.8470351Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:54.8470718Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:54.8471046Z res = mod(**inputs) 2025-12-04T08:53:54.8471413Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T08:53:54.8471843Z outputs = self.model.decoder( 2025-12-04T08:53:54.8472243Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T08:53:54.8472643Z layer_outputs = decoder_layer( 2025-12-04T08:53:54.8473005Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:54.8473389Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:54.8473779Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8474176Z return func(*args, **kwargs) 2025-12-04T08:53:54.8474562Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 455, in forward 2025-12-04T08:53:54.8474971Z hidden_states = residual + hidden_states 2025-12-04T08:53:54.8475109Z 2025-12-04T08:53:54.8475222Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:54.8475592Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:54.8475923Z res = mod(**inputs) 2025-12-04T08:53:54.8476300Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T08:53:54.8476746Z outputs = self.model.decoder( 2025-12-04T08:53:54.8477156Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T08:53:54.8477580Z layer_outputs = decoder_layer( 2025-12-04T08:53:54.8477963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:54.8478422Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:54.8478848Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8479271Z return func(*args, **kwargs) 2025-12-04T08:53:54.8479691Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T08:53:54.8480135Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:53:54.8480584Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8480974Z return func(*args, **kwargs) 2025-12-04T08:53:54.8481376Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 228, in forward 2025-12-04T08:53:54.8481884Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T08:53:54.8482114Z 2025-12-04T08:53:54.8482228Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:54.8482621Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:54.8482960Z res = mod(**inputs) 2025-12-04T08:53:54.8483363Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T08:53:54.8483789Z outputs = self.model.decoder( 2025-12-04T08:53:54.8484218Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T08:53:54.8484637Z layer_outputs = decoder_layer( 2025-12-04T08:53:54.8485018Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:54.8485414Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:54.8485822Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8486266Z return func(*args, **kwargs) 2025-12-04T08:53:54.8486732Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T08:53:54.8487189Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:53:54.8487616Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8488025Z return func(*args, **kwargs) 2025-12-04T08:53:54.8488458Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 248, in forward 2025-12-04T08:53:54.8488889Z key_states = self.k_proj(current_states) 2025-12-04T08:53:54.8489045Z 2025-12-04T08:53:54.8489158Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:54.8489575Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:54.8489933Z res = mod(**inputs) 2025-12-04T08:53:54.8490327Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T08:53:54.8490760Z outputs = self.model.decoder( 2025-12-04T08:53:54.8491180Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T08:53:54.8491632Z layer_outputs = decoder_layer( 2025-12-04T08:53:54.8492023Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:54.8492419Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:54.8492834Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8493253Z return func(*args, **kwargs) 2025-12-04T08:53:54.8493674Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T08:53:54.8494094Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:53:54.8494501Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8494869Z return func(*args, **kwargs) 2025-12-04T08:53:54.8495245Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 249, in forward 2025-12-04T08:53:54.8495654Z value_states = self.v_proj(current_states) 2025-12-04T08:53:54.8495799Z 2025-12-04T08:53:54.8495892Z cudagraph partition due to non gpu ops 2025-12-04T08:53:54.8496128Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:54.8496494Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:54.8496824Z res = mod(**inputs) 2025-12-04T08:53:54.8497183Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T08:53:54.8497581Z outputs = self.model.decoder( 2025-12-04T08:53:54.8497971Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T08:53:54.8498370Z layer_outputs = decoder_layer( 2025-12-04T08:53:54.8498719Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:54.8499091Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:54.8499480Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8499855Z return func(*args, **kwargs) 2025-12-04T08:53:54.8500241Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T08:53:54.8500664Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:53:54.8501069Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8501459Z return func(*args, **kwargs) 2025-12-04T08:53:54.8501837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 267, in forward 2025-12-04T08:53:54.8502250Z attn_output, attn_weights = attention_interface( 2025-12-04T08:53:54.8502693Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T08:53:54.8503199Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T08:53:54.8503390Z 2025-12-04T08:53:54.8503494Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:54.8503876Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:54.8504200Z res = mod(**inputs) 2025-12-04T08:53:54.8504575Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T08:53:54.8504980Z outputs = self.model.decoder( 2025-12-04T08:53:54.8505372Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T08:53:54.8505764Z layer_outputs = decoder_layer( 2025-12-04T08:53:54.8506123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:54.8506509Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:54.8506884Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8507264Z return func(*args, **kwargs) 2025-12-04T08:53:54.8507643Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T08:53:54.8508062Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:53:54.8508458Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8508833Z return func(*args, **kwargs) 2025-12-04T08:53:54.8509222Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 281, in forward 2025-12-04T08:53:54.8509621Z attn_output = self.out_proj(attn_output) 2025-12-04T08:53:54.8509761Z 2025-12-04T08:53:54.8509869Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:54.8510240Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:54.8510575Z res = mod(**inputs) 2025-12-04T08:53:54.8510933Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T08:53:54.8511333Z outputs = self.model.decoder( 2025-12-04T08:53:54.8511716Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T08:53:54.8512118Z layer_outputs = decoder_layer( 2025-12-04T08:53:54.8512453Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:54.8512811Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:54.8513182Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8513539Z return func(*args, **kwargs) 2025-12-04T08:53:54.8513913Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 451, in forward 2025-12-04T08:53:54.8514351Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:53:54.8514524Z 2025-12-04T08:53:54.8514634Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:54.8514985Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:54.8515332Z res = mod(**inputs) 2025-12-04T08:53:54.8515698Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T08:53:54.8516096Z outputs = self.model.decoder( 2025-12-04T08:53:54.8516483Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T08:53:54.8516921Z layer_outputs = decoder_layer( 2025-12-04T08:53:54.8517289Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:54.8517655Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:54.8518070Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8518533Z return func(*args, **kwargs) 2025-12-04T08:53:54.8518960Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 451, in forward 2025-12-04T08:53:54.8519461Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:53:54.8519896Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T08:53:54.8520275Z return self.act(input) 2025-12-04T08:53:54.8520416Z 2025-12-04T08:53:54.8520531Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:54.8521057Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:54.8521396Z res = mod(**inputs) 2025-12-04T08:53:54.8521776Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T08:53:54.8522163Z outputs = self.model.decoder( 2025-12-04T08:53:54.8522554Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T08:53:54.8522977Z layer_outputs = decoder_layer( 2025-12-04T08:53:54.8523360Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:54.8523771Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:54.8524189Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8524597Z return func(*args, **kwargs) 2025-12-04T08:53:54.8524975Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 453, in forward 2025-12-04T08:53:54.8525383Z hidden_states = self.fc2(hidden_states) 2025-12-04T08:53:54.8525524Z 2025-12-04T08:53:54.8525637Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:54.8526001Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:54.8526330Z res = mod(**inputs) 2025-12-04T08:53:54.8526712Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T08:53:54.8527125Z outputs = self.model.decoder( 2025-12-04T08:53:54.8527516Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T08:53:54.8527926Z layer_outputs = decoder_layer( 2025-12-04T08:53:54.8528290Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:54.8528671Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:54.8529062Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8529456Z return func(*args, **kwargs) 2025-12-04T08:53:54.8529853Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T08:53:54.8530327Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:53:54.8530742Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8531131Z return func(*args, **kwargs) 2025-12-04T08:53:54.8531515Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 228, in forward 2025-12-04T08:53:54.8532038Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T08:53:54.8532272Z 2025-12-04T08:53:54.8532386Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:54.8532767Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:54.8533131Z res = mod(**inputs) 2025-12-04T08:53:54.8533500Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T08:53:54.8533926Z outputs = self.model.decoder( 2025-12-04T08:53:54.8534346Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T08:53:54.8534780Z layer_outputs = decoder_layer( 2025-12-04T08:53:54.8535170Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:54.8535598Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:54.8536030Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8536440Z return func(*args, **kwargs) 2025-12-04T08:53:54.8536860Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T08:53:54.8537324Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:53:54.8537757Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8538159Z return func(*args, **kwargs) 2025-12-04T08:53:54.8538568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 248, in forward 2025-12-04T08:53:54.8539003Z key_states = self.k_proj(current_states) 2025-12-04T08:53:54.8539153Z 2025-12-04T08:53:54.8539268Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:54.8539662Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:54.8540013Z res = mod(**inputs) 2025-12-04T08:53:54.8540407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T08:53:54.8540831Z outputs = self.model.decoder( 2025-12-04T08:53:54.8541245Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T08:53:54.8541673Z layer_outputs = decoder_layer( 2025-12-04T08:53:54.8542047Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:54.8542446Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:54.8542858Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8543276Z return func(*args, **kwargs) 2025-12-04T08:53:54.8543654Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T08:53:54.8544083Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:53:54.8544502Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8544908Z return func(*args, **kwargs) 2025-12-04T08:53:54.8545333Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 249, in forward 2025-12-04T08:53:54.8545775Z value_states = self.v_proj(current_states) 2025-12-04T08:53:54.8545927Z 2025-12-04T08:53:54.8546024Z cudagraph partition due to non gpu ops 2025-12-04T08:53:54.8546276Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:54.8546690Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:54.8547048Z res = mod(**inputs) 2025-12-04T08:53:54.8547444Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T08:53:54.8547888Z outputs = self.model.decoder( 2025-12-04T08:53:54.8548323Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T08:53:54.8548758Z layer_outputs = decoder_layer( 2025-12-04T08:53:54.8549131Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:54.8549529Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:54.8549950Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8550374Z return func(*args, **kwargs) 2025-12-04T08:53:54.8550781Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T08:53:54.8551243Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:53:54.8551682Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8552081Z return func(*args, **kwargs) 2025-12-04T08:53:54.8552484Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 267, in forward 2025-12-04T08:53:54.8552939Z attn_output, attn_weights = attention_interface( 2025-12-04T08:53:54.8553427Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T08:53:54.8553955Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T08:53:54.8553961Z 2025-12-04T08:53:54.8554082Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:54.8554301Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:54.8554381Z res = mod(**inputs) 2025-12-04T08:53:54.8554667Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T08:53:54.8554749Z outputs = self.model.decoder( 2025-12-04T08:53:54.8555039Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T08:53:54.8555121Z layer_outputs = decoder_layer( 2025-12-04T08:53:54.8555373Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:54.8555470Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:54.8555741Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8555827Z return func(*args, **kwargs) 2025-12-04T08:53:54.8556111Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T08:53:54.8556222Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:53:54.8556507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8556583Z return func(*args, **kwargs) 2025-12-04T08:53:54.8556898Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 281, in forward 2025-12-04T08:53:54.8556991Z attn_output = self.out_proj(attn_output) 2025-12-04T08:53:54.8556996Z 2025-12-04T08:53:54.8557111Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:54.8557341Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:54.8557416Z res = mod(**inputs) 2025-12-04T08:53:54.8557718Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T08:53:54.8557811Z outputs = self.model.decoder( 2025-12-04T08:53:54.8558117Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T08:53:54.8558223Z layer_outputs = decoder_layer( 2025-12-04T08:53:54.8558548Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:54.8558646Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:54.8558934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8559011Z return func(*args, **kwargs) 2025-12-04T08:53:54.8559293Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 451, in forward 2025-12-04T08:53:54.8559460Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:53:54.8559465Z 2025-12-04T08:53:54.8559589Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:54.8559811Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:54.8559883Z res = mod(**inputs) 2025-12-04T08:53:54.8560177Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T08:53:54.8560269Z outputs = self.model.decoder( 2025-12-04T08:53:54.8560569Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T08:53:54.8560655Z layer_outputs = decoder_layer( 2025-12-04T08:53:54.8560946Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:54.8561033Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:54.8561313Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8561388Z return func(*args, **kwargs) 2025-12-04T08:53:54.8561667Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 451, in forward 2025-12-04T08:53:54.8561804Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:53:54.8562046Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T08:53:54.8562131Z return self.act(input) 2025-12-04T08:53:54.8562135Z 2025-12-04T08:53:54.8562244Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:54.8562456Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:54.8562538Z res = mod(**inputs) 2025-12-04T08:53:54.8562818Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T08:53:54.8562906Z outputs = self.model.decoder( 2025-12-04T08:53:54.8563185Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T08:53:54.8563264Z layer_outputs = decoder_layer( 2025-12-04T08:53:54.8563518Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:54.8563603Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:54.8563904Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8563989Z return func(*args, **kwargs) 2025-12-04T08:53:54.8564266Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 453, in forward 2025-12-04T08:53:54.8564362Z hidden_states = self.fc2(hidden_states) 2025-12-04T08:53:54.8564387Z 2025-12-04T08:53:54.8564500Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:54.8564713Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:54.8564811Z res = mod(**inputs) 2025-12-04T08:53:54.8565089Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T08:53:54.8565169Z outputs = self.model.decoder( 2025-12-04T08:53:54.8565467Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T08:53:54.8565543Z layer_outputs = decoder_layer( 2025-12-04T08:53:54.8565792Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:54.8565908Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:54.8566176Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8566259Z return func(*args, **kwargs) 2025-12-04T08:53:54.8566537Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 455, in forward 2025-12-04T08:53:54.8566632Z hidden_states = residual + hidden_states 2025-12-04T08:53:54.8566635Z 2025-12-04T08:53:54.8566744Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:54.8566959Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:54.8567034Z res = mod(**inputs) 2025-12-04T08:53:54.8567322Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T08:53:54.8567401Z outputs = self.model.decoder( 2025-12-04T08:53:54.8567699Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T08:53:54.8567777Z layer_outputs = decoder_layer( 2025-12-04T08:53:54.8568034Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:54.8568119Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:54.8568384Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8568469Z return func(*args, **kwargs) 2025-12-04T08:53:54.8568750Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T08:53:54.8568864Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:53:54.8569139Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8569215Z return func(*args, **kwargs) 2025-12-04T08:53:54.8569499Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 228, in forward 2025-12-04T08:53:54.8569663Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T08:53:54.8569668Z 2025-12-04T08:53:54.8569777Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:54.8569997Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:54.8570067Z res = mod(**inputs) 2025-12-04T08:53:54.8570389Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T08:53:54.8570470Z outputs = self.model.decoder( 2025-12-04T08:53:54.8570751Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T08:53:54.8570836Z layer_outputs = decoder_layer( 2025-12-04T08:53:54.8571082Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:54.8571166Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:54.8571439Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8571533Z return func(*args, **kwargs) 2025-12-04T08:53:54.8571819Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T08:53:54.8571929Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:53:54.8572192Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8572275Z return func(*args, **kwargs) 2025-12-04T08:53:54.8572549Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 248, in forward 2025-12-04T08:53:54.8572664Z key_states = self.k_proj(current_states) 2025-12-04T08:53:54.8572668Z 2025-12-04T08:53:54.8572777Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:54.8572992Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:54.8573074Z res = mod(**inputs) 2025-12-04T08:53:54.8573355Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T08:53:54.8573433Z outputs = self.model.decoder( 2025-12-04T08:53:54.8573722Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T08:53:54.8573799Z layer_outputs = decoder_layer( 2025-12-04T08:53:54.8574049Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:54.8574135Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:54.8574400Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8574482Z return func(*args, **kwargs) 2025-12-04T08:53:54.8574759Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T08:53:54.8574873Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:53:54.8575138Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8575214Z return func(*args, **kwargs) 2025-12-04T08:53:54.8575496Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 249, in forward 2025-12-04T08:53:54.8575589Z value_states = self.v_proj(current_states) 2025-12-04T08:53:54.8575593Z 2025-12-04T08:53:54.8575684Z cudagraph partition due to non gpu ops 2025-12-04T08:53:54.8575801Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:54.8576015Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:54.8576091Z res = mod(**inputs) 2025-12-04T08:53:54.8576369Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T08:53:54.8576445Z outputs = self.model.decoder( 2025-12-04T08:53:54.8576715Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T08:53:54.8576814Z layer_outputs = decoder_layer( 2025-12-04T08:53:54.8577043Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:54.8577131Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:54.8577381Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8577478Z return func(*args, **kwargs) 2025-12-04T08:53:54.8577741Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T08:53:54.8577840Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:53:54.8578115Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8578186Z return func(*args, **kwargs) 2025-12-04T08:53:54.8578454Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 267, in forward 2025-12-04T08:53:54.8578556Z attn_output, attn_weights = attention_interface( 2025-12-04T08:53:54.8578857Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T08:53:54.8579021Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T08:53:54.8579024Z 2025-12-04T08:53:54.8579131Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:54.8579335Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:54.8579409Z res = mod(**inputs) 2025-12-04T08:53:54.8579671Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T08:53:54.8579752Z outputs = self.model.decoder( 2025-12-04T08:53:54.8580014Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T08:53:54.8580088Z layer_outputs = decoder_layer( 2025-12-04T08:53:54.8580322Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:54.8580404Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:54.8580658Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8580729Z return func(*args, **kwargs) 2025-12-04T08:53:54.8580990Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T08:53:54.8581098Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:53:54.8581347Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8581418Z return func(*args, **kwargs) 2025-12-04T08:53:54.8581685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 281, in forward 2025-12-04T08:53:54.8581768Z attn_output = self.out_proj(attn_output) 2025-12-04T08:53:54.8581772Z 2025-12-04T08:53:54.8581880Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:54.8582089Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:54.8582156Z res = mod(**inputs) 2025-12-04T08:53:54.8582436Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T08:53:54.8582518Z outputs = self.model.decoder( 2025-12-04T08:53:54.8582791Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T08:53:54.8582877Z layer_outputs = decoder_layer( 2025-12-04T08:53:54.8583133Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:54.8583224Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:54.8583476Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8583554Z return func(*args, **kwargs) 2025-12-04T08:53:54.8583852Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 451, in forward 2025-12-04T08:53:54.8583982Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:53:54.8583986Z 2025-12-04T08:53:54.8584124Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:54.8584342Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:54.8584412Z res = mod(**inputs) 2025-12-04T08:53:54.8584712Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T08:53:54.8584791Z outputs = self.model.decoder( 2025-12-04T08:53:54.8585070Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T08:53:54.8585155Z layer_outputs = decoder_layer( 2025-12-04T08:53:54.8585427Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:54.8585519Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:54.8585782Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8585858Z return func(*args, **kwargs) 2025-12-04T08:53:54.8586141Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 451, in forward 2025-12-04T08:53:54.8586267Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:53:54.8586510Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T08:53:54.8586588Z return self.act(input) 2025-12-04T08:53:54.8586592Z 2025-12-04T08:53:54.8586702Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:54.8586925Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:54.8586996Z res = mod(**inputs) 2025-12-04T08:53:54.8587273Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T08:53:54.8587362Z outputs = self.model.decoder( 2025-12-04T08:53:54.8587641Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T08:53:54.8587726Z layer_outputs = decoder_layer( 2025-12-04T08:53:54.8587967Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:54.8588052Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:54.8588323Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8588399Z return func(*args, **kwargs) 2025-12-04T08:53:54.8588678Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 453, in forward 2025-12-04T08:53:54.8588775Z hidden_states = self.fc2(hidden_states) 2025-12-04T08:53:54.8588779Z 2025-12-04T08:53:54.8588888Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:54.8589109Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:54.8589181Z res = mod(**inputs) 2025-12-04T08:53:54.8589458Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T08:53:54.8589565Z outputs = self.model.decoder( 2025-12-04T08:53:54.8589855Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T08:53:54.8589941Z layer_outputs = decoder_layer( 2025-12-04T08:53:54.8590183Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:54.8590294Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:54.8590570Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8590643Z return func(*args, **kwargs) 2025-12-04T08:53:54.8590936Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T08:53:54.8591052Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:53:54.8591318Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8591398Z return func(*args, **kwargs) 2025-12-04T08:53:54.8591672Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 228, in forward 2025-12-04T08:53:54.8591859Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T08:53:54.8591863Z 2025-12-04T08:53:54.8591983Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:54.8592199Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:54.8592277Z res = mod(**inputs) 2025-12-04T08:53:54.8592565Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T08:53:54.8592647Z outputs = self.model.decoder( 2025-12-04T08:53:54.8592951Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T08:53:54.8593028Z layer_outputs = decoder_layer( 2025-12-04T08:53:54.8593269Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:54.8593362Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:54.8593629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8593709Z return func(*args, **kwargs) 2025-12-04T08:53:54.8593983Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T08:53:54.8594089Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:53:54.8594367Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8594442Z return func(*args, **kwargs) 2025-12-04T08:53:54.8594729Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 248, in forward 2025-12-04T08:53:54.8594827Z key_states = self.k_proj(current_states) 2025-12-04T08:53:54.8594830Z 2025-12-04T08:53:54.8594941Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:54.8595169Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:54.8595242Z res = mod(**inputs) 2025-12-04T08:53:54.8595529Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T08:53:54.8595620Z outputs = self.model.decoder( 2025-12-04T08:53:54.8595908Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T08:53:54.8595995Z layer_outputs = decoder_layer( 2025-12-04T08:53:54.8596264Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:54.8596351Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:54.8596627Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8596705Z return func(*args, **kwargs) 2025-12-04T08:53:54.8597043Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T08:53:54.8597160Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:53:54.8597428Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8597541Z return func(*args, **kwargs) 2025-12-04T08:53:54.8597827Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 249, in forward 2025-12-04T08:53:54.8597924Z value_states = self.v_proj(current_states) 2025-12-04T08:53:54.8597929Z 2025-12-04T08:53:54.8598025Z cudagraph partition due to non gpu ops 2025-12-04T08:53:54.8598137Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:54.8598445Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:54.8598549Z res = mod(**inputs) 2025-12-04T08:53:54.8598857Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T08:53:54.8598946Z outputs = self.model.decoder( 2025-12-04T08:53:54.8599237Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T08:53:54.8599320Z layer_outputs = decoder_layer( 2025-12-04T08:53:54.8599586Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:54.8599673Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:54.8599963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8600039Z return func(*args, **kwargs) 2025-12-04T08:53:54.8600321Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T08:53:54.8600441Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:53:54.8600729Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8600804Z return func(*args, **kwargs) 2025-12-04T08:53:54.8601090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 267, in forward 2025-12-04T08:53:54.8601196Z attn_output, attn_weights = attention_interface( 2025-12-04T08:53:54.8601528Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T08:53:54.8601672Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T08:53:54.8601676Z 2025-12-04T08:53:54.8601786Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:54.8602009Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:54.8602083Z res = mod(**inputs) 2025-12-04T08:53:54.8602377Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T08:53:54.8602462Z outputs = self.model.decoder( 2025-12-04T08:53:54.8602760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T08:53:54.8602850Z layer_outputs = decoder_layer( 2025-12-04T08:53:54.8603127Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:54.8603217Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:54.8603507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8603584Z return func(*args, **kwargs) 2025-12-04T08:53:54.8603898Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T08:53:54.8604008Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:53:54.8604288Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8604393Z return func(*args, **kwargs) 2025-12-04T08:53:54.8604681Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 281, in forward 2025-12-04T08:53:54.8604781Z attn_output = self.out_proj(attn_output) 2025-12-04T08:53:54.8604784Z 2025-12-04T08:53:54.8604899Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:54.8605121Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:54.8605202Z res = mod(**inputs) 2025-12-04T08:53:54.8605491Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T08:53:54.8605596Z outputs = self.model.decoder( 2025-12-04T08:53:54.8605888Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T08:53:54.8605969Z layer_outputs = decoder_layer( 2025-12-04T08:53:54.8606225Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:54.8606309Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:54.8606582Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8606665Z return func(*args, **kwargs) 2025-12-04T08:53:54.8606951Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 451, in forward 2025-12-04T08:53:54.8607085Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:53:54.8607096Z 2025-12-04T08:53:54.8607209Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:54.8607433Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:54.8607511Z res = mod(**inputs) 2025-12-04T08:53:54.8607799Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T08:53:54.8607883Z outputs = self.model.decoder( 2025-12-04T08:53:54.8608188Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T08:53:54.8608268Z layer_outputs = decoder_layer( 2025-12-04T08:53:54.8608532Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:54.8608618Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:54.8608906Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8608991Z return func(*args, **kwargs) 2025-12-04T08:53:54.8609272Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 451, in forward 2025-12-04T08:53:54.8609404Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:53:54.8609660Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T08:53:54.8609738Z return self.act(input) 2025-12-04T08:53:54.8609742Z 2025-12-04T08:53:54.8609885Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:54.8610105Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:54.8610177Z res = mod(**inputs) 2025-12-04T08:53:54.8610471Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T08:53:54.8610553Z outputs = self.model.decoder( 2025-12-04T08:53:54.8610861Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T08:53:54.8610944Z layer_outputs = decoder_layer( 2025-12-04T08:53:54.8611220Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:54.8611316Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:54.8611590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8611667Z return func(*args, **kwargs) 2025-12-04T08:53:54.8611959Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 453, in forward 2025-12-04T08:53:54.8612050Z hidden_states = self.fc2(hidden_states) 2025-12-04T08:53:54.8612074Z 2025-12-04T08:53:54.8612193Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:54.8612418Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:54.8612488Z res = mod(**inputs) 2025-12-04T08:53:54.8612783Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T08:53:54.8612867Z outputs = self.model.decoder( 2025-12-04T08:53:54.8613157Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T08:53:54.8613244Z layer_outputs = decoder_layer( 2025-12-04T08:53:54.8613494Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:54.8613589Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:54.8613860Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8613939Z return func(*args, **kwargs) 2025-12-04T08:53:54.8614230Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 455, in forward 2025-12-04T08:53:54.8614317Z hidden_states = residual + hidden_states 2025-12-04T08:53:54.8614323Z 2025-12-04T08:53:54.8614440Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:54.8614660Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:54.8614730Z res = mod(**inputs) 2025-12-04T08:53:54.8615026Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T08:53:54.8615106Z outputs = self.model.decoder( 2025-12-04T08:53:54.8615391Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T08:53:54.8615481Z layer_outputs = decoder_layer( 2025-12-04T08:53:54.8615728Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:54.8615821Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:54.8616093Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8616171Z return func(*args, **kwargs) 2025-12-04T08:53:54.8616469Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T08:53:54.8616596Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:53:54.8616861Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8616942Z return func(*args, **kwargs) 2025-12-04T08:53:54.8617228Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 228, in forward 2025-12-04T08:53:54.8617426Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T08:53:54.8617431Z 2025-12-04T08:53:54.8617542Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:54.8617753Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:54.8617852Z res = mod(**inputs) 2025-12-04T08:53:54.8618133Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T08:53:54.8618222Z outputs = self.model.decoder( 2025-12-04T08:53:54.8618504Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T08:53:54.8618583Z layer_outputs = decoder_layer( 2025-12-04T08:53:54.8618837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:54.8618941Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:54.8619204Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8619288Z return func(*args, **kwargs) 2025-12-04T08:53:54.8619566Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T08:53:54.8619680Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:53:54.8619946Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8620020Z return func(*args, **kwargs) 2025-12-04T08:53:54.8620305Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 248, in forward 2025-12-04T08:53:54.8620392Z key_states = self.k_proj(current_states) 2025-12-04T08:53:54.8620398Z 2025-12-04T08:53:54.8620513Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:54.8620874Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:54.8620954Z res = mod(**inputs) 2025-12-04T08:53:54.8621239Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T08:53:54.8621320Z outputs = self.model.decoder( 2025-12-04T08:53:54.8621595Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T08:53:54.8621684Z layer_outputs = decoder_layer( 2025-12-04T08:53:54.8621926Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:54.8622019Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:54.8622283Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8622362Z return func(*args, **kwargs) 2025-12-04T08:53:54.8622645Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T08:53:54.8622751Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:53:54.8623023Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8623097Z return func(*args, **kwargs) 2025-12-04T08:53:54.8623412Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 249, in forward 2025-12-04T08:53:54.8623518Z value_states = self.v_proj(current_states) 2025-12-04T08:53:54.8623523Z 2025-12-04T08:53:54.8623610Z cudagraph partition due to non gpu ops 2025-12-04T08:53:54.8623718Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:54.8623943Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:54.8624040Z res = mod(**inputs) 2025-12-04T08:53:54.8624323Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T08:53:54.8624440Z outputs = self.model.decoder( 2025-12-04T08:53:54.8624717Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T08:53:54.8624805Z layer_outputs = decoder_layer( 2025-12-04T08:53:54.8625051Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:54.8625136Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:54.8625409Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8625509Z return func(*args, **kwargs) 2025-12-04T08:53:54.8625797Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T08:53:54.8625904Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:53:54.8626166Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8626249Z return func(*args, **kwargs) 2025-12-04T08:53:54.8626527Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 267, in forward 2025-12-04T08:53:54.8626644Z attn_output, attn_weights = attention_interface( 2025-12-04T08:53:54.8626964Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T08:53:54.8627110Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T08:53:54.8627116Z 2025-12-04T08:53:54.8627233Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:54.8627449Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:54.8627519Z res = mod(**inputs) 2025-12-04T08:53:54.8627806Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T08:53:54.8627888Z outputs = self.model.decoder( 2025-12-04T08:53:54.8628175Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T08:53:54.8628251Z layer_outputs = decoder_layer( 2025-12-04T08:53:54.8628505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:54.8628598Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:54.8628861Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8628944Z return func(*args, **kwargs) 2025-12-04T08:53:54.8629223Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T08:53:54.8629328Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:53:54.8629599Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8629673Z return func(*args, **kwargs) 2025-12-04T08:53:54.8629970Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 281, in forward 2025-12-04T08:53:54.8630069Z attn_output = self.out_proj(attn_output) 2025-12-04T08:53:54.8630073Z 2025-12-04T08:53:54.8630182Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:54.8630402Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:54.8630474Z res = mod(**inputs) 2025-12-04T08:53:54.8630768Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T08:53:54.8630857Z outputs = self.model.decoder( 2025-12-04T08:53:54.8631133Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T08:53:54.8631234Z layer_outputs = decoder_layer( 2025-12-04T08:53:54.8631488Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:54.8631572Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:54.8631841Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8631913Z return func(*args, **kwargs) 2025-12-04T08:53:54.8632185Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 451, in forward 2025-12-04T08:53:54.8632340Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:53:54.8632345Z 2025-12-04T08:53:54.8632454Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:54.8632674Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:54.8632745Z res = mod(**inputs) 2025-12-04T08:53:54.8633023Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T08:53:54.8633110Z outputs = self.model.decoder( 2025-12-04T08:53:54.8633386Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T08:53:54.8633464Z layer_outputs = decoder_layer( 2025-12-04T08:53:54.8633717Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:54.8633803Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:54.8634073Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8634148Z return func(*args, **kwargs) 2025-12-04T08:53:54.8634421Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 451, in forward 2025-12-04T08:53:54.8634555Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:53:54.8634790Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T08:53:54.8634875Z return self.act(input) 2025-12-04T08:53:54.8634879Z 2025-12-04T08:53:54.8634989Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:54.8635200Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:54.8635279Z res = mod(**inputs) 2025-12-04T08:53:54.8635556Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T08:53:54.8635635Z outputs = self.model.decoder( 2025-12-04T08:53:54.8635921Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T08:53:54.8636000Z layer_outputs = decoder_layer( 2025-12-04T08:53:54.8636247Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:54.8636331Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:54.8636611Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8636696Z return func(*args, **kwargs) 2025-12-04T08:53:54.8636972Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 453, in forward 2025-12-04T08:53:54.8637063Z hidden_states = self.fc2(hidden_states) 2025-12-04T08:53:54.8637101Z 2025-12-04T08:53:54.8637213Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:54.8637429Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:54.8637526Z res = mod(**inputs) 2025-12-04T08:53:54.8637805Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T08:53:54.8637883Z outputs = self.model.decoder( 2025-12-04T08:53:54.8638171Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T08:53:54.8638301Z layer_outputs = decoder_layer( 2025-12-04T08:53:54.8638569Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:54.8638679Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:54.8638954Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8639037Z return func(*args, **kwargs) 2025-12-04T08:53:54.8639322Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T08:53:54.8639432Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:53:54.8639708Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8639786Z return func(*args, **kwargs) 2025-12-04T08:53:54.8640083Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 228, in forward 2025-12-04T08:53:54.8640247Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T08:53:54.8640253Z 2025-12-04T08:53:54.8640361Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:54.8640590Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:54.8640663Z res = mod(**inputs) 2025-12-04T08:53:54.8640957Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T08:53:54.8641040Z outputs = self.model.decoder( 2025-12-04T08:53:54.8641327Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T08:53:54.8641417Z layer_outputs = decoder_layer( 2025-12-04T08:53:54.8641681Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:54.8641768Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:54.8642060Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8642137Z return func(*args, **kwargs) 2025-12-04T08:53:54.8642440Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T08:53:54.8642548Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:53:54.8642854Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8642939Z return func(*args, **kwargs) 2025-12-04T08:53:54.8643232Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 248, in forward 2025-12-04T08:53:54.8643341Z key_states = self.k_proj(current_states) 2025-12-04T08:53:54.8643353Z 2025-12-04T08:53:54.8643466Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:54.8643702Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:54.8643783Z res = mod(**inputs) 2025-12-04T08:53:54.8644103Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T08:53:54.8644187Z outputs = self.model.decoder( 2025-12-04T08:53:54.8644494Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T08:53:54.8644599Z layer_outputs = decoder_layer( 2025-12-04T08:53:54.8644867Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:54.8644955Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:54.8645251Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8645335Z return func(*args, **kwargs) 2025-12-04T08:53:54.8645646Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T08:53:54.8645773Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:53:54.8646116Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8646203Z return func(*args, **kwargs) 2025-12-04T08:53:54.8646501Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 249, in forward 2025-12-04T08:53:54.8646596Z value_states = self.v_proj(current_states) 2025-12-04T08:53:54.8646601Z 2025-12-04T08:53:54.8646697Z cudagraph partition due to non gpu ops 2025-12-04T08:53:54.8646815Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:54.8647029Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:54.8647098Z res = mod(**inputs) 2025-12-04T08:53:54.8647382Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T08:53:54.8647464Z outputs = self.model.decoder( 2025-12-04T08:53:54.8647755Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T08:53:54.8647832Z layer_outputs = decoder_layer( 2025-12-04T08:53:54.8648083Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:54.8648169Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:54.8648420Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8648494Z return func(*args, **kwargs) 2025-12-04T08:53:54.8648756Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T08:53:54.8648855Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:53:54.8649112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8649183Z return func(*args, **kwargs) 2025-12-04T08:53:54.8649459Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 267, in forward 2025-12-04T08:53:54.8649572Z attn_output, attn_weights = attention_interface( 2025-12-04T08:53:54.8649896Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T08:53:54.8650069Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T08:53:54.8650074Z 2025-12-04T08:53:54.8650186Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:54.8650400Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:54.8650481Z res = mod(**inputs) 2025-12-04T08:53:54.8650778Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T08:53:54.8650869Z outputs = self.model.decoder( 2025-12-04T08:53:54.8651153Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T08:53:54.8651245Z layer_outputs = decoder_layer( 2025-12-04T08:53:54.8651487Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:54.8651568Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:54.8651818Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8651894Z return func(*args, **kwargs) 2025-12-04T08:53:54.8652155Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 420, in forward 2025-12-04T08:53:54.8652282Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:53:54.8652530Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8652600Z return func(*args, **kwargs) 2025-12-04T08:53:54.8652868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 281, in forward 2025-12-04T08:53:54.8652953Z attn_output = self.out_proj(attn_output) 2025-12-04T08:53:54.8652956Z 2025-12-04T08:53:54.8653067Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:54.8653269Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:54.8653335Z res = mod(**inputs) 2025-12-04T08:53:54.8653605Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T08:53:54.8653683Z outputs = self.model.decoder( 2025-12-04T08:53:54.8653945Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T08:53:54.8654029Z layer_outputs = decoder_layer( 2025-12-04T08:53:54.8654274Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:54.8654366Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:54.8654629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8654702Z return func(*args, **kwargs) 2025-12-04T08:53:54.8654985Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 451, in forward 2025-12-04T08:53:54.8655113Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:53:54.8655117Z 2025-12-04T08:53:54.8655232Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:54.8655452Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:54.8655535Z res = mod(**inputs) 2025-12-04T08:53:54.8655822Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T08:53:54.8655904Z outputs = self.model.decoder( 2025-12-04T08:53:54.8656183Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T08:53:54.8656268Z layer_outputs = decoder_layer( 2025-12-04T08:53:54.8656529Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:54.8656623Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:54.8656887Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8656965Z return func(*args, **kwargs) 2025-12-04T08:53:54.8657266Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 451, in forward 2025-12-04T08:53:54.8657393Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:53:54.8657626Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T08:53:54.8657730Z return self.act(input) 2025-12-04T08:53:54.8657734Z 2025-12-04T08:53:54.8657897Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:54.8658120Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:54.8658189Z res = mod(**inputs) 2025-12-04T08:53:54.8658471Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T08:53:54.8658550Z outputs = self.model.decoder( 2025-12-04T08:53:54.8658852Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T08:53:54.8658937Z layer_outputs = decoder_layer( 2025-12-04T08:53:54.8659176Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:54.8659263Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:54.8659535Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8659609Z return func(*args, **kwargs) 2025-12-04T08:53:54.8659893Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 453, in forward 2025-12-04T08:53:54.8659981Z hidden_states = self.fc2(hidden_states) 2025-12-04T08:53:54.8659984Z 2025-12-04T08:53:54.8660094Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:54.8660317Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:54.8660391Z res = mod(**inputs) 2025-12-04T08:53:54.8660671Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1865, in forward 2025-12-04T08:53:54.8660757Z outputs = self.model.decoder( 2025-12-04T08:53:54.8661051Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1116, in forward 2025-12-04T08:53:54.8661132Z layer_outputs = decoder_layer( 2025-12-04T08:53:54.8661361Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:53:54.8661440Z return super().__call__(*args, **kwargs) 2025-12-04T08:53:54.8661699Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:53:54.8661771Z return func(*args, **kwargs) 2025-12-04T08:53:54.8662058Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 455, in forward 2025-12-04T08:53:54.8662144Z hidden_states = residual + hidden_states 2025-12-04T08:53:54.8662148Z 2025-12-04T08:53:54.8662258Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:54.8662485Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:54.8662557Z res = mod(**inputs) 2025-12-04T08:53:54.8662843Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1881, in forward 2025-12-04T08:53:54.8662965Z logits = self.lm_head(outputs[0]) 2025-12-04T08:53:54.8662970Z 2025-12-04T08:53:54.8663091Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:53:54.8663310Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:53:54.8663382Z res = mod(**inputs) 2025-12-04T08:53:54.8663681Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mbart/modeling_mbart.py", line 1887, in forward 2025-12-04T08:53:54.8663853Z loss = loss_fct(logits.view(-1, self.config.vocab_size), labels.view(-1)) 2025-12-04T08:53:54.8663857Z 2025-12-04T08:54:07.7308353Z Compilation time (from dynamo_timed): 17.50504326 2025-12-04T08:54:07.7607322Z pass 2025-12-04T08:54:07.7607803Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T08:54:07.7608873Z TIMING: _recursive_pre_grad_passes:0.00698 _recursive_joint_graph_passes:0.67326 _recursive_post_grad_passes:0.06979 async_compile.wait:1.21063 code_gen:10.51826 inductor_compile:11.87001 backend_compile:15.07189 gc:0.00018 entire_frame_compile:17.50504 total_wall_time:17.50504 2025-12-04T08:54:07.7609923Z STATS: call_* op count: 337 | FakeTensorMode.__torch_dispatch__:7389 | FakeTensor.__torch_dispatch__:4449 | ProxyTorchDispatchMode.__torch_dispatch__:2049 2025-12-04T08:54:07.7610775Z Dynamo produced 1 graphs covering 337 ops with 0 graph breaks (0 unique) 2025-12-04T08:54:10.3061471Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/__init__.py:65: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-12-04T08:54:10.3062475Z import pynvml # type: ignore[import] 2025-12-04T08:54:13.8718525Z 2025-12-04T08:54:16.2824369Z loading model: 0it [00:00, ?it/s] 2025-12-04T08:54:16.2824938Z loading model: 0it [00:02, ?it/s] 2025-12-04T08:54:16.2839964Z cpu eval MT5ForConditionalGeneration 2025-12-04T08:54:17.4278248Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T08:54:17.9934788Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T08:54:18.5624827Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T08:54:30.9065578Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9066098Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9066478Z res = mod(**inputs) 2025-12-04T08:54:30.9066925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T08:54:30.9067361Z decoder_outputs = self.decoder( 2025-12-04T08:54:30.9067794Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9069045Z layer_outputs = layer_module( 2025-12-04T08:54:30.9069458Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9069870Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9071540Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9072055Z return func(*args, **kwargs) 2025-12-04T08:54:30.9072501Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T08:54:30.9072984Z self_attention_outputs = self.layer[0]( 2025-12-04T08:54:30.9073407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9073824Z return func(*args, **kwargs) 2025-12-04T08:54:30.9074544Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T08:54:30.9074992Z attention_output = self.SelfAttention( 2025-12-04T08:54:30.9075419Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9075843Z return func(*args, **kwargs) 2025-12-04T08:54:30.9076311Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 424, in forward 2025-12-04T08:54:30.9076749Z position_bias = position_bias + causal_mask 2025-12-04T08:54:30.9076923Z 2025-12-04T08:54:30.9077098Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9077515Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9077881Z res = mod(**inputs) 2025-12-04T08:54:30.9078666Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T08:54:30.9079127Z encoder_outputs = self.encoder( 2025-12-04T08:54:30.9079564Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9079992Z layer_outputs = layer_module( 2025-12-04T08:54:30.9080451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9080858Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9081279Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9081728Z return func(*args, **kwargs) 2025-12-04T08:54:30.9082129Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T08:54:30.9082549Z self_attention_outputs = self.layer[0]( 2025-12-04T08:54:30.9083006Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9083410Z return func(*args, **kwargs) 2025-12-04T08:54:30.9083809Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T08:54:30.9084246Z attention_output = self.SelfAttention( 2025-12-04T08:54:30.9084675Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9085098Z return func(*args, **kwargs) 2025-12-04T08:54:30.9085497Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 367, in forward 2025-12-04T08:54:30.9085928Z query_states = self.q(hidden_states) 2025-12-04T08:54:30.9086083Z 2025-12-04T08:54:30.9086205Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9086626Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9086993Z res = mod(**inputs) 2025-12-04T08:54:30.9087385Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T08:54:30.9087796Z encoder_outputs = self.encoder( 2025-12-04T08:54:30.9088223Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9088643Z layer_outputs = layer_module( 2025-12-04T08:54:30.9089026Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9089437Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9089870Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9090283Z return func(*args, **kwargs) 2025-12-04T08:54:30.9090708Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T08:54:30.9091136Z self_attention_outputs = self.layer[0]( 2025-12-04T08:54:30.9091564Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9091964Z return func(*args, **kwargs) 2025-12-04T08:54:30.9092381Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T08:54:30.9092818Z attention_output = self.SelfAttention( 2025-12-04T08:54:30.9093259Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9093723Z return func(*args, **kwargs) 2025-12-04T08:54:30.9094120Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 388, in forward 2025-12-04T08:54:30.9094547Z key_states = self.k(current_states) 2025-12-04T08:54:30.9094691Z 2025-12-04T08:54:30.9094816Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9095225Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9095601Z res = mod(**inputs) 2025-12-04T08:54:30.9096055Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T08:54:30.9096488Z encoder_outputs = self.encoder( 2025-12-04T08:54:30.9096925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9097363Z layer_outputs = layer_module( 2025-12-04T08:54:30.9097763Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9098197Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9098642Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9099084Z return func(*args, **kwargs) 2025-12-04T08:54:30.9099537Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T08:54:30.9099986Z self_attention_outputs = self.layer[0]( 2025-12-04T08:54:30.9100430Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9100853Z return func(*args, **kwargs) 2025-12-04T08:54:30.9101254Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T08:54:30.9101691Z attention_output = self.SelfAttention( 2025-12-04T08:54:30.9102123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9102541Z return func(*args, **kwargs) 2025-12-04T08:54:30.9102975Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 389, in forward 2025-12-04T08:54:30.9103417Z value_states = self.v(current_states) 2025-12-04T08:54:30.9103583Z 2025-12-04T08:54:30.9103680Z cudagraph partition due to non gpu ops 2025-12-04T08:54:30.9103925Z cudagraph partition due to non gpu ops 2025-12-04T08:54:30.9104185Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9104591Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9104966Z res = mod(**inputs) 2025-12-04T08:54:30.9105377Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T08:54:30.9105807Z encoder_outputs = self.encoder( 2025-12-04T08:54:30.9106250Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9106677Z layer_outputs = layer_module( 2025-12-04T08:54:30.9107071Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9107473Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9107896Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9108327Z return func(*args, **kwargs) 2025-12-04T08:54:30.9108718Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T08:54:30.9109165Z self_attention_outputs = self.layer[0]( 2025-12-04T08:54:30.9109590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9110002Z return func(*args, **kwargs) 2025-12-04T08:54:30.9110406Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T08:54:30.9110831Z attention_output = self.SelfAttention( 2025-12-04T08:54:30.9111250Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9111667Z return func(*args, **kwargs) 2025-12-04T08:54:30.9112067Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 447, in forward 2025-12-04T08:54:30.9112486Z attn_output = self.o(attn_output) 2025-12-04T08:54:30.9112629Z 2025-12-04T08:54:30.9112750Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9113143Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9113500Z res = mod(**inputs) 2025-12-04T08:54:30.9113886Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T08:54:30.9114299Z encoder_outputs = self.encoder( 2025-12-04T08:54:30.9114707Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9115124Z layer_outputs = layer_module( 2025-12-04T08:54:30.9115514Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9115907Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9116323Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9116734Z return func(*args, **kwargs) 2025-12-04T08:54:30.9117129Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T08:54:30.9117559Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T08:54:30.9118003Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T08:54:30.9118550Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T08:54:30.9119006Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 184, in forward 2025-12-04T08:54:30.9119474Z hidden_gelu = self.act(self.wi_0(hidden_states)) 2025-12-04T08:54:30.9119657Z 2025-12-04T08:54:30.9119782Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9120193Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9120553Z res = mod(**inputs) 2025-12-04T08:54:30.9121131Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T08:54:30.9121554Z encoder_outputs = self.encoder( 2025-12-04T08:54:30.9122028Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9122446Z layer_outputs = layer_module( 2025-12-04T08:54:30.9122833Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9123239Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9123685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9124102Z return func(*args, **kwargs) 2025-12-04T08:54:30.9124497Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T08:54:30.9124966Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T08:54:30.9125388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T08:54:30.9126221Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T08:54:30.9126686Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 185, in forward 2025-12-04T08:54:30.9127105Z hidden_linear = self.wi_1(hidden_states) 2025-12-04T08:54:30.9127256Z 2025-12-04T08:54:30.9127370Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9127820Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9128181Z res = mod(**inputs) 2025-12-04T08:54:30.9128586Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T08:54:30.9129010Z encoder_outputs = self.encoder( 2025-12-04T08:54:30.9129505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9130002Z layer_outputs = layer_module( 2025-12-04T08:54:30.9130537Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9130994Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9131661Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9132209Z return func(*args, **kwargs) 2025-12-04T08:54:30.9132716Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T08:54:30.9133241Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T08:54:30.9133769Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T08:54:30.9151220Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T08:54:30.9151912Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 186, in forward 2025-12-04T08:54:30.9152404Z hidden_states = hidden_gelu * hidden_linear 2025-12-04T08:54:30.9152574Z 2025-12-04T08:54:30.9152710Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9153124Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9153508Z res = mod(**inputs) 2025-12-04T08:54:30.9153924Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T08:54:30.9154352Z encoder_outputs = self.encoder( 2025-12-04T08:54:30.9154780Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9155208Z layer_outputs = layer_module( 2025-12-04T08:54:30.9155612Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9156017Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9156552Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9156996Z return func(*args, **kwargs) 2025-12-04T08:54:30.9157399Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T08:54:30.9157851Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T08:54:30.9158444Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T08:54:30.9158956Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T08:54:30.9159465Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 199, in forward 2025-12-04T08:54:30.9159973Z hidden_states = self.wo(hidden_states) 2025-12-04T08:54:30.9160136Z 2025-12-04T08:54:30.9160264Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9160684Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9161061Z res = mod(**inputs) 2025-12-04T08:54:30.9161484Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T08:54:30.9161968Z encoder_outputs = self.encoder( 2025-12-04T08:54:30.9162407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9162837Z layer_outputs = layer_module( 2025-12-04T08:54:30.9163238Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9163665Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9164096Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9164533Z return func(*args, **kwargs) 2025-12-04T08:54:30.9164949Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T08:54:30.9165386Z self_attention_outputs = self.layer[0]( 2025-12-04T08:54:30.9165816Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9166248Z return func(*args, **kwargs) 2025-12-04T08:54:30.9166659Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T08:54:30.9167094Z attention_output = self.SelfAttention( 2025-12-04T08:54:30.9167536Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9167968Z return func(*args, **kwargs) 2025-12-04T08:54:30.9168374Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 367, in forward 2025-12-04T08:54:30.9168809Z query_states = self.q(hidden_states) 2025-12-04T08:54:30.9168971Z 2025-12-04T08:54:30.9169092Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9169506Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9169865Z res = mod(**inputs) 2025-12-04T08:54:30.9170268Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T08:54:30.9170704Z encoder_outputs = self.encoder( 2025-12-04T08:54:30.9171127Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9171551Z layer_outputs = layer_module( 2025-12-04T08:54:30.9171951Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9172369Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9172813Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9173237Z return func(*args, **kwargs) 2025-12-04T08:54:30.9173648Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T08:54:30.9174094Z self_attention_outputs = self.layer[0]( 2025-12-04T08:54:30.9174522Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9174930Z return func(*args, **kwargs) 2025-12-04T08:54:30.9175347Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T08:54:30.9175768Z attention_output = self.SelfAttention( 2025-12-04T08:54:30.9176178Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9176582Z return func(*args, **kwargs) 2025-12-04T08:54:30.9176976Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 388, in forward 2025-12-04T08:54:30.9177380Z key_states = self.k(current_states) 2025-12-04T08:54:30.9177550Z 2025-12-04T08:54:30.9177664Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9178066Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9178423Z res = mod(**inputs) 2025-12-04T08:54:30.9178799Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T08:54:30.9179218Z encoder_outputs = self.encoder( 2025-12-04T08:54:30.9179627Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9180031Z layer_outputs = layer_module( 2025-12-04T08:54:30.9180418Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9180822Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9181237Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9181643Z return func(*args, **kwargs) 2025-12-04T08:54:30.9182040Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T08:54:30.9182457Z self_attention_outputs = self.layer[0]( 2025-12-04T08:54:30.9182882Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9183282Z return func(*args, **kwargs) 2025-12-04T08:54:30.9183673Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T08:54:30.9184097Z attention_output = self.SelfAttention( 2025-12-04T08:54:30.9184506Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9184910Z return func(*args, **kwargs) 2025-12-04T08:54:30.9185303Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 389, in forward 2025-12-04T08:54:30.9185724Z value_states = self.v(current_states) 2025-12-04T08:54:30.9185872Z 2025-12-04T08:54:30.9185962Z cudagraph partition due to non gpu ops 2025-12-04T08:54:30.9186204Z cudagraph partition due to non gpu ops 2025-12-04T08:54:30.9186461Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9186846Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9187203Z res = mod(**inputs) 2025-12-04T08:54:30.9187613Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T08:54:30.9188044Z encoder_outputs = self.encoder( 2025-12-04T08:54:30.9188464Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9188903Z layer_outputs = layer_module( 2025-12-04T08:54:30.9189304Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9189700Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9190129Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9190564Z return func(*args, **kwargs) 2025-12-04T08:54:30.9190965Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T08:54:30.9191405Z self_attention_outputs = self.layer[0]( 2025-12-04T08:54:30.9191832Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9192247Z return func(*args, **kwargs) 2025-12-04T08:54:30.9192638Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T08:54:30.9193099Z attention_output = self.SelfAttention( 2025-12-04T08:54:30.9193531Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9193939Z return func(*args, **kwargs) 2025-12-04T08:54:30.9194340Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 447, in forward 2025-12-04T08:54:30.9194775Z attn_output = self.o(attn_output) 2025-12-04T08:54:30.9194925Z 2025-12-04T08:54:30.9195049Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9195461Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9195830Z res = mod(**inputs) 2025-12-04T08:54:30.9196232Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T08:54:30.9196669Z encoder_outputs = self.encoder( 2025-12-04T08:54:30.9197088Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9197515Z layer_outputs = layer_module( 2025-12-04T08:54:30.9197903Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9198406Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9198855Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9199288Z return func(*args, **kwargs) 2025-12-04T08:54:30.9199711Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T08:54:30.9200163Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T08:54:30.9200620Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T08:54:30.9201099Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T08:54:30.9201583Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 184, in forward 2025-12-04T08:54:30.9202080Z hidden_gelu = self.act(self.wi_0(hidden_states)) 2025-12-04T08:54:30.9202268Z 2025-12-04T08:54:30.9202382Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9202778Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9203126Z res = mod(**inputs) 2025-12-04T08:54:30.9203576Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T08:54:30.9204005Z encoder_outputs = self.encoder( 2025-12-04T08:54:30.9204422Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9204837Z layer_outputs = layer_module( 2025-12-04T08:54:30.9205252Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9205651Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9206055Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9206499Z return func(*args, **kwargs) 2025-12-04T08:54:30.9206893Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T08:54:30.9207323Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T08:54:30.9207748Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T08:54:30.9208207Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T08:54:30.9208666Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 185, in forward 2025-12-04T08:54:30.9209095Z hidden_linear = self.wi_1(hidden_states) 2025-12-04T08:54:30.9209254Z 2025-12-04T08:54:30.9209371Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9209774Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9210143Z res = mod(**inputs) 2025-12-04T08:54:30.9210525Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T08:54:30.9210947Z encoder_outputs = self.encoder( 2025-12-04T08:54:30.9211365Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9211784Z layer_outputs = layer_module( 2025-12-04T08:54:30.9212163Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9212571Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9212994Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9213403Z return func(*args, **kwargs) 2025-12-04T08:54:30.9213806Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T08:54:30.9214243Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T08:54:30.9214675Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T08:54:30.9215127Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T08:54:30.9215599Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 186, in forward 2025-12-04T08:54:30.9216059Z hidden_states = hidden_gelu * hidden_linear 2025-12-04T08:54:30.9216224Z 2025-12-04T08:54:30.9216351Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9216761Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9217132Z res = mod(**inputs) 2025-12-04T08:54:30.9217532Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T08:54:30.9217958Z encoder_outputs = self.encoder( 2025-12-04T08:54:30.9218382Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9218810Z layer_outputs = layer_module( 2025-12-04T08:54:30.9219217Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9219614Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9220035Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9220450Z return func(*args, **kwargs) 2025-12-04T08:54:30.9221197Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T08:54:30.9221659Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T08:54:30.9222143Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T08:54:30.9222623Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T08:54:30.9223078Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 199, in forward 2025-12-04T08:54:30.9223507Z hidden_states = self.wo(hidden_states) 2025-12-04T08:54:30.9223660Z 2025-12-04T08:54:30.9223787Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9224197Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9224577Z res = mod(**inputs) 2025-12-04T08:54:30.9224967Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T08:54:30.9225383Z encoder_outputs = self.encoder( 2025-12-04T08:54:30.9225784Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9226198Z layer_outputs = layer_module( 2025-12-04T08:54:30.9226580Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9226980Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9227391Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9227807Z return func(*args, **kwargs) 2025-12-04T08:54:30.9228206Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T08:54:30.9228624Z self_attention_outputs = self.layer[0]( 2025-12-04T08:54:30.9229059Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9229487Z return func(*args, **kwargs) 2025-12-04T08:54:30.9229879Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T08:54:30.9230299Z attention_output = self.SelfAttention( 2025-12-04T08:54:30.9230739Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9231147Z return func(*args, **kwargs) 2025-12-04T08:54:30.9231533Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 367, in forward 2025-12-04T08:54:30.9231951Z query_states = self.q(hidden_states) 2025-12-04T08:54:30.9232107Z 2025-12-04T08:54:30.9232220Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9232612Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9232958Z res = mod(**inputs) 2025-12-04T08:54:30.9233343Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T08:54:30.9233758Z encoder_outputs = self.encoder( 2025-12-04T08:54:30.9234173Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9234630Z layer_outputs = layer_module( 2025-12-04T08:54:30.9235025Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9235437Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9235867Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9236313Z return func(*args, **kwargs) 2025-12-04T08:54:30.9236729Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T08:54:30.9237169Z self_attention_outputs = self.layer[0]( 2025-12-04T08:54:30.9237624Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9238047Z return func(*args, **kwargs) 2025-12-04T08:54:30.9238521Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T08:54:30.9238965Z attention_output = self.SelfAttention( 2025-12-04T08:54:30.9239408Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9239835Z return func(*args, **kwargs) 2025-12-04T08:54:30.9240274Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 388, in forward 2025-12-04T08:54:30.9240702Z key_states = self.k(current_states) 2025-12-04T08:54:30.9240860Z 2025-12-04T08:54:30.9240980Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9241398Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9241760Z res = mod(**inputs) 2025-12-04T08:54:30.9242159Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T08:54:30.9242591Z encoder_outputs = self.encoder( 2025-12-04T08:54:30.9243023Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9243449Z layer_outputs = layer_module( 2025-12-04T08:54:30.9243840Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9244250Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9244681Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9245099Z return func(*args, **kwargs) 2025-12-04T08:54:30.9245509Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T08:54:30.9246001Z self_attention_outputs = self.layer[0]( 2025-12-04T08:54:30.9246429Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9246845Z return func(*args, **kwargs) 2025-12-04T08:54:30.9247247Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T08:54:30.9247687Z attention_output = self.SelfAttention( 2025-12-04T08:54:30.9248112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9248528Z return func(*args, **kwargs) 2025-12-04T08:54:30.9248930Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 389, in forward 2025-12-04T08:54:30.9249350Z value_states = self.v(current_states) 2025-12-04T08:54:30.9249513Z 2025-12-04T08:54:30.9249606Z cudagraph partition due to non gpu ops 2025-12-04T08:54:30.9249848Z cudagraph partition due to non gpu ops 2025-12-04T08:54:30.9250114Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9250546Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9250915Z res = mod(**inputs) 2025-12-04T08:54:30.9251319Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T08:54:30.9251750Z encoder_outputs = self.encoder( 2025-12-04T08:54:30.9252240Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9252670Z layer_outputs = layer_module( 2025-12-04T08:54:30.9253065Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9253496Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9253914Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9254328Z return func(*args, **kwargs) 2025-12-04T08:54:30.9254715Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T08:54:30.9255136Z self_attention_outputs = self.layer[0]( 2025-12-04T08:54:30.9255556Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9255980Z return func(*args, **kwargs) 2025-12-04T08:54:30.9256362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T08:54:30.9256785Z attention_output = self.SelfAttention( 2025-12-04T08:54:30.9257202Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9257612Z return func(*args, **kwargs) 2025-12-04T08:54:30.9257999Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 447, in forward 2025-12-04T08:54:30.9258417Z attn_output = self.o(attn_output) 2025-12-04T08:54:30.9258557Z 2025-12-04T08:54:30.9258679Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9259071Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9259424Z res = mod(**inputs) 2025-12-04T08:54:30.9259807Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T08:54:30.9260225Z encoder_outputs = self.encoder( 2025-12-04T08:54:30.9260694Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9261138Z layer_outputs = layer_module( 2025-12-04T08:54:30.9261525Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9261919Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9262344Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9262727Z return func(*args, **kwargs) 2025-12-04T08:54:30.9263092Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T08:54:30.9263491Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T08:54:30.9263907Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T08:54:30.9264343Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T08:54:30.9264781Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 184, in forward 2025-12-04T08:54:30.9265186Z hidden_gelu = self.act(self.wi_0(hidden_states)) 2025-12-04T08:54:30.9265350Z 2025-12-04T08:54:30.9265482Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9265852Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9266179Z res = mod(**inputs) 2025-12-04T08:54:30.9266542Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T08:54:30.9266937Z encoder_outputs = self.encoder( 2025-12-04T08:54:30.9267346Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9267742Z layer_outputs = layer_module( 2025-12-04T08:54:30.9268132Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9268552Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9268945Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9269334Z return func(*args, **kwargs) 2025-12-04T08:54:30.9269715Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T08:54:30.9270113Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T08:54:30.9270517Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T08:54:30.9270937Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T08:54:30.9271355Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 185, in forward 2025-12-04T08:54:30.9271750Z hidden_linear = self.wi_1(hidden_states) 2025-12-04T08:54:30.9271890Z 2025-12-04T08:54:30.9271997Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9272366Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9272701Z res = mod(**inputs) 2025-12-04T08:54:30.9273057Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T08:54:30.9273452Z encoder_outputs = self.encoder( 2025-12-04T08:54:30.9273841Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9274233Z layer_outputs = layer_module( 2025-12-04T08:54:30.9274586Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9274977Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9275399Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9275797Z return func(*args, **kwargs) 2025-12-04T08:54:30.9276196Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T08:54:30.9276629Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T08:54:30.9277056Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T08:54:30.9277506Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T08:54:30.9277965Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 186, in forward 2025-12-04T08:54:30.9278469Z hidden_states = hidden_gelu * hidden_linear 2025-12-04T08:54:30.9278632Z 2025-12-04T08:54:30.9278754Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9279155Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9279541Z res = mod(**inputs) 2025-12-04T08:54:30.9279949Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T08:54:30.9280396Z encoder_outputs = self.encoder( 2025-12-04T08:54:30.9280804Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9281218Z layer_outputs = layer_module( 2025-12-04T08:54:30.9281603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9282018Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9282431Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9282838Z return func(*args, **kwargs) 2025-12-04T08:54:30.9283254Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T08:54:30.9283675Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T08:54:30.9284108Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T08:54:30.9284565Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T08:54:30.9285015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 199, in forward 2025-12-04T08:54:30.9285451Z hidden_states = self.wo(hidden_states) 2025-12-04T08:54:30.9285611Z 2025-12-04T08:54:30.9285726Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9286121Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9286465Z res = mod(**inputs) 2025-12-04T08:54:30.9286851Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T08:54:30.9287265Z encoder_outputs = self.encoder( 2025-12-04T08:54:30.9287673Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9288081Z layer_outputs = layer_module( 2025-12-04T08:54:30.9288463Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9288864Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9289290Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9289693Z return func(*args, **kwargs) 2025-12-04T08:54:30.9290089Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T08:54:30.9290511Z self_attention_outputs = self.layer[0]( 2025-12-04T08:54:30.9290922Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9291330Z return func(*args, **kwargs) 2025-12-04T08:54:30.9291729Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T08:54:30.9292166Z attention_output = self.SelfAttention( 2025-12-04T08:54:30.9292596Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9293008Z return func(*args, **kwargs) 2025-12-04T08:54:30.9293405Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 367, in forward 2025-12-04T08:54:30.9293815Z query_states = self.q(hidden_states) 2025-12-04T08:54:30.9293972Z 2025-12-04T08:54:30.9294087Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9294491Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9294846Z res = mod(**inputs) 2025-12-04T08:54:30.9295246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T08:54:30.9295677Z encoder_outputs = self.encoder( 2025-12-04T08:54:30.9296088Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9296506Z layer_outputs = layer_module( 2025-12-04T08:54:30.9296886Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9297306Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9297728Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9298148Z return func(*args, **kwargs) 2025-12-04T08:54:30.9298544Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T08:54:30.9298965Z self_attention_outputs = self.layer[0]( 2025-12-04T08:54:30.9299392Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9299794Z return func(*args, **kwargs) 2025-12-04T08:54:30.9300232Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T08:54:30.9300679Z attention_output = self.SelfAttention( 2025-12-04T08:54:30.9301097Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9301511Z return func(*args, **kwargs) 2025-12-04T08:54:30.9301909Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 388, in forward 2025-12-04T08:54:30.9302339Z key_states = self.k(current_states) 2025-12-04T08:54:30.9302484Z 2025-12-04T08:54:30.9302599Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9303000Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9303357Z res = mod(**inputs) 2025-12-04T08:54:30.9303731Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T08:54:30.9304129Z encoder_outputs = self.encoder( 2025-12-04T08:54:30.9304515Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9304927Z layer_outputs = layer_module( 2025-12-04T08:54:30.9305299Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9305698Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9306113Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9306520Z return func(*args, **kwargs) 2025-12-04T08:54:30.9306887Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T08:54:30.9307306Z self_attention_outputs = self.layer[0]( 2025-12-04T08:54:30.9307728Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9308127Z return func(*args, **kwargs) 2025-12-04T08:54:30.9308524Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T08:54:30.9308947Z attention_output = self.SelfAttention( 2025-12-04T08:54:30.9309364Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9309765Z return func(*args, **kwargs) 2025-12-04T08:54:30.9310158Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 389, in forward 2025-12-04T08:54:30.9310573Z value_states = self.v(current_states) 2025-12-04T08:54:30.9310756Z 2025-12-04T08:54:30.9310846Z cudagraph partition due to non gpu ops 2025-12-04T08:54:30.9311082Z cudagraph partition due to non gpu ops 2025-12-04T08:54:30.9311339Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9311736Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9312085Z res = mod(**inputs) 2025-12-04T08:54:30.9312489Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T08:54:30.9312913Z encoder_outputs = self.encoder( 2025-12-04T08:54:30.9313337Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9313758Z layer_outputs = layer_module( 2025-12-04T08:54:30.9314156Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9314568Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9314990Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9315412Z return func(*args, **kwargs) 2025-12-04T08:54:30.9319330Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T08:54:30.9320257Z self_attention_outputs = self.layer[0]( 2025-12-04T08:54:30.9320908Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9321403Z return func(*args, **kwargs) 2025-12-04T08:54:30.9321837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T08:54:30.9322310Z attention_output = self.SelfAttention( 2025-12-04T08:54:30.9322778Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9323202Z return func(*args, **kwargs) 2025-12-04T08:54:30.9323610Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 447, in forward 2025-12-04T08:54:30.9324050Z attn_output = self.o(attn_output) 2025-12-04T08:54:30.9324211Z 2025-12-04T08:54:30.9324341Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9324761Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9325128Z res = mod(**inputs) 2025-12-04T08:54:30.9325538Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T08:54:30.9325972Z encoder_outputs = self.encoder( 2025-12-04T08:54:30.9326401Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9326822Z layer_outputs = layer_module( 2025-12-04T08:54:30.9327227Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9327641Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9328068Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9328492Z return func(*args, **kwargs) 2025-12-04T08:54:30.9328898Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T08:54:30.9329330Z self_attention_outputs = self.layer[0]( 2025-12-04T08:54:30.9329766Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9330170Z return func(*args, **kwargs) 2025-12-04T08:54:30.9330907Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 489, in forward 2025-12-04T08:54:30.9331391Z hidden_states = hidden_states + self.dropout(attention_output[0]) 2025-12-04T08:54:30.9331597Z 2025-12-04T08:54:30.9331716Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9332118Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9332479Z res = mod(**inputs) 2025-12-04T08:54:30.9332917Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T08:54:30.9333344Z encoder_outputs = self.encoder( 2025-12-04T08:54:30.9333806Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9334223Z layer_outputs = layer_module( 2025-12-04T08:54:30.9334603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9335012Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9335432Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9335835Z return func(*args, **kwargs) 2025-12-04T08:54:30.9336281Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T08:54:30.9336722Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T08:54:30.9337170Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T08:54:30.9337624Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T08:54:30.9338087Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 184, in forward 2025-12-04T08:54:30.9338532Z hidden_gelu = self.act(self.wi_0(hidden_states)) 2025-12-04T08:54:30.9338699Z 2025-12-04T08:54:30.9338827Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9339223Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9339589Z res = mod(**inputs) 2025-12-04T08:54:30.9339999Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T08:54:30.9340412Z encoder_outputs = self.encoder( 2025-12-04T08:54:30.9340822Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9341237Z layer_outputs = layer_module( 2025-12-04T08:54:30.9341625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9342021Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9342442Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9342859Z return func(*args, **kwargs) 2025-12-04T08:54:30.9343254Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T08:54:30.9343688Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T08:54:30.9344125Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T08:54:30.9344582Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T08:54:30.9345045Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 185, in forward 2025-12-04T08:54:30.9345483Z hidden_linear = self.wi_1(hidden_states) 2025-12-04T08:54:30.9345649Z 2025-12-04T08:54:30.9345770Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9346221Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9346594Z res = mod(**inputs) 2025-12-04T08:54:30.9346992Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T08:54:30.9347418Z encoder_outputs = self.encoder( 2025-12-04T08:54:30.9347866Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9348300Z layer_outputs = layer_module( 2025-12-04T08:54:30.9348701Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9349136Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9349554Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9349977Z return func(*args, **kwargs) 2025-12-04T08:54:30.9350388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T08:54:30.9350827Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T08:54:30.9351266Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T08:54:30.9351775Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T08:54:30.9352247Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 186, in forward 2025-12-04T08:54:30.9352697Z hidden_states = hidden_gelu * hidden_linear 2025-12-04T08:54:30.9352865Z 2025-12-04T08:54:30.9352986Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9353396Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9353767Z res = mod(**inputs) 2025-12-04T08:54:30.9354163Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T08:54:30.9354599Z encoder_outputs = self.encoder( 2025-12-04T08:54:30.9355025Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9355447Z layer_outputs = layer_module( 2025-12-04T08:54:30.9355883Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9356307Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9356742Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9357168Z return func(*args, **kwargs) 2025-12-04T08:54:30.9357585Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T08:54:30.9358040Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T08:54:30.9358650Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T08:54:30.9359133Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T08:54:30.9359615Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 199, in forward 2025-12-04T08:54:30.9360058Z hidden_states = self.wo(hidden_states) 2025-12-04T08:54:30.9360217Z 2025-12-04T08:54:30.9360339Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9360767Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9361140Z res = mod(**inputs) 2025-12-04T08:54:30.9361544Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T08:54:30.9361972Z encoder_outputs = self.encoder( 2025-12-04T08:54:30.9362469Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9362901Z layer_outputs = layer_module( 2025-12-04T08:54:30.9363296Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9363728Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9364233Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9364670Z return func(*args, **kwargs) 2025-12-04T08:54:30.9365081Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T08:54:30.9365544Z self_attention_outputs = self.layer[0]( 2025-12-04T08:54:30.9365991Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9366401Z return func(*args, **kwargs) 2025-12-04T08:54:30.9366824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T08:54:30.9367274Z attention_output = self.SelfAttention( 2025-12-04T08:54:30.9367697Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9368143Z return func(*args, **kwargs) 2025-12-04T08:54:30.9368546Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 367, in forward 2025-12-04T08:54:30.9368960Z query_states = self.q(hidden_states) 2025-12-04T08:54:30.9369111Z 2025-12-04T08:54:30.9369233Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9369624Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9369978Z res = mod(**inputs) 2025-12-04T08:54:30.9370368Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T08:54:30.9370774Z encoder_outputs = self.encoder( 2025-12-04T08:54:30.9371182Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9371597Z layer_outputs = layer_module( 2025-12-04T08:54:30.9371979Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9372374Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9372794Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9373204Z return func(*args, **kwargs) 2025-12-04T08:54:30.9373590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T08:54:30.9374024Z self_attention_outputs = self.layer[0]( 2025-12-04T08:54:30.9374442Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9374847Z return func(*args, **kwargs) 2025-12-04T08:54:30.9375229Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T08:54:30.9375656Z attention_output = self.SelfAttention( 2025-12-04T08:54:30.9376079Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9376495Z return func(*args, **kwargs) 2025-12-04T08:54:30.9376883Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 388, in forward 2025-12-04T08:54:30.9377304Z key_states = self.k(current_states) 2025-12-04T08:54:30.9377451Z 2025-12-04T08:54:30.9377575Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9377999Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9378359Z res = mod(**inputs) 2025-12-04T08:54:30.9378743Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T08:54:30.9379160Z encoder_outputs = self.encoder( 2025-12-04T08:54:30.9379594Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9380014Z layer_outputs = layer_module( 2025-12-04T08:54:30.9380404Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9380828Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9381254Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9381671Z return func(*args, **kwargs) 2025-12-04T08:54:30.9382074Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T08:54:30.9382528Z self_attention_outputs = self.layer[0]( 2025-12-04T08:54:30.9382993Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9383453Z return func(*args, **kwargs) 2025-12-04T08:54:30.9383852Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T08:54:30.9384319Z attention_output = self.SelfAttention( 2025-12-04T08:54:30.9384750Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9385165Z return func(*args, **kwargs) 2025-12-04T08:54:30.9385565Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 389, in forward 2025-12-04T08:54:30.9386004Z value_states = self.v(current_states) 2025-12-04T08:54:30.9386157Z 2025-12-04T08:54:30.9386253Z cudagraph partition due to non gpu ops 2025-12-04T08:54:30.9386490Z cudagraph partition due to non gpu ops 2025-12-04T08:54:30.9386745Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9387148Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9387517Z res = mod(**inputs) 2025-12-04T08:54:30.9387907Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T08:54:30.9388326Z encoder_outputs = self.encoder( 2025-12-04T08:54:30.9388749Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9389170Z layer_outputs = layer_module( 2025-12-04T08:54:30.9389546Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9389946Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9390360Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9390774Z return func(*args, **kwargs) 2025-12-04T08:54:30.9391166Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T08:54:30.9391585Z self_attention_outputs = self.layer[0]( 2025-12-04T08:54:30.9392003Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9392407Z return func(*args, **kwargs) 2025-12-04T08:54:30.9392801Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T08:54:30.9393232Z attention_output = self.SelfAttention( 2025-12-04T08:54:30.9393674Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9394078Z return func(*args, **kwargs) 2025-12-04T08:54:30.9394474Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 447, in forward 2025-12-04T08:54:30.9394889Z attn_output = self.o(attn_output) 2025-12-04T08:54:30.9395034Z 2025-12-04T08:54:30.9395174Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9395580Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9395970Z res = mod(**inputs) 2025-12-04T08:54:30.9396354Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T08:54:30.9396793Z encoder_outputs = self.encoder( 2025-12-04T08:54:30.9397203Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9397616Z layer_outputs = layer_module( 2025-12-04T08:54:30.9397992Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9398558Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9399080Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9399506Z return func(*args, **kwargs) 2025-12-04T08:54:30.9399905Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T08:54:30.9400356Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T08:54:30.9400795Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T08:54:30.9401256Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T08:54:30.9401730Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 184, in forward 2025-12-04T08:54:30.9402192Z hidden_gelu = self.act(self.wi_0(hidden_states)) 2025-12-04T08:54:30.9402362Z 2025-12-04T08:54:30.9402490Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9402886Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9403246Z res = mod(**inputs) 2025-12-04T08:54:30.9403641Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T08:54:30.9404069Z encoder_outputs = self.encoder( 2025-12-04T08:54:30.9404478Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9404903Z layer_outputs = layer_module( 2025-12-04T08:54:30.9405296Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9405699Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9406135Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9406561Z return func(*args, **kwargs) 2025-12-04T08:54:30.9406966Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T08:54:30.9407396Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T08:54:30.9407837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T08:54:30.9408308Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T08:54:30.9408769Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 185, in forward 2025-12-04T08:54:30.9409221Z hidden_linear = self.wi_1(hidden_states) 2025-12-04T08:54:30.9409381Z 2025-12-04T08:54:30.9409495Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9409885Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9410233Z res = mod(**inputs) 2025-12-04T08:54:30.9410649Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T08:54:30.9411067Z encoder_outputs = self.encoder( 2025-12-04T08:54:30.9411471Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9411897Z layer_outputs = layer_module( 2025-12-04T08:54:30.9412281Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9412686Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9413095Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9413505Z return func(*args, **kwargs) 2025-12-04T08:54:30.9413902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T08:54:30.9414360Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T08:54:30.9414796Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T08:54:30.9415254Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T08:54:30.9415712Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 186, in forward 2025-12-04T08:54:30.9416131Z hidden_states = hidden_gelu * hidden_linear 2025-12-04T08:54:30.9416294Z 2025-12-04T08:54:30.9416405Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9416798Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9417161Z res = mod(**inputs) 2025-12-04T08:54:30.9417536Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T08:54:30.9417953Z encoder_outputs = self.encoder( 2025-12-04T08:54:30.9418364Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9418780Z layer_outputs = layer_module( 2025-12-04T08:54:30.9419153Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9419554Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9419971Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9420376Z return func(*args, **kwargs) 2025-12-04T08:54:30.9420989Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T08:54:30.9421495Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T08:54:30.9421927Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T08:54:30.9422381Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T08:54:30.9422841Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 199, in forward 2025-12-04T08:54:30.9423251Z hidden_states = self.wo(hidden_states) 2025-12-04T08:54:30.9423396Z 2025-12-04T08:54:30.9423504Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9423879Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9424217Z res = mod(**inputs) 2025-12-04T08:54:30.9424658Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T08:54:30.9425045Z encoder_outputs = self.encoder( 2025-12-04T08:54:30.9425425Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9425828Z layer_outputs = layer_module( 2025-12-04T08:54:30.9426225Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9426597Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9426999Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9427450Z return func(*args, **kwargs) 2025-12-04T08:54:30.9427855Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T08:54:30.9428262Z self_attention_outputs = self.layer[0]( 2025-12-04T08:54:30.9428665Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9429051Z return func(*args, **kwargs) 2025-12-04T08:54:30.9429420Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T08:54:30.9429854Z attention_output = self.SelfAttention( 2025-12-04T08:54:30.9430249Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9430623Z return func(*args, **kwargs) 2025-12-04T08:54:30.9430998Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 367, in forward 2025-12-04T08:54:30.9431394Z query_states = self.q(hidden_states) 2025-12-04T08:54:30.9431536Z 2025-12-04T08:54:30.9431651Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9432013Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9432348Z res = mod(**inputs) 2025-12-04T08:54:30.9432711Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T08:54:30.9433104Z encoder_outputs = self.encoder( 2025-12-04T08:54:30.9433483Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9433872Z layer_outputs = layer_module( 2025-12-04T08:54:30.9434235Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9434604Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9434995Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9435383Z return func(*args, **kwargs) 2025-12-04T08:54:30.9435761Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T08:54:30.9436153Z self_attention_outputs = self.layer[0]( 2025-12-04T08:54:30.9436570Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9436997Z return func(*args, **kwargs) 2025-12-04T08:54:30.9437381Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T08:54:30.9437813Z attention_output = self.SelfAttention( 2025-12-04T08:54:30.9438305Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9438749Z return func(*args, **kwargs) 2025-12-04T08:54:30.9439180Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 388, in forward 2025-12-04T08:54:30.9439611Z key_states = self.k(current_states) 2025-12-04T08:54:30.9439760Z 2025-12-04T08:54:30.9439888Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9440295Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9440654Z res = mod(**inputs) 2025-12-04T08:54:30.9441060Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T08:54:30.9441475Z encoder_outputs = self.encoder( 2025-12-04T08:54:30.9441880Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9442353Z layer_outputs = layer_module( 2025-12-04T08:54:30.9442745Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9443162Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9443591Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9444007Z return func(*args, **kwargs) 2025-12-04T08:54:30.9444409Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T08:54:30.9444863Z self_attention_outputs = self.layer[0]( 2025-12-04T08:54:30.9445301Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9445724Z return func(*args, **kwargs) 2025-12-04T08:54:30.9446124Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T08:54:30.9446553Z attention_output = self.SelfAttention( 2025-12-04T08:54:30.9446989Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9447409Z return func(*args, **kwargs) 2025-12-04T08:54:30.9447799Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 389, in forward 2025-12-04T08:54:30.9448223Z value_states = self.v(current_states) 2025-12-04T08:54:30.9448379Z 2025-12-04T08:54:30.9448467Z cudagraph partition due to non gpu ops 2025-12-04T08:54:30.9448703Z cudagraph partition due to non gpu ops 2025-12-04T08:54:30.9448957Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9449367Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9449743Z res = mod(**inputs) 2025-12-04T08:54:30.9450140Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T08:54:30.9450564Z encoder_outputs = self.encoder( 2025-12-04T08:54:30.9450993Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9451413Z layer_outputs = layer_module( 2025-12-04T08:54:30.9451773Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9452162Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9452553Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9452935Z return func(*args, **kwargs) 2025-12-04T08:54:30.9453312Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T08:54:30.9453714Z self_attention_outputs = self.layer[0]( 2025-12-04T08:54:30.9454117Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9454529Z return func(*args, **kwargs) 2025-12-04T08:54:30.9454903Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T08:54:30.9455302Z attention_output = self.SelfAttention( 2025-12-04T08:54:30.9455703Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9456085Z return func(*args, **kwargs) 2025-12-04T08:54:30.9456449Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 447, in forward 2025-12-04T08:54:30.9456836Z attn_output = self.o(attn_output) 2025-12-04T08:54:30.9456986Z 2025-12-04T08:54:30.9457090Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9457457Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9457783Z res = mod(**inputs) 2025-12-04T08:54:30.9458139Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T08:54:30.9458514Z encoder_outputs = self.encoder( 2025-12-04T08:54:30.9458895Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9459303Z layer_outputs = layer_module( 2025-12-04T08:54:30.9459656Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9460033Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9460423Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9460809Z return func(*args, **kwargs) 2025-12-04T08:54:30.9461183Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T08:54:30.9461618Z self_attention_outputs = self.layer[0]( 2025-12-04T08:54:30.9462007Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9462389Z return func(*args, **kwargs) 2025-12-04T08:54:30.9462752Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 489, in forward 2025-12-04T08:54:30.9463206Z hidden_states = hidden_states + self.dropout(attention_output[0]) 2025-12-04T08:54:30.9463392Z 2025-12-04T08:54:30.9463501Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9463856Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9464188Z res = mod(**inputs) 2025-12-04T08:54:30.9464543Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T08:54:30.9464927Z encoder_outputs = self.encoder( 2025-12-04T08:54:30.9465300Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9465685Z layer_outputs = layer_module( 2025-12-04T08:54:30.9466043Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9466410Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9466801Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9467181Z return func(*args, **kwargs) 2025-12-04T08:54:30.9467555Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T08:54:30.9467944Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T08:54:30.9468345Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T08:54:30.9468800Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T08:54:30.9469236Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 184, in forward 2025-12-04T08:54:30.9469646Z hidden_gelu = self.act(self.wi_0(hidden_states)) 2025-12-04T08:54:30.9469815Z 2025-12-04T08:54:30.9469922Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9470313Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9470645Z res = mod(**inputs) 2025-12-04T08:54:30.9471014Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T08:54:30.9471424Z encoder_outputs = self.encoder( 2025-12-04T08:54:30.9471813Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9472194Z layer_outputs = layer_module( 2025-12-04T08:54:30.9472557Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9472930Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9473313Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9473741Z return func(*args, **kwargs) 2025-12-04T08:54:30.9474137Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T08:54:30.9474569Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T08:54:30.9474994Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T08:54:30.9475462Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T08:54:30.9475916Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 185, in forward 2025-12-04T08:54:30.9476336Z hidden_linear = self.wi_1(hidden_states) 2025-12-04T08:54:30.9476483Z 2025-12-04T08:54:30.9476597Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9476993Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9477353Z res = mod(**inputs) 2025-12-04T08:54:30.9477731Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T08:54:30.9478145Z encoder_outputs = self.encoder( 2025-12-04T08:54:30.9478651Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9479084Z layer_outputs = layer_module( 2025-12-04T08:54:30.9479458Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9479863Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9480285Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9480698Z return func(*args, **kwargs) 2025-12-04T08:54:30.9481097Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T08:54:30.9481530Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T08:54:30.9481945Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T08:54:30.9482366Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T08:54:30.9482793Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 186, in forward 2025-12-04T08:54:30.9483221Z hidden_states = hidden_gelu * hidden_linear 2025-12-04T08:54:30.9483374Z 2025-12-04T08:54:30.9483516Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9483902Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9484256Z res = mod(**inputs) 2025-12-04T08:54:30.9484639Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T08:54:30.9485047Z encoder_outputs = self.encoder( 2025-12-04T08:54:30.9485470Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9485886Z layer_outputs = layer_module( 2025-12-04T08:54:30.9486291Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9486684Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9487105Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9487514Z return func(*args, **kwargs) 2025-12-04T08:54:30.9487904Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T08:54:30.9488342Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T08:54:30.9488798Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T08:54:30.9489255Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T08:54:30.9489700Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 199, in forward 2025-12-04T08:54:30.9490122Z hidden_states = self.wo(hidden_states) 2025-12-04T08:54:30.9490269Z 2025-12-04T08:54:30.9490389Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9490790Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9491150Z res = mod(**inputs) 2025-12-04T08:54:30.9491537Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T08:54:30.9491950Z encoder_outputs = self.encoder( 2025-12-04T08:54:30.9492348Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9492762Z layer_outputs = layer_module( 2025-12-04T08:54:30.9493143Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9493539Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9493948Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9494356Z return func(*args, **kwargs) 2025-12-04T08:54:30.9494757Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T08:54:30.9495177Z self_attention_outputs = self.layer[0]( 2025-12-04T08:54:30.9495598Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9496009Z return func(*args, **kwargs) 2025-12-04T08:54:30.9496408Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T08:54:30.9496826Z attention_output = self.SelfAttention( 2025-12-04T08:54:30.9497248Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9497657Z return func(*args, **kwargs) 2025-12-04T08:54:30.9498030Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 367, in forward 2025-12-04T08:54:30.9498441Z query_states = self.q(hidden_states) 2025-12-04T08:54:30.9498594Z 2025-12-04T08:54:30.9498728Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9499120Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9499444Z res = mod(**inputs) 2025-12-04T08:54:30.9499805Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T08:54:30.9500225Z encoder_outputs = self.encoder( 2025-12-04T08:54:30.9500611Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9500992Z layer_outputs = layer_module( 2025-12-04T08:54:30.9501376Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9501758Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9502171Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9502579Z return func(*args, **kwargs) 2025-12-04T08:54:30.9502973Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T08:54:30.9503408Z self_attention_outputs = self.layer[0]( 2025-12-04T08:54:30.9503854Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9504242Z return func(*args, **kwargs) 2025-12-04T08:54:30.9504635Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T08:54:30.9505062Z attention_output = self.SelfAttention( 2025-12-04T08:54:30.9505474Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9505878Z return func(*args, **kwargs) 2025-12-04T08:54:30.9506272Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 388, in forward 2025-12-04T08:54:30.9506681Z key_states = self.k(current_states) 2025-12-04T08:54:30.9506838Z 2025-12-04T08:54:30.9506953Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9507347Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9507703Z res = mod(**inputs) 2025-12-04T08:54:30.9508071Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T08:54:30.9508156Z encoder_outputs = self.encoder( 2025-12-04T08:54:30.9508420Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9508500Z layer_outputs = layer_module( 2025-12-04T08:54:30.9508763Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9508850Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9509118Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9509192Z return func(*args, **kwargs) 2025-12-04T08:54:30.9509460Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T08:54:30.9509556Z self_attention_outputs = self.layer[0]( 2025-12-04T08:54:30.9509818Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9509894Z return func(*args, **kwargs) 2025-12-04T08:54:30.9510175Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T08:54:30.9510265Z attention_output = self.SelfAttention( 2025-12-04T08:54:30.9510557Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9510634Z return func(*args, **kwargs) 2025-12-04T08:54:30.9510893Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 389, in forward 2025-12-04T08:54:30.9510987Z value_states = self.v(current_states) 2025-12-04T08:54:30.9510991Z 2025-12-04T08:54:30.9511099Z cudagraph partition due to non gpu ops 2025-12-04T08:54:30.9511196Z cudagraph partition due to non gpu ops 2025-12-04T08:54:30.9511311Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9511549Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9511625Z res = mod(**inputs) 2025-12-04T08:54:30.9511890Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T08:54:30.9511973Z encoder_outputs = self.encoder( 2025-12-04T08:54:30.9512256Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9512331Z layer_outputs = layer_module( 2025-12-04T08:54:30.9512570Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9512673Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9512932Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9513014Z return func(*args, **kwargs) 2025-12-04T08:54:30.9513276Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T08:54:30.9513363Z self_attention_outputs = self.layer[0]( 2025-12-04T08:54:30.9513632Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9513705Z return func(*args, **kwargs) 2025-12-04T08:54:30.9513974Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T08:54:30.9514063Z attention_output = self.SelfAttention( 2025-12-04T08:54:30.9514328Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9514408Z return func(*args, **kwargs) 2025-12-04T08:54:30.9514667Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 447, in forward 2025-12-04T08:54:30.9514754Z attn_output = self.o(attn_output) 2025-12-04T08:54:30.9514764Z 2025-12-04T08:54:30.9514876Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9515092Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9515170Z res = mod(**inputs) 2025-12-04T08:54:30.9515443Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T08:54:30.9515521Z encoder_outputs = self.encoder( 2025-12-04T08:54:30.9515790Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9515867Z layer_outputs = layer_module( 2025-12-04T08:54:30.9516115Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9516200Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9516474Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9516557Z return func(*args, **kwargs) 2025-12-04T08:54:30.9516856Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T08:54:30.9516960Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T08:54:30.9517236Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T08:54:30.9517366Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T08:54:30.9517666Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 184, in forward 2025-12-04T08:54:30.9517776Z hidden_gelu = self.act(self.wi_0(hidden_states)) 2025-12-04T08:54:30.9517780Z 2025-12-04T08:54:30.9517897Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9518149Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9518302Z res = mod(**inputs) 2025-12-04T08:54:30.9518589Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T08:54:30.9518676Z encoder_outputs = self.encoder( 2025-12-04T08:54:30.9518948Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9519034Z layer_outputs = layer_module( 2025-12-04T08:54:30.9519307Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9519397Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9519678Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9519765Z return func(*args, **kwargs) 2025-12-04T08:54:30.9520031Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T08:54:30.9520129Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T08:54:30.9520390Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T08:54:30.9520526Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T08:54:30.9521086Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 185, in forward 2025-12-04T08:54:30.9521193Z hidden_linear = self.wi_1(hidden_states) 2025-12-04T08:54:30.9521206Z 2025-12-04T08:54:30.9521324Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9521541Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9521620Z res = mod(**inputs) 2025-12-04T08:54:30.9521890Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T08:54:30.9521969Z encoder_outputs = self.encoder( 2025-12-04T08:54:30.9522244Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9522321Z layer_outputs = layer_module( 2025-12-04T08:54:30.9522571Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9522657Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9522923Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9523006Z return func(*args, **kwargs) 2025-12-04T08:54:30.9523269Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T08:54:30.9523381Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T08:54:30.9523650Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T08:54:30.9523777Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T08:54:30.9524103Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 186, in forward 2025-12-04T08:54:30.9524202Z hidden_states = hidden_gelu * hidden_linear 2025-12-04T08:54:30.9524206Z 2025-12-04T08:54:30.9524318Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9524543Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9524643Z res = mod(**inputs) 2025-12-04T08:54:30.9524917Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T08:54:30.9525024Z encoder_outputs = self.encoder( 2025-12-04T08:54:30.9525294Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9525380Z layer_outputs = layer_module( 2025-12-04T08:54:30.9525630Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9525717Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9525992Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9526099Z return func(*args, **kwargs) 2025-12-04T08:54:30.9526368Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T08:54:30.9526464Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T08:54:30.9526723Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T08:54:30.9526855Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T08:54:30.9527115Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 199, in forward 2025-12-04T08:54:30.9527204Z hidden_states = self.wo(hidden_states) 2025-12-04T08:54:30.9527214Z 2025-12-04T08:54:30.9527325Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9527543Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9527618Z res = mod(**inputs) 2025-12-04T08:54:30.9527885Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T08:54:30.9527964Z encoder_outputs = self.encoder( 2025-12-04T08:54:30.9528234Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9528313Z layer_outputs = layer_module( 2025-12-04T08:54:30.9528561Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9528646Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9528910Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9528993Z return func(*args, **kwargs) 2025-12-04T08:54:30.9529252Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T08:54:30.9529342Z self_attention_outputs = self.layer[0]( 2025-12-04T08:54:30.9529617Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9529691Z return func(*args, **kwargs) 2025-12-04T08:54:30.9529959Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T08:54:30.9530053Z attention_output = self.SelfAttention( 2025-12-04T08:54:30.9530317Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9530397Z return func(*args, **kwargs) 2025-12-04T08:54:30.9530701Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 367, in forward 2025-12-04T08:54:30.9530788Z query_states = self.q(hidden_states) 2025-12-04T08:54:30.9530799Z 2025-12-04T08:54:30.9530909Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9531144Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9531220Z res = mod(**inputs) 2025-12-04T08:54:30.9531484Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T08:54:30.9531579Z encoder_outputs = self.encoder( 2025-12-04T08:54:30.9531852Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9531929Z layer_outputs = layer_module( 2025-12-04T08:54:30.9532179Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9532264Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9532525Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9532623Z return func(*args, **kwargs) 2025-12-04T08:54:30.9532892Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T08:54:30.9532982Z self_attention_outputs = self.layer[0]( 2025-12-04T08:54:30.9533253Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9533330Z return func(*args, **kwargs) 2025-12-04T08:54:30.9533601Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T08:54:30.9533690Z attention_output = self.SelfAttention( 2025-12-04T08:54:30.9533956Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9534036Z return func(*args, **kwargs) 2025-12-04T08:54:30.9534302Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 388, in forward 2025-12-04T08:54:30.9534388Z key_states = self.k(current_states) 2025-12-04T08:54:30.9534401Z 2025-12-04T08:54:30.9534511Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9534726Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9534802Z res = mod(**inputs) 2025-12-04T08:54:30.9535069Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T08:54:30.9535146Z encoder_outputs = self.encoder( 2025-12-04T08:54:30.9535424Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9535500Z layer_outputs = layer_module( 2025-12-04T08:54:30.9535754Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9535840Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9536105Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9536187Z return func(*args, **kwargs) 2025-12-04T08:54:30.9536452Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T08:54:30.9536543Z self_attention_outputs = self.layer[0]( 2025-12-04T08:54:30.9536815Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9536888Z return func(*args, **kwargs) 2025-12-04T08:54:30.9537179Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T08:54:30.9537270Z attention_output = self.SelfAttention( 2025-12-04T08:54:30.9537531Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9537612Z return func(*args, **kwargs) 2025-12-04T08:54:30.9537890Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 389, in forward 2025-12-04T08:54:30.9537975Z value_states = self.v(current_states) 2025-12-04T08:54:30.9538005Z 2025-12-04T08:54:30.9538093Z cudagraph partition due to non gpu ops 2025-12-04T08:54:30.9538179Z cudagraph partition due to non gpu ops 2025-12-04T08:54:30.9538297Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9538512Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9538582Z res = mod(**inputs) 2025-12-04T08:54:30.9538855Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T08:54:30.9538934Z decoder_outputs = self.decoder( 2025-12-04T08:54:30.9539229Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9539309Z layer_outputs = layer_module( 2025-12-04T08:54:30.9539556Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9539653Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9539916Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9539990Z return func(*args, **kwargs) 2025-12-04T08:54:30.9540262Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T08:54:30.9540348Z self_attention_outputs = self.layer[0]( 2025-12-04T08:54:30.9540624Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9540700Z return func(*args, **kwargs) 2025-12-04T08:54:30.9540965Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T08:54:30.9541062Z attention_output = self.SelfAttention( 2025-12-04T08:54:30.9541326Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9541400Z return func(*args, **kwargs) 2025-12-04T08:54:30.9541670Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 367, in forward 2025-12-04T08:54:30.9541753Z query_states = self.q(hidden_states) 2025-12-04T08:54:30.9541759Z 2025-12-04T08:54:30.9541876Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9542090Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9542159Z res = mod(**inputs) 2025-12-04T08:54:30.9542432Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T08:54:30.9542512Z decoder_outputs = self.decoder( 2025-12-04T08:54:30.9542783Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9542860Z layer_outputs = layer_module( 2025-12-04T08:54:30.9543103Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9543194Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9543481Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9543557Z return func(*args, **kwargs) 2025-12-04T08:54:30.9543825Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T08:54:30.9543916Z self_attention_outputs = self.layer[0]( 2025-12-04T08:54:30.9544200Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9544275Z return func(*args, **kwargs) 2025-12-04T08:54:30.9544538Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T08:54:30.9544654Z attention_output = self.SelfAttention( 2025-12-04T08:54:30.9544919Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9544992Z return func(*args, **kwargs) 2025-12-04T08:54:30.9545264Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 388, in forward 2025-12-04T08:54:30.9545347Z key_states = self.k(current_states) 2025-12-04T08:54:30.9545351Z 2025-12-04T08:54:30.9545470Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9545703Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9545775Z res = mod(**inputs) 2025-12-04T08:54:30.9546047Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T08:54:30.9546127Z decoder_outputs = self.decoder( 2025-12-04T08:54:30.9546399Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9546476Z layer_outputs = layer_module( 2025-12-04T08:54:30.9546721Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9546813Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9547077Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9547151Z return func(*args, **kwargs) 2025-12-04T08:54:30.9547423Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T08:54:30.9547511Z self_attention_outputs = self.layer[0]( 2025-12-04T08:54:30.9547779Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9547853Z return func(*args, **kwargs) 2025-12-04T08:54:30.9548115Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T08:54:30.9548210Z attention_output = self.SelfAttention( 2025-12-04T08:54:30.9548477Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9548549Z return func(*args, **kwargs) 2025-12-04T08:54:30.9548818Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 389, in forward 2025-12-04T08:54:30.9548903Z value_states = self.v(current_states) 2025-12-04T08:54:30.9548907Z 2025-12-04T08:54:30.9549002Z cudagraph partition due to non gpu ops 2025-12-04T08:54:30.9549088Z cudagraph partition due to non gpu ops 2025-12-04T08:54:30.9549199Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9549421Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9549490Z res = mod(**inputs) 2025-12-04T08:54:30.9549768Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T08:54:30.9549914Z decoder_outputs = self.decoder( 2025-12-04T08:54:30.9550189Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9550272Z layer_outputs = layer_module( 2025-12-04T08:54:30.9550511Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9550612Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9550899Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9550972Z return func(*args, **kwargs) 2025-12-04T08:54:30.9551311Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T08:54:30.9551400Z self_attention_outputs = self.layer[0]( 2025-12-04T08:54:30.9551673Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9551752Z return func(*args, **kwargs) 2025-12-04T08:54:30.9552025Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T08:54:30.9552113Z attention_output = self.SelfAttention( 2025-12-04T08:54:30.9552426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9552504Z return func(*args, **kwargs) 2025-12-04T08:54:30.9552789Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 447, in forward 2025-12-04T08:54:30.9552877Z attn_output = self.o(attn_output) 2025-12-04T08:54:30.9552881Z 2025-12-04T08:54:30.9552995Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9553224Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9553297Z res = mod(**inputs) 2025-12-04T08:54:30.9553585Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T08:54:30.9553675Z decoder_outputs = self.decoder( 2025-12-04T08:54:30.9553957Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9554047Z layer_outputs = layer_module( 2025-12-04T08:54:30.9554295Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9554383Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9554673Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9554749Z return func(*args, **kwargs) 2025-12-04T08:54:30.9555032Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 589, in forward 2025-12-04T08:54:30.9555125Z cross_attention_outputs = self.layer[1]( 2025-12-04T08:54:30.9555400Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9555482Z return func(*args, **kwargs) 2025-12-04T08:54:30.9555753Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 517, in forward 2025-12-04T08:54:30.9555850Z attention_output = self.EncDecAttention( 2025-12-04T08:54:30.9556129Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9556209Z return func(*args, **kwargs) 2025-12-04T08:54:30.9556491Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 367, in forward 2025-12-04T08:54:30.9556581Z query_states = self.q(hidden_states) 2025-12-04T08:54:30.9556585Z 2025-12-04T08:54:30.9556719Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9556959Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9557028Z res = mod(**inputs) 2025-12-04T08:54:30.9557294Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T08:54:30.9557398Z encoder_outputs = self.encoder( 2025-12-04T08:54:30.9557661Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9557744Z layer_outputs = layer_module( 2025-12-04T08:54:30.9558007Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9558091Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9558469Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9558549Z return func(*args, **kwargs) 2025-12-04T08:54:30.9558829Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T08:54:30.9558920Z self_attention_outputs = self.layer[0]( 2025-12-04T08:54:30.9559214Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9559299Z return func(*args, **kwargs) 2025-12-04T08:54:30.9559578Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T08:54:30.9559671Z attention_output = self.SelfAttention( 2025-12-04T08:54:30.9559945Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9560020Z return func(*args, **kwargs) 2025-12-04T08:54:30.9560293Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 447, in forward 2025-12-04T08:54:30.9560377Z attn_output = self.o(attn_output) 2025-12-04T08:54:30.9560382Z 2025-12-04T08:54:30.9560492Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9560716Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9560790Z res = mod(**inputs) 2025-12-04T08:54:30.9561062Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T08:54:30.9561141Z encoder_outputs = self.encoder( 2025-12-04T08:54:30.9561408Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9561493Z layer_outputs = layer_module( 2025-12-04T08:54:30.9561736Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9561821Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9562091Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9562164Z return func(*args, **kwargs) 2025-12-04T08:54:30.9562432Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T08:54:30.9562518Z self_attention_outputs = self.layer[0]( 2025-12-04T08:54:30.9562780Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9562864Z return func(*args, **kwargs) 2025-12-04T08:54:30.9563124Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 489, in forward 2025-12-04T08:54:30.9563269Z hidden_states = hidden_states + self.dropout(attention_output[0]) 2025-12-04T08:54:30.9563284Z 2025-12-04T08:54:30.9563413Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9563630Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9563707Z res = mod(**inputs) 2025-12-04T08:54:30.9563971Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T08:54:30.9564070Z encoder_outputs = self.encoder( 2025-12-04T08:54:30.9564341Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9564417Z layer_outputs = layer_module( 2025-12-04T08:54:30.9564687Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9564771Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9565035Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9565116Z return func(*args, **kwargs) 2025-12-04T08:54:30.9565377Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T08:54:30.9565476Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T08:54:30.9565762Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T08:54:30.9565887Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T08:54:30.9566153Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 184, in forward 2025-12-04T08:54:30.9566262Z hidden_gelu = self.act(self.wi_0(hidden_states)) 2025-12-04T08:54:30.9566266Z 2025-12-04T08:54:30.9566381Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9566607Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9566677Z res = mod(**inputs) 2025-12-04T08:54:30.9566948Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T08:54:30.9567026Z encoder_outputs = self.encoder( 2025-12-04T08:54:30.9567294Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9567377Z layer_outputs = layer_module( 2025-12-04T08:54:30.9567621Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9567707Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9567975Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9568048Z return func(*args, **kwargs) 2025-12-04T08:54:30.9568317Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T08:54:30.9568415Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T08:54:30.9568677Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T08:54:30.9568814Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T08:54:30.9569077Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 185, in forward 2025-12-04T08:54:30.9569163Z hidden_linear = self.wi_1(hidden_states) 2025-12-04T08:54:30.9569174Z 2025-12-04T08:54:30.9569286Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9569498Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9569573Z res = mod(**inputs) 2025-12-04T08:54:30.9569861Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T08:54:30.9569942Z encoder_outputs = self.encoder( 2025-12-04T08:54:30.9570211Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9570289Z layer_outputs = layer_module( 2025-12-04T08:54:30.9570566Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9570655Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9570925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9571039Z return func(*args, **kwargs) 2025-12-04T08:54:30.9571305Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T08:54:30.9571401Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T08:54:30.9571671Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T08:54:30.9571794Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T08:54:30.9572075Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 186, in forward 2025-12-04T08:54:30.9572197Z hidden_states = hidden_gelu * hidden_linear 2025-12-04T08:54:30.9572201Z 2025-12-04T08:54:30.9572316Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9572545Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9572618Z res = mod(**inputs) 2025-12-04T08:54:30.9572901Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-12-04T08:54:30.9572982Z encoder_outputs = self.encoder( 2025-12-04T08:54:30.9573255Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9573341Z layer_outputs = layer_module( 2025-12-04T08:54:30.9573590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9573679Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9573959Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9574036Z return func(*args, **kwargs) 2025-12-04T08:54:30.9574331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T08:54:30.9574429Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T08:54:30.9574692Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T08:54:30.9574822Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T08:54:30.9575082Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 199, in forward 2025-12-04T08:54:30.9575168Z hidden_states = self.wo(hidden_states) 2025-12-04T08:54:30.9575178Z 2025-12-04T08:54:30.9575291Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9575506Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9575582Z res = mod(**inputs) 2025-12-04T08:54:30.9575847Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T08:54:30.9575928Z decoder_outputs = self.decoder( 2025-12-04T08:54:30.9576199Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9576277Z layer_outputs = layer_module( 2025-12-04T08:54:30.9576547Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9576636Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9576897Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9576981Z return func(*args, **kwargs) 2025-12-04T08:54:30.9577290Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 589, in forward 2025-12-04T08:54:30.9577381Z cross_attention_outputs = self.layer[1]( 2025-12-04T08:54:30.9577650Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9577753Z return func(*args, **kwargs) 2025-12-04T08:54:30.9578024Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 517, in forward 2025-12-04T08:54:30.9578119Z attention_output = self.EncDecAttention( 2025-12-04T08:54:30.9578384Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9578464Z return func(*args, **kwargs) 2025-12-04T08:54:30.9578728Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 388, in forward 2025-12-04T08:54:30.9578832Z key_states = self.k(current_states) 2025-12-04T08:54:30.9578844Z 2025-12-04T08:54:30.9578957Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9579175Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9579257Z res = mod(**inputs) 2025-12-04T08:54:30.9579531Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T08:54:30.9579613Z decoder_outputs = self.decoder( 2025-12-04T08:54:30.9579899Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9579978Z layer_outputs = layer_module( 2025-12-04T08:54:30.9580238Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9580325Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9580599Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9580681Z return func(*args, **kwargs) 2025-12-04T08:54:30.9580963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 589, in forward 2025-12-04T08:54:30.9581050Z cross_attention_outputs = self.layer[1]( 2025-12-04T08:54:30.9581325Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9581399Z return func(*args, **kwargs) 2025-12-04T08:54:30.9581671Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 517, in forward 2025-12-04T08:54:30.9581761Z attention_output = self.EncDecAttention( 2025-12-04T08:54:30.9582025Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9582111Z return func(*args, **kwargs) 2025-12-04T08:54:30.9582372Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 389, in forward 2025-12-04T08:54:30.9582455Z value_states = self.v(current_states) 2025-12-04T08:54:30.9582467Z 2025-12-04T08:54:30.9582554Z cudagraph partition due to non gpu ops 2025-12-04T08:54:30.9582641Z cudagraph partition due to non gpu ops 2025-12-04T08:54:30.9582758Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9582992Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9583062Z res = mod(**inputs) 2025-12-04T08:54:30.9583338Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T08:54:30.9583415Z decoder_outputs = self.decoder( 2025-12-04T08:54:30.9583708Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9583786Z layer_outputs = layer_module( 2025-12-04T08:54:30.9584031Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9584140Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9584405Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9584477Z return func(*args, **kwargs) 2025-12-04T08:54:30.9584752Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 589, in forward 2025-12-04T08:54:30.9584839Z cross_attention_outputs = self.layer[1]( 2025-12-04T08:54:30.9585110Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9585202Z return func(*args, **kwargs) 2025-12-04T08:54:30.9585468Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 517, in forward 2025-12-04T08:54:30.9585568Z attention_output = self.EncDecAttention( 2025-12-04T08:54:30.9585831Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9585908Z return func(*args, **kwargs) 2025-12-04T08:54:30.9586176Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 447, in forward 2025-12-04T08:54:30.9586259Z attn_output = self.o(attn_output) 2025-12-04T08:54:30.9586263Z 2025-12-04T08:54:30.9586382Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9586609Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9586681Z res = mod(**inputs) 2025-12-04T08:54:30.9586957Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T08:54:30.9587035Z decoder_outputs = self.decoder( 2025-12-04T08:54:30.9587321Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9587401Z layer_outputs = layer_module( 2025-12-04T08:54:30.9587643Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9587732Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9588008Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9588082Z return func(*args, **kwargs) 2025-12-04T08:54:30.9588357Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T08:54:30.9588455Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T08:54:30.9588730Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T08:54:30.9588855Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T08:54:30.9589126Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 184, in forward 2025-12-04T08:54:30.9589244Z hidden_gelu = self.act(self.wi_0(hidden_states)) 2025-12-04T08:54:30.9589248Z 2025-12-04T08:54:30.9589358Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9589616Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9589688Z res = mod(**inputs) 2025-12-04T08:54:30.9589966Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T08:54:30.9590053Z decoder_outputs = self.decoder( 2025-12-04T08:54:30.9590345Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9590425Z layer_outputs = layer_module( 2025-12-04T08:54:30.9590676Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9590777Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9591054Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9591127Z return func(*args, **kwargs) 2025-12-04T08:54:30.9591401Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T08:54:30.9591504Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T08:54:30.9591775Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T08:54:30.9591919Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T08:54:30.9592205Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 185, in forward 2025-12-04T08:54:30.9592294Z hidden_linear = self.wi_1(hidden_states) 2025-12-04T08:54:30.9592300Z 2025-12-04T08:54:30.9592422Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9592651Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9592722Z res = mod(**inputs) 2025-12-04T08:54:30.9592999Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T08:54:30.9593080Z decoder_outputs = self.decoder( 2025-12-04T08:54:30.9593367Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9593448Z layer_outputs = layer_module( 2025-12-04T08:54:30.9593698Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9593792Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9594068Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9594148Z return func(*args, **kwargs) 2025-12-04T08:54:30.9594428Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T08:54:30.9594528Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T08:54:30.9594816Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T08:54:30.9594942Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T08:54:30.9595226Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 186, in forward 2025-12-04T08:54:30.9595335Z hidden_states = hidden_gelu * hidden_linear 2025-12-04T08:54:30.9595339Z 2025-12-04T08:54:30.9595455Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9595679Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9595753Z res = mod(**inputs) 2025-12-04T08:54:30.9596032Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T08:54:30.9596122Z decoder_outputs = self.decoder( 2025-12-04T08:54:30.9596422Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9596501Z layer_outputs = layer_module( 2025-12-04T08:54:30.9596753Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9596839Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9597141Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9597217Z return func(*args, **kwargs) 2025-12-04T08:54:30.9597495Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T08:54:30.9597595Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T08:54:30.9597855Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T08:54:30.9597977Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T08:54:30.9598325Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 199, in forward 2025-12-04T08:54:30.9598424Z hidden_states = self.wo(hidden_states) 2025-12-04T08:54:30.9598452Z 2025-12-04T08:54:30.9598578Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9598800Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9598870Z res = mod(**inputs) 2025-12-04T08:54:30.9599153Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T08:54:30.9599236Z decoder_outputs = self.decoder( 2025-12-04T08:54:30.9599526Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9599605Z layer_outputs = layer_module( 2025-12-04T08:54:30.9599849Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9599942Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9600214Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9600295Z return func(*args, **kwargs) 2025-12-04T08:54:30.9600574Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T08:54:30.9600665Z self_attention_outputs = self.layer[0]( 2025-12-04T08:54:30.9600946Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9601022Z return func(*args, **kwargs) 2025-12-04T08:54:30.9601294Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T08:54:30.9601396Z attention_output = self.SelfAttention( 2025-12-04T08:54:30.9601666Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9601741Z return func(*args, **kwargs) 2025-12-04T08:54:30.9602019Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 367, in forward 2025-12-04T08:54:30.9602106Z query_states = self.q(hidden_states) 2025-12-04T08:54:30.9602110Z 2025-12-04T08:54:30.9602233Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9602458Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9602528Z res = mod(**inputs) 2025-12-04T08:54:30.9602807Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T08:54:30.9602908Z decoder_outputs = self.decoder( 2025-12-04T08:54:30.9603191Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9603271Z layer_outputs = layer_module( 2025-12-04T08:54:30.9603523Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9603647Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9603921Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9603996Z return func(*args, **kwargs) 2025-12-04T08:54:30.9604292Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T08:54:30.9604382Z self_attention_outputs = self.layer[0]( 2025-12-04T08:54:30.9604661Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9604737Z return func(*args, **kwargs) 2025-12-04T08:54:30.9605006Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T08:54:30.9605106Z attention_output = self.SelfAttention( 2025-12-04T08:54:30.9605397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9605473Z return func(*args, **kwargs) 2025-12-04T08:54:30.9605751Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 388, in forward 2025-12-04T08:54:30.9605837Z key_states = self.k(current_states) 2025-12-04T08:54:30.9605841Z 2025-12-04T08:54:30.9605962Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9606186Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9606259Z res = mod(**inputs) 2025-12-04T08:54:30.9606542Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T08:54:30.9606621Z decoder_outputs = self.decoder( 2025-12-04T08:54:30.9606897Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9606979Z layer_outputs = layer_module( 2025-12-04T08:54:30.9607229Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9607323Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9607595Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9607670Z return func(*args, **kwargs) 2025-12-04T08:54:30.9607956Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T08:54:30.9608043Z self_attention_outputs = self.layer[0]( 2025-12-04T08:54:30.9608317Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9608390Z return func(*args, **kwargs) 2025-12-04T08:54:30.9608657Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T08:54:30.9608753Z attention_output = self.SelfAttention( 2025-12-04T08:54:30.9609015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9609089Z return func(*args, **kwargs) 2025-12-04T08:54:30.9609357Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 389, in forward 2025-12-04T08:54:30.9609441Z value_states = self.v(current_states) 2025-12-04T08:54:30.9609445Z 2025-12-04T08:54:30.9609577Z cudagraph partition due to non gpu ops 2025-12-04T08:54:30.9609666Z cudagraph partition due to non gpu ops 2025-12-04T08:54:30.9609777Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9610001Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9610073Z res = mod(**inputs) 2025-12-04T08:54:30.9610357Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T08:54:30.9610444Z decoder_outputs = self.decoder( 2025-12-04T08:54:30.9610707Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9610813Z layer_outputs = layer_module( 2025-12-04T08:54:30.9611062Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9611143Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9611399Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9611469Z return func(*args, **kwargs) 2025-12-04T08:54:30.9611721Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T08:54:30.9611822Z self_attention_outputs = self.layer[0]( 2025-12-04T08:54:30.9612068Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9612144Z return func(*args, **kwargs) 2025-12-04T08:54:30.9612392Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T08:54:30.9612476Z attention_output = self.SelfAttention( 2025-12-04T08:54:30.9612733Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9612803Z return func(*args, **kwargs) 2025-12-04T08:54:30.9613056Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 447, in forward 2025-12-04T08:54:30.9613133Z attn_output = self.o(attn_output) 2025-12-04T08:54:30.9613139Z 2025-12-04T08:54:30.9613244Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9613460Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9613525Z res = mod(**inputs) 2025-12-04T08:54:30.9613782Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T08:54:30.9613858Z decoder_outputs = self.decoder( 2025-12-04T08:54:30.9614107Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9614185Z layer_outputs = layer_module( 2025-12-04T08:54:30.9614414Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9614496Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9614763Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9614838Z return func(*args, **kwargs) 2025-12-04T08:54:30.9615105Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 589, in forward 2025-12-04T08:54:30.9615192Z cross_attention_outputs = self.layer[1]( 2025-12-04T08:54:30.9615455Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9615536Z return func(*args, **kwargs) 2025-12-04T08:54:30.9615796Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 517, in forward 2025-12-04T08:54:30.9615907Z attention_output = self.EncDecAttention( 2025-12-04T08:54:30.9616179Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9616252Z return func(*args, **kwargs) 2025-12-04T08:54:30.9616535Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 367, in forward 2025-12-04T08:54:30.9616631Z query_states = self.q(hidden_states) 2025-12-04T08:54:30.9616636Z 2025-12-04T08:54:30.9616742Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9616957Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9617042Z res = mod(**inputs) 2025-12-04T08:54:30.9617306Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T08:54:30.9617381Z decoder_outputs = self.decoder( 2025-12-04T08:54:30.9617636Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9617715Z layer_outputs = layer_module( 2025-12-04T08:54:30.9617948Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9618046Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9618305Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9618375Z return func(*args, **kwargs) 2025-12-04T08:54:30.9618631Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 589, in forward 2025-12-04T08:54:30.9618714Z cross_attention_outputs = self.layer[1]( 2025-12-04T08:54:30.9618963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9619042Z return func(*args, **kwargs) 2025-12-04T08:54:30.9619289Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 517, in forward 2025-12-04T08:54:30.9619377Z attention_output = self.EncDecAttention( 2025-12-04T08:54:30.9619634Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9619705Z return func(*args, **kwargs) 2025-12-04T08:54:30.9619984Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 388, in forward 2025-12-04T08:54:30.9620067Z key_states = self.k(current_states) 2025-12-04T08:54:30.9620071Z 2025-12-04T08:54:30.9620182Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9620406Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9620474Z res = mod(**inputs) 2025-12-04T08:54:30.9620929Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T08:54:30.9621055Z decoder_outputs = self.decoder( 2025-12-04T08:54:30.9621313Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9621394Z layer_outputs = layer_module( 2025-12-04T08:54:30.9621628Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9621705Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9621963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9622032Z return func(*args, **kwargs) 2025-12-04T08:54:30.9622296Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 589, in forward 2025-12-04T08:54:30.9622429Z cross_attention_outputs = self.layer[1]( 2025-12-04T08:54:30.9622678Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9622755Z return func(*args, **kwargs) 2025-12-04T08:54:30.9623005Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 517, in forward 2025-12-04T08:54:30.9623118Z attention_output = self.EncDecAttention( 2025-12-04T08:54:30.9623373Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9623470Z return func(*args, **kwargs) 2025-12-04T08:54:30.9623722Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 389, in forward 2025-12-04T08:54:30.9623802Z value_states = self.v(current_states) 2025-12-04T08:54:30.9623806Z 2025-12-04T08:54:30.9623889Z cudagraph partition due to non gpu ops 2025-12-04T08:54:30.9623979Z cudagraph partition due to non gpu ops 2025-12-04T08:54:30.9624086Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9624292Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9624394Z res = mod(**inputs) 2025-12-04T08:54:30.9624663Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T08:54:30.9624750Z decoder_outputs = self.decoder( 2025-12-04T08:54:30.9625018Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9625101Z layer_outputs = layer_module( 2025-12-04T08:54:30.9625352Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9625434Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9625735Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9625809Z return func(*args, **kwargs) 2025-12-04T08:54:30.9626072Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 589, in forward 2025-12-04T08:54:30.9626169Z cross_attention_outputs = self.layer[1]( 2025-12-04T08:54:30.9626435Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9626509Z return func(*args, **kwargs) 2025-12-04T08:54:30.9626781Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 517, in forward 2025-12-04T08:54:30.9626868Z attention_output = self.EncDecAttention( 2025-12-04T08:54:30.9627124Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9627196Z return func(*args, **kwargs) 2025-12-04T08:54:30.9627448Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 447, in forward 2025-12-04T08:54:30.9627534Z attn_output = self.o(attn_output) 2025-12-04T08:54:30.9627539Z 2025-12-04T08:54:30.9627644Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9627859Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9627936Z res = mod(**inputs) 2025-12-04T08:54:30.9628203Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T08:54:30.9628289Z decoder_outputs = self.decoder( 2025-12-04T08:54:30.9628556Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9628633Z layer_outputs = layer_module( 2025-12-04T08:54:30.9628911Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9628995Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9629260Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9629335Z return func(*args, **kwargs) 2025-12-04T08:54:30.9629612Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T08:54:30.9629719Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T08:54:30.9629981Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T08:54:30.9630124Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T08:54:30.9630395Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 184, in forward 2025-12-04T08:54:30.9630500Z hidden_gelu = self.act(self.wi_0(hidden_states)) 2025-12-04T08:54:30.9630504Z 2025-12-04T08:54:30.9630621Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9630837Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9630923Z res = mod(**inputs) 2025-12-04T08:54:30.9631197Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T08:54:30.9631277Z decoder_outputs = self.decoder( 2025-12-04T08:54:30.9631551Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9631630Z layer_outputs = layer_module( 2025-12-04T08:54:30.9631873Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9631968Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9632231Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9632305Z return func(*args, **kwargs) 2025-12-04T08:54:30.9632575Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T08:54:30.9632673Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T08:54:30.9632942Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T08:54:30.9633065Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T08:54:30.9633328Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 185, in forward 2025-12-04T08:54:30.9633420Z hidden_linear = self.wi_1(hidden_states) 2025-12-04T08:54:30.9633424Z 2025-12-04T08:54:30.9633537Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9633760Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9633830Z res = mod(**inputs) 2025-12-04T08:54:30.9634094Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T08:54:30.9634183Z decoder_outputs = self.decoder( 2025-12-04T08:54:30.9634452Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9634529Z layer_outputs = layer_module( 2025-12-04T08:54:30.9634782Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9634869Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9635142Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9635234Z return func(*args, **kwargs) 2025-12-04T08:54:30.9635498Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T08:54:30.9635601Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T08:54:30.9635866Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T08:54:30.9636005Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T08:54:30.9636275Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 186, in forward 2025-12-04T08:54:30.9636388Z hidden_states = hidden_gelu * hidden_linear 2025-12-04T08:54:30.9636392Z 2025-12-04T08:54:30.9636512Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9636725Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9636796Z res = mod(**inputs) 2025-12-04T08:54:30.9637070Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T08:54:30.9637148Z decoder_outputs = self.decoder( 2025-12-04T08:54:30.9637426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9637526Z layer_outputs = layer_module( 2025-12-04T08:54:30.9637783Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9637875Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9638139Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9638265Z return func(*args, **kwargs) 2025-12-04T08:54:30.9638557Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T08:54:30.9638653Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T08:54:30.9638929Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T08:54:30.9639054Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T08:54:30.9639330Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 199, in forward 2025-12-04T08:54:30.9639427Z hidden_states = self.wo(hidden_states) 2025-12-04T08:54:30.9639431Z 2025-12-04T08:54:30.9639547Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9639776Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9639845Z res = mod(**inputs) 2025-12-04T08:54:30.9640128Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T08:54:30.9640217Z decoder_outputs = self.decoder( 2025-12-04T08:54:30.9640482Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9640558Z layer_outputs = layer_module( 2025-12-04T08:54:30.9640822Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9640937Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9641354Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9641434Z return func(*args, **kwargs) 2025-12-04T08:54:30.9641704Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T08:54:30.9641799Z self_attention_outputs = self.layer[0]( 2025-12-04T08:54:30.9642090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9642170Z return func(*args, **kwargs) 2025-12-04T08:54:30.9642445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T08:54:30.9642539Z attention_output = self.SelfAttention( 2025-12-04T08:54:30.9642834Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9642911Z return func(*args, **kwargs) 2025-12-04T08:54:30.9643179Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 367, in forward 2025-12-04T08:54:30.9643294Z query_states = self.q(hidden_states) 2025-12-04T08:54:30.9643298Z 2025-12-04T08:54:30.9643414Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9643645Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9643717Z res = mod(**inputs) 2025-12-04T08:54:30.9643991Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T08:54:30.9644081Z decoder_outputs = self.decoder( 2025-12-04T08:54:30.9644372Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9644453Z layer_outputs = layer_module( 2025-12-04T08:54:30.9644714Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9644800Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9645092Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9645164Z return func(*args, **kwargs) 2025-12-04T08:54:30.9645427Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T08:54:30.9645522Z self_attention_outputs = self.layer[0]( 2025-12-04T08:54:30.9645782Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9645856Z return func(*args, **kwargs) 2025-12-04T08:54:30.9646125Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T08:54:30.9646215Z attention_output = self.SelfAttention( 2025-12-04T08:54:30.9646482Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9646556Z return func(*args, **kwargs) 2025-12-04T08:54:30.9646817Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 388, in forward 2025-12-04T08:54:30.9646908Z key_states = self.k(current_states) 2025-12-04T08:54:30.9646912Z 2025-12-04T08:54:30.9647023Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9647245Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9647314Z res = mod(**inputs) 2025-12-04T08:54:30.9647580Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T08:54:30.9647668Z decoder_outputs = self.decoder( 2025-12-04T08:54:30.9647934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9648009Z layer_outputs = layer_module( 2025-12-04T08:54:30.9648260Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9648345Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9648642Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9648717Z return func(*args, **kwargs) 2025-12-04T08:54:30.9648993Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T08:54:30.9649089Z self_attention_outputs = self.layer[0]( 2025-12-04T08:54:30.9649396Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9649472Z return func(*args, **kwargs) 2025-12-04T08:54:30.9649748Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T08:54:30.9649854Z attention_output = self.SelfAttention( 2025-12-04T08:54:30.9650131Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9650203Z return func(*args, **kwargs) 2025-12-04T08:54:30.9650475Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 389, in forward 2025-12-04T08:54:30.9650566Z value_states = self.v(current_states) 2025-12-04T08:54:30.9650570Z 2025-12-04T08:54:30.9650655Z cudagraph partition due to non gpu ops 2025-12-04T08:54:30.9650816Z cudagraph partition due to non gpu ops 2025-12-04T08:54:30.9650929Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9651156Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9651233Z res = mod(**inputs) 2025-12-04T08:54:30.9651511Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T08:54:30.9651592Z decoder_outputs = self.decoder( 2025-12-04T08:54:30.9651867Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9651945Z layer_outputs = layer_module( 2025-12-04T08:54:30.9652199Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9652284Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9652558Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9652641Z return func(*args, **kwargs) 2025-12-04T08:54:30.9652925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T08:54:30.9653013Z self_attention_outputs = self.layer[0]( 2025-12-04T08:54:30.9653303Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9653374Z return func(*args, **kwargs) 2025-12-04T08:54:30.9653658Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T08:54:30.9653746Z attention_output = self.SelfAttention( 2025-12-04T08:54:30.9654069Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9654152Z return func(*args, **kwargs) 2025-12-04T08:54:30.9654438Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 447, in forward 2025-12-04T08:54:30.9654520Z attn_output = self.o(attn_output) 2025-12-04T08:54:30.9654531Z 2025-12-04T08:54:30.9654641Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9654857Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9654932Z res = mod(**inputs) 2025-12-04T08:54:30.9655251Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T08:54:30.9655353Z decoder_outputs = self.decoder( 2025-12-04T08:54:30.9655644Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9655720Z layer_outputs = layer_module( 2025-12-04T08:54:30.9655975Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9656077Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9656354Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9656435Z return func(*args, **kwargs) 2025-12-04T08:54:30.9656728Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T08:54:30.9656815Z self_attention_outputs = self.layer[0]( 2025-12-04T08:54:30.9657095Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9657168Z return func(*args, **kwargs) 2025-12-04T08:54:30.9657449Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 489, in forward 2025-12-04T08:54:30.9657592Z hidden_states = hidden_states + self.dropout(attention_output[0]) 2025-12-04T08:54:30.9657613Z 2025-12-04T08:54:30.9657727Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9657951Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9658020Z res = mod(**inputs) 2025-12-04T08:54:30.9658308Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T08:54:30.9658388Z decoder_outputs = self.decoder( 2025-12-04T08:54:30.9658666Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9658751Z layer_outputs = layer_module( 2025-12-04T08:54:30.9658999Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9659082Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9659355Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9659432Z return func(*args, **kwargs) 2025-12-04T08:54:30.9659711Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 589, in forward 2025-12-04T08:54:30.9659801Z cross_attention_outputs = self.layer[1]( 2025-12-04T08:54:30.9660075Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9660156Z return func(*args, **kwargs) 2025-12-04T08:54:30.9660432Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 517, in forward 2025-12-04T08:54:30.9660529Z attention_output = self.EncDecAttention( 2025-12-04T08:54:30.9660812Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9660889Z return func(*args, **kwargs) 2025-12-04T08:54:30.9661171Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 367, in forward 2025-12-04T08:54:30.9661257Z query_states = self.q(hidden_states) 2025-12-04T08:54:30.9661261Z 2025-12-04T08:54:30.9661378Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9661620Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9661686Z res = mod(**inputs) 2025-12-04T08:54:30.9661958Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T08:54:30.9662054Z decoder_outputs = self.decoder( 2025-12-04T08:54:30.9662320Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9662402Z layer_outputs = layer_module( 2025-12-04T08:54:30.9662646Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9662747Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9663017Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9663089Z return func(*args, **kwargs) 2025-12-04T08:54:30.9663377Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 589, in forward 2025-12-04T08:54:30.9663463Z cross_attention_outputs = self.layer[1]( 2025-12-04T08:54:30.9663727Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9663808Z return func(*args, **kwargs) 2025-12-04T08:54:30.9664070Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 517, in forward 2025-12-04T08:54:30.9664161Z attention_output = self.EncDecAttention( 2025-12-04T08:54:30.9664456Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9664530Z return func(*args, **kwargs) 2025-12-04T08:54:30.9664800Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 388, in forward 2025-12-04T08:54:30.9664885Z key_states = self.k(current_states) 2025-12-04T08:54:30.9664889Z 2025-12-04T08:54:30.9665001Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9665222Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9665294Z res = mod(**inputs) 2025-12-04T08:54:30.9665565Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T08:54:30.9665643Z decoder_outputs = self.decoder( 2025-12-04T08:54:30.9665908Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9665995Z layer_outputs = layer_module( 2025-12-04T08:54:30.9666238Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9666319Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9666590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9666663Z return func(*args, **kwargs) 2025-12-04T08:54:30.9666934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 589, in forward 2025-12-04T08:54:30.9667021Z cross_attention_outputs = self.layer[1]( 2025-12-04T08:54:30.9667281Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9667363Z return func(*args, **kwargs) 2025-12-04T08:54:30.9667628Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 517, in forward 2025-12-04T08:54:30.9667721Z attention_output = self.EncDecAttention( 2025-12-04T08:54:30.9667992Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9668065Z return func(*args, **kwargs) 2025-12-04T08:54:30.9668334Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 389, in forward 2025-12-04T08:54:30.9668417Z value_states = self.v(current_states) 2025-12-04T08:54:30.9668420Z 2025-12-04T08:54:30.9668526Z cudagraph partition due to non gpu ops 2025-12-04T08:54:30.9668621Z cudagraph partition due to non gpu ops 2025-12-04T08:54:30.9668733Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9668948Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9669025Z res = mod(**inputs) 2025-12-04T08:54:30.9669310Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T08:54:30.9669396Z decoder_outputs = self.decoder( 2025-12-04T08:54:30.9669662Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9669757Z layer_outputs = layer_module( 2025-12-04T08:54:30.9670007Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9670092Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9670362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9670435Z return func(*args, **kwargs) 2025-12-04T08:54:30.9670695Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 589, in forward 2025-12-04T08:54:30.9670811Z cross_attention_outputs = self.layer[1]( 2025-12-04T08:54:30.9671085Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9671157Z return func(*args, **kwargs) 2025-12-04T08:54:30.9671430Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 517, in forward 2025-12-04T08:54:30.9671519Z attention_output = self.EncDecAttention( 2025-12-04T08:54:30.9671789Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9671861Z return func(*args, **kwargs) 2025-12-04T08:54:30.9672122Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 447, in forward 2025-12-04T08:54:30.9672212Z attn_output = self.o(attn_output) 2025-12-04T08:54:30.9672218Z 2025-12-04T08:54:30.9672330Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9672547Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9672624Z res = mod(**inputs) 2025-12-04T08:54:30.9672892Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T08:54:30.9672980Z decoder_outputs = self.decoder( 2025-12-04T08:54:30.9673242Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9673320Z layer_outputs = layer_module( 2025-12-04T08:54:30.9673570Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9673652Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9673922Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9673999Z return func(*args, **kwargs) 2025-12-04T08:54:30.9674263Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T08:54:30.9674368Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T08:54:30.9674630Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T08:54:30.9674754Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T08:54:30.9675041Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 184, in forward 2025-12-04T08:54:30.9675150Z hidden_gelu = self.act(self.wi_0(hidden_states)) 2025-12-04T08:54:30.9675153Z 2025-12-04T08:54:30.9675270Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9675487Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9675557Z res = mod(**inputs) 2025-12-04T08:54:30.9675849Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T08:54:30.9675930Z decoder_outputs = self.decoder( 2025-12-04T08:54:30.9676228Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9676305Z layer_outputs = layer_module( 2025-12-04T08:54:30.9676562Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9676655Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9676927Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9676999Z return func(*args, **kwargs) 2025-12-04T08:54:30.9677293Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T08:54:30.9677391Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T08:54:30.9677659Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T08:54:30.9677783Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T08:54:30.9678048Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 185, in forward 2025-12-04T08:54:30.9678142Z hidden_linear = self.wi_1(hidden_states) 2025-12-04T08:54:30.9678145Z 2025-12-04T08:54:30.9678335Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9678567Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9678637Z res = mod(**inputs) 2025-12-04T08:54:30.9678909Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T08:54:30.9679004Z decoder_outputs = self.decoder( 2025-12-04T08:54:30.9679277Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9679356Z layer_outputs = layer_module( 2025-12-04T08:54:30.9679623Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9679707Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9679981Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9680054Z return func(*args, **kwargs) 2025-12-04T08:54:30.9680318Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T08:54:30.9680420Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T08:54:30.9680687Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T08:54:30.9680809Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T08:54:30.9681080Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 186, in forward 2025-12-04T08:54:30.9681174Z hidden_states = hidden_gelu * hidden_linear 2025-12-04T08:54:30.9681178Z 2025-12-04T08:54:30.9681289Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9681511Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9681579Z res = mod(**inputs) 2025-12-04T08:54:30.9681840Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T08:54:30.9681914Z decoder_outputs = self.decoder( 2025-12-04T08:54:30.9682177Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9682265Z layer_outputs = layer_module( 2025-12-04T08:54:30.9682496Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9682617Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9682865Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9682935Z return func(*args, **kwargs) 2025-12-04T08:54:30.9683190Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T08:54:30.9683279Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T08:54:30.9683531Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T08:54:30.9683662Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T08:54:30.9683910Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 199, in forward 2025-12-04T08:54:30.9684000Z hidden_states = self.wo(hidden_states) 2025-12-04T08:54:30.9684003Z 2025-12-04T08:54:30.9684110Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9684323Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9684388Z res = mod(**inputs) 2025-12-04T08:54:30.9684641Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T08:54:30.9684722Z decoder_outputs = self.decoder( 2025-12-04T08:54:30.9684975Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9685046Z layer_outputs = layer_module( 2025-12-04T08:54:30.9685293Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9685373Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9685636Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9685713Z return func(*args, **kwargs) 2025-12-04T08:54:30.9685975Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T08:54:30.9686069Z self_attention_outputs = self.layer[0]( 2025-12-04T08:54:30.9686334Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9686407Z return func(*args, **kwargs) 2025-12-04T08:54:30.9686678Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T08:54:30.9686768Z attention_output = self.SelfAttention( 2025-12-04T08:54:30.9687040Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9687112Z return func(*args, **kwargs) 2025-12-04T08:54:30.9687373Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 367, in forward 2025-12-04T08:54:30.9687466Z query_states = self.q(hidden_states) 2025-12-04T08:54:30.9687470Z 2025-12-04T08:54:30.9687580Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9687820Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9687892Z res = mod(**inputs) 2025-12-04T08:54:30.9688159Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T08:54:30.9688243Z decoder_outputs = self.decoder( 2025-12-04T08:54:30.9688525Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9688604Z layer_outputs = layer_module( 2025-12-04T08:54:30.9688855Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9688962Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9689234Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9689306Z return func(*args, **kwargs) 2025-12-04T08:54:30.9689572Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T08:54:30.9689668Z self_attention_outputs = self.layer[0]( 2025-12-04T08:54:30.9689929Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9690022Z return func(*args, **kwargs) 2025-12-04T08:54:30.9690305Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T08:54:30.9690393Z attention_output = self.SelfAttention( 2025-12-04T08:54:30.9690673Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9690747Z return func(*args, **kwargs) 2025-12-04T08:54:30.9691018Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 388, in forward 2025-12-04T08:54:30.9691110Z key_states = self.k(current_states) 2025-12-04T08:54:30.9691114Z 2025-12-04T08:54:30.9691224Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9691454Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9691523Z res = mod(**inputs) 2025-12-04T08:54:30.9691810Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T08:54:30.9691895Z decoder_outputs = self.decoder( 2025-12-04T08:54:30.9692169Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9692247Z layer_outputs = layer_module( 2025-12-04T08:54:30.9692494Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9692577Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9692854Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9692927Z return func(*args, **kwargs) 2025-12-04T08:54:30.9693197Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T08:54:30.9693294Z self_attention_outputs = self.layer[0]( 2025-12-04T08:54:30.9693571Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9693642Z return func(*args, **kwargs) 2025-12-04T08:54:30.9693909Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T08:54:30.9693999Z attention_output = self.SelfAttention( 2025-12-04T08:54:30.9694277Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9694368Z return func(*args, **kwargs) 2025-12-04T08:54:30.9694640Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 389, in forward 2025-12-04T08:54:30.9694730Z value_states = self.v(current_states) 2025-12-04T08:54:30.9694734Z 2025-12-04T08:54:30.9694819Z cudagraph partition due to non gpu ops 2025-12-04T08:54:30.9694908Z cudagraph partition due to non gpu ops 2025-12-04T08:54:30.9695043Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9695262Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9695618Z res = mod(**inputs) 2025-12-04T08:54:30.9695913Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T08:54:30.9695994Z decoder_outputs = self.decoder( 2025-12-04T08:54:30.9696280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9696357Z layer_outputs = layer_module( 2025-12-04T08:54:30.9696618Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9696704Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9696996Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9697080Z return func(*args, **kwargs) 2025-12-04T08:54:30.9697361Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T08:54:30.9697451Z self_attention_outputs = self.layer[0]( 2025-12-04T08:54:30.9697731Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9697805Z return func(*args, **kwargs) 2025-12-04T08:54:30.9698075Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T08:54:30.9698164Z attention_output = self.SelfAttention( 2025-12-04T08:54:30.9698433Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9698516Z return func(*args, **kwargs) 2025-12-04T08:54:30.9698793Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 447, in forward 2025-12-04T08:54:30.9698883Z attn_output = self.o(attn_output) 2025-12-04T08:54:30.9698887Z 2025-12-04T08:54:30.9698997Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9699215Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9699290Z res = mod(**inputs) 2025-12-04T08:54:30.9699564Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T08:54:30.9699645Z decoder_outputs = self.decoder( 2025-12-04T08:54:30.9699932Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9700007Z layer_outputs = layer_module( 2025-12-04T08:54:30.9700263Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9700349Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9700612Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9700694Z return func(*args, **kwargs) 2025-12-04T08:54:30.9700961Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 589, in forward 2025-12-04T08:54:30.9701045Z cross_attention_outputs = self.layer[1]( 2025-12-04T08:54:30.9701321Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9701390Z return func(*args, **kwargs) 2025-12-04T08:54:30.9701638Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 517, in forward 2025-12-04T08:54:30.9701725Z attention_output = self.EncDecAttention( 2025-12-04T08:54:30.9701994Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9702072Z return func(*args, **kwargs) 2025-12-04T08:54:30.9702312Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 367, in forward 2025-12-04T08:54:30.9702416Z query_states = self.q(hidden_states) 2025-12-04T08:54:30.9702420Z 2025-12-04T08:54:30.9702521Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9702721Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9702793Z res = mod(**inputs) 2025-12-04T08:54:30.9703042Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T08:54:30.9703115Z decoder_outputs = self.decoder( 2025-12-04T08:54:30.9703387Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9703458Z layer_outputs = layer_module( 2025-12-04T08:54:30.9703690Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9703768Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9704014Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9704090Z return func(*args, **kwargs) 2025-12-04T08:54:30.9704339Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 589, in forward 2025-12-04T08:54:30.9704422Z cross_attention_outputs = self.layer[1]( 2025-12-04T08:54:30.9704678Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9704749Z return func(*args, **kwargs) 2025-12-04T08:54:30.9705005Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 517, in forward 2025-12-04T08:54:30.9705091Z attention_output = self.EncDecAttention( 2025-12-04T08:54:30.9705347Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9705423Z return func(*args, **kwargs) 2025-12-04T08:54:30.9705665Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 388, in forward 2025-12-04T08:54:30.9705747Z key_states = self.k(current_states) 2025-12-04T08:54:30.9705752Z 2025-12-04T08:54:30.9705854Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9706051Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9706122Z res = mod(**inputs) 2025-12-04T08:54:30.9706368Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T08:54:30.9706443Z decoder_outputs = self.decoder( 2025-12-04T08:54:30.9706694Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9706765Z layer_outputs = layer_module( 2025-12-04T08:54:30.9706996Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9707073Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9707331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9707407Z return func(*args, **kwargs) 2025-12-04T08:54:30.9707657Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 589, in forward 2025-12-04T08:54:30.9707740Z cross_attention_outputs = self.layer[1]( 2025-12-04T08:54:30.9708002Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9708071Z return func(*args, **kwargs) 2025-12-04T08:54:30.9708323Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 517, in forward 2025-12-04T08:54:30.9708423Z attention_output = self.EncDecAttention( 2025-12-04T08:54:30.9708674Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9708749Z return func(*args, **kwargs) 2025-12-04T08:54:30.9708993Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 389, in forward 2025-12-04T08:54:30.9709075Z value_states = self.v(current_states) 2025-12-04T08:54:30.9709079Z 2025-12-04T08:54:30.9709156Z cudagraph partition due to non gpu ops 2025-12-04T08:54:30.9709251Z cudagraph partition due to non gpu ops 2025-12-04T08:54:30.9709361Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9709560Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9709622Z res = mod(**inputs) 2025-12-04T08:54:30.9709872Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T08:54:30.9709947Z decoder_outputs = self.decoder( 2025-12-04T08:54:30.9710206Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9710280Z layer_outputs = layer_module( 2025-12-04T08:54:30.9710509Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9710597Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9710847Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9710917Z return func(*args, **kwargs) 2025-12-04T08:54:30.9711172Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 589, in forward 2025-12-04T08:54:30.9711254Z cross_attention_outputs = self.layer[1]( 2025-12-04T08:54:30.9711511Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9711579Z return func(*args, **kwargs) 2025-12-04T08:54:30.9711831Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 517, in forward 2025-12-04T08:54:30.9711936Z attention_output = self.EncDecAttention( 2025-12-04T08:54:30.9712178Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9712247Z return func(*args, **kwargs) 2025-12-04T08:54:30.9712508Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 447, in forward 2025-12-04T08:54:30.9712585Z attn_output = self.o(attn_output) 2025-12-04T08:54:30.9712589Z 2025-12-04T08:54:30.9712697Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9712899Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9712964Z res = mod(**inputs) 2025-12-04T08:54:30.9713218Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T08:54:30.9713310Z decoder_outputs = self.decoder( 2025-12-04T08:54:30.9713564Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9713635Z layer_outputs = layer_module( 2025-12-04T08:54:30.9713861Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9713962Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9714207Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9714274Z return func(*args, **kwargs) 2025-12-04T08:54:30.9714539Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 589, in forward 2025-12-04T08:54:30.9714620Z cross_attention_outputs = self.layer[1]( 2025-12-04T08:54:30.9714870Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9714939Z return func(*args, **kwargs) 2025-12-04T08:54:30.9715179Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 529, in forward 2025-12-04T08:54:30.9715336Z layer_output = hidden_states + self.dropout(attention_output[0]) 2025-12-04T08:54:30.9715339Z 2025-12-04T08:54:30.9715442Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9715647Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9715711Z res = mod(**inputs) 2025-12-04T08:54:30.9715962Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T08:54:30.9716040Z decoder_outputs = self.decoder( 2025-12-04T08:54:30.9716286Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9716357Z layer_outputs = layer_module( 2025-12-04T08:54:30.9716588Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9716666Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9716932Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9717002Z return func(*args, **kwargs) 2025-12-04T08:54:30.9717252Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T08:54:30.9717357Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T08:54:30.9717604Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T08:54:30.9717722Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T08:54:30.9717980Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 184, in forward 2025-12-04T08:54:30.9718080Z hidden_gelu = self.act(self.wi_0(hidden_states)) 2025-12-04T08:54:30.9718083Z 2025-12-04T08:54:30.9718196Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9718467Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9718542Z res = mod(**inputs) 2025-12-04T08:54:30.9718826Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T08:54:30.9718912Z decoder_outputs = self.decoder( 2025-12-04T08:54:30.9719198Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9719281Z layer_outputs = layer_module( 2025-12-04T08:54:30.9719565Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9719665Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9719946Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9720022Z return func(*args, **kwargs) 2025-12-04T08:54:30.9720311Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T08:54:30.9720411Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T08:54:30.9721162Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T08:54:30.9721437Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T08:54:30.9721692Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 185, in forward 2025-12-04T08:54:30.9721783Z hidden_linear = self.wi_1(hidden_states) 2025-12-04T08:54:30.9721788Z 2025-12-04T08:54:30.9721898Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9722123Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9722193Z res = mod(**inputs) 2025-12-04T08:54:30.9722495Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T08:54:30.9722583Z decoder_outputs = self.decoder( 2025-12-04T08:54:30.9722848Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9722927Z layer_outputs = layer_module( 2025-12-04T08:54:30.9723180Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9723264Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9723540Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9723615Z return func(*args, **kwargs) 2025-12-04T08:54:30.9723877Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T08:54:30.9723981Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T08:54:30.9724242Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T08:54:30.9724362Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T08:54:30.9724631Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 186, in forward 2025-12-04T08:54:30.9724727Z hidden_states = hidden_gelu * hidden_linear 2025-12-04T08:54:30.9724731Z 2025-12-04T08:54:30.9724847Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9725067Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9725132Z res = mod(**inputs) 2025-12-04T08:54:30.9725390Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T08:54:30.9725466Z decoder_outputs = self.decoder( 2025-12-04T08:54:30.9725724Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9725799Z layer_outputs = layer_module( 2025-12-04T08:54:30.9726027Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9726116Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9726378Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9726452Z return func(*args, **kwargs) 2025-12-04T08:54:30.9726757Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T08:54:30.9726865Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T08:54:30.9727121Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T08:54:30.9727240Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T08:54:30.9727515Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 199, in forward 2025-12-04T08:54:30.9727611Z hidden_states = self.wo(hidden_states) 2025-12-04T08:54:30.9727631Z 2025-12-04T08:54:30.9727746Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9727976Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9728050Z res = mod(**inputs) 2025-12-04T08:54:30.9728319Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T08:54:30.9728410Z decoder_outputs = self.decoder( 2025-12-04T08:54:30.9728680Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9728776Z layer_outputs = layer_module( 2025-12-04T08:54:30.9729031Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9729116Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9729388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9729466Z return func(*args, **kwargs) 2025-12-04T08:54:30.9729730Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T08:54:30.9729828Z self_attention_outputs = self.layer[0]( 2025-12-04T08:54:30.9730096Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9730169Z return func(*args, **kwargs) 2025-12-04T08:54:30.9730440Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T08:54:30.9730776Z attention_output = self.SelfAttention( 2025-12-04T08:54:30.9731042Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9731114Z return func(*args, **kwargs) 2025-12-04T08:54:30.9731366Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 367, in forward 2025-12-04T08:54:30.9731455Z query_states = self.q(hidden_states) 2025-12-04T08:54:30.9731459Z 2025-12-04T08:54:30.9731566Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9731780Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9731847Z res = mod(**inputs) 2025-12-04T08:54:30.9732099Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T08:54:30.9732185Z decoder_outputs = self.decoder( 2025-12-04T08:54:30.9732445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9732524Z layer_outputs = layer_module( 2025-12-04T08:54:30.9732780Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9732866Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9733140Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9733217Z return func(*args, **kwargs) 2025-12-04T08:54:30.9733503Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T08:54:30.9733603Z self_attention_outputs = self.layer[0]( 2025-12-04T08:54:30.9733870Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9733946Z return func(*args, **kwargs) 2025-12-04T08:54:30.9734236Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T08:54:30.9734330Z attention_output = self.SelfAttention( 2025-12-04T08:54:30.9734629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9734704Z return func(*args, **kwargs) 2025-12-04T08:54:30.9734978Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 388, in forward 2025-12-04T08:54:30.9735073Z key_states = self.k(current_states) 2025-12-04T08:54:30.9735077Z 2025-12-04T08:54:30.9735192Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9735421Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9735512Z res = mod(**inputs) 2025-12-04T08:54:30.9735792Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T08:54:30.9735878Z decoder_outputs = self.decoder( 2025-12-04T08:54:30.9736142Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9736221Z layer_outputs = layer_module( 2025-12-04T08:54:30.9736470Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9736555Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9736825Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9736900Z return func(*args, **kwargs) 2025-12-04T08:54:30.9737160Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T08:54:30.9737257Z self_attention_outputs = self.layer[0]( 2025-12-04T08:54:30.9737525Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9737598Z return func(*args, **kwargs) 2025-12-04T08:54:30.9737867Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T08:54:30.9737956Z attention_output = self.SelfAttention( 2025-12-04T08:54:30.9738225Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9738299Z return func(*args, **kwargs) 2025-12-04T08:54:30.9738561Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 389, in forward 2025-12-04T08:54:30.9738651Z value_states = self.v(current_states) 2025-12-04T08:54:30.9738656Z 2025-12-04T08:54:30.9738743Z cudagraph partition due to non gpu ops 2025-12-04T08:54:30.9738834Z cudagraph partition due to non gpu ops 2025-12-04T08:54:30.9738947Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9739162Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9739240Z res = mod(**inputs) 2025-12-04T08:54:30.9739504Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T08:54:30.9739581Z decoder_outputs = self.decoder( 2025-12-04T08:54:30.9739875Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9739954Z layer_outputs = layer_module( 2025-12-04T08:54:30.9740205Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9740292Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9740573Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9740654Z return func(*args, **kwargs) 2025-12-04T08:54:30.9740920Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T08:54:30.9741028Z self_attention_outputs = self.layer[0]( 2025-12-04T08:54:30.9741298Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9741370Z return func(*args, **kwargs) 2025-12-04T08:54:30.9741638Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T08:54:30.9741727Z attention_output = self.SelfAttention( 2025-12-04T08:54:30.9741987Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9742087Z return func(*args, **kwargs) 2025-12-04T08:54:30.9742350Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 447, in forward 2025-12-04T08:54:30.9742438Z attn_output = self.o(attn_output) 2025-12-04T08:54:30.9742442Z 2025-12-04T08:54:30.9742558Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9742775Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9742851Z res = mod(**inputs) 2025-12-04T08:54:30.9743121Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T08:54:30.9743200Z decoder_outputs = self.decoder( 2025-12-04T08:54:30.9743478Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9743556Z layer_outputs = layer_module( 2025-12-04T08:54:30.9743807Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9743894Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9744157Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9744242Z return func(*args, **kwargs) 2025-12-04T08:54:30.9744505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 589, in forward 2025-12-04T08:54:30.9744594Z cross_attention_outputs = self.layer[1]( 2025-12-04T08:54:30.9744868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9744942Z return func(*args, **kwargs) 2025-12-04T08:54:30.9745213Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 517, in forward 2025-12-04T08:54:30.9745309Z attention_output = self.EncDecAttention( 2025-12-04T08:54:30.9745574Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9745657Z return func(*args, **kwargs) 2025-12-04T08:54:30.9745919Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 367, in forward 2025-12-04T08:54:30.9746011Z query_states = self.q(hidden_states) 2025-12-04T08:54:30.9746015Z 2025-12-04T08:54:30.9746129Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9746365Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9746443Z res = mod(**inputs) 2025-12-04T08:54:30.9746709Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T08:54:30.9746790Z decoder_outputs = self.decoder( 2025-12-04T08:54:30.9747079Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9747157Z layer_outputs = layer_module( 2025-12-04T08:54:30.9747404Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9747505Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9747769Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9747848Z return func(*args, **kwargs) 2025-12-04T08:54:30.9748112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 589, in forward 2025-12-04T08:54:30.9748200Z cross_attention_outputs = self.layer[1]( 2025-12-04T08:54:30.9748471Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9748572Z return func(*args, **kwargs) 2025-12-04T08:54:30.9748845Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 517, in forward 2025-12-04T08:54:30.9748936Z attention_output = self.EncDecAttention( 2025-12-04T08:54:30.9749203Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9749285Z return func(*args, **kwargs) 2025-12-04T08:54:30.9749549Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 388, in forward 2025-12-04T08:54:30.9749638Z key_states = self.k(current_states) 2025-12-04T08:54:30.9749642Z 2025-12-04T08:54:30.9749754Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9749968Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9750046Z res = mod(**inputs) 2025-12-04T08:54:30.9750314Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T08:54:30.9750392Z decoder_outputs = self.decoder( 2025-12-04T08:54:30.9750665Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9750743Z layer_outputs = layer_module( 2025-12-04T08:54:30.9751003Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9751082Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9751330Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9751408Z return func(*args, **kwargs) 2025-12-04T08:54:30.9751661Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 589, in forward 2025-12-04T08:54:30.9751750Z cross_attention_outputs = self.layer[1]( 2025-12-04T08:54:30.9752020Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9752093Z return func(*args, **kwargs) 2025-12-04T08:54:30.9752368Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 517, in forward 2025-12-04T08:54:30.9752461Z attention_output = self.EncDecAttention( 2025-12-04T08:54:30.9752729Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9752828Z return func(*args, **kwargs) 2025-12-04T08:54:30.9753093Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 389, in forward 2025-12-04T08:54:30.9753179Z value_states = self.v(current_states) 2025-12-04T08:54:30.9753183Z 2025-12-04T08:54:30.9753266Z cudagraph partition due to non gpu ops 2025-12-04T08:54:30.9753348Z cudagraph partition due to non gpu ops 2025-12-04T08:54:30.9753480Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9753691Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9753756Z res = mod(**inputs) 2025-12-04T08:54:30.9754060Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T08:54:30.9754137Z decoder_outputs = self.decoder( 2025-12-04T08:54:30.9754421Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9754498Z layer_outputs = layer_module( 2025-12-04T08:54:30.9754740Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9754829Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9755123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9755198Z return func(*args, **kwargs) 2025-12-04T08:54:30.9755480Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 589, in forward 2025-12-04T08:54:30.9755568Z cross_attention_outputs = self.layer[1]( 2025-12-04T08:54:30.9755845Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9755918Z return func(*args, **kwargs) 2025-12-04T08:54:30.9756191Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 517, in forward 2025-12-04T08:54:30.9756287Z attention_output = self.EncDecAttention( 2025-12-04T08:54:30.9756569Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9756642Z return func(*args, **kwargs) 2025-12-04T08:54:30.9756921Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 447, in forward 2025-12-04T08:54:30.9757003Z attn_output = self.o(attn_output) 2025-12-04T08:54:30.9757007Z 2025-12-04T08:54:30.9757124Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9757341Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9757411Z res = mod(**inputs) 2025-12-04T08:54:30.9757695Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T08:54:30.9757773Z decoder_outputs = self.decoder( 2025-12-04T08:54:30.9758055Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9758129Z layer_outputs = layer_module( 2025-12-04T08:54:30.9758460Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9758560Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9758833Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9758909Z return func(*args, **kwargs) 2025-12-04T08:54:30.9759188Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T08:54:30.9759290Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T08:54:30.9759606Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T08:54:30.9759739Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T08:54:30.9759988Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 184, in forward 2025-12-04T08:54:30.9760102Z hidden_gelu = self.act(self.wi_0(hidden_states)) 2025-12-04T08:54:30.9760120Z 2025-12-04T08:54:30.9760229Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9760443Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9760530Z res = mod(**inputs) 2025-12-04T08:54:30.9760784Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T08:54:30.9760867Z decoder_outputs = self.decoder( 2025-12-04T08:54:30.9761123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9761197Z layer_outputs = layer_module( 2025-12-04T08:54:30.9761434Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9761532Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9761803Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9761874Z return func(*args, **kwargs) 2025-12-04T08:54:30.9762124Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T08:54:30.9762229Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T08:54:30.9762500Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T08:54:30.9762626Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T08:54:30.9762899Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 185, in forward 2025-12-04T08:54:30.9762985Z hidden_linear = self.wi_1(hidden_states) 2025-12-04T08:54:30.9762989Z 2025-12-04T08:54:30.9763108Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9763327Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9763398Z res = mod(**inputs) 2025-12-04T08:54:30.9763669Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T08:54:30.9763748Z decoder_outputs = self.decoder( 2025-12-04T08:54:30.9764027Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9764103Z layer_outputs = layer_module( 2025-12-04T08:54:30.9764355Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9764442Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9764692Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9764764Z return func(*args, **kwargs) 2025-12-04T08:54:30.9765021Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T08:54:30.9765112Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T08:54:30.9765366Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T08:54:30.9765482Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T08:54:30.9765728Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 186, in forward 2025-12-04T08:54:30.9765844Z hidden_states = hidden_gelu * hidden_linear 2025-12-04T08:54:30.9765848Z 2025-12-04T08:54:30.9765955Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9766164Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9766232Z res = mod(**inputs) 2025-12-04T08:54:30.9766497Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T08:54:30.9766578Z decoder_outputs = self.decoder( 2025-12-04T08:54:30.9766828Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9766918Z layer_outputs = layer_module( 2025-12-04T08:54:30.9767162Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9767242Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9767499Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9767570Z return func(*args, **kwargs) 2025-12-04T08:54:30.9767820Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T08:54:30.9767935Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T08:54:30.9768184Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T08:54:30.9768299Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T08:54:30.9768556Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 199, in forward 2025-12-04T08:54:30.9768638Z hidden_states = self.wo(hidden_states) 2025-12-04T08:54:30.9768642Z 2025-12-04T08:54:30.9768754Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9768961Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9769027Z res = mod(**inputs) 2025-12-04T08:54:30.9769285Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T08:54:30.9769360Z decoder_outputs = self.decoder( 2025-12-04T08:54:30.9769619Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9769693Z layer_outputs = layer_module( 2025-12-04T08:54:30.9769921Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9770009Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9770261Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9770332Z return func(*args, **kwargs) 2025-12-04T08:54:30.9770586Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T08:54:30.9770674Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T08:54:30.9770927Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 218, in forward 2025-12-04T08:54:30.9771061Z hidden_states = hidden_states + self.dropout(forwarded_states) 2025-12-04T08:54:30.9771065Z 2025-12-04T08:54:30.9771171Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9771379Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9771445Z res = mod(**inputs) 2025-12-04T08:54:30.9771716Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T08:54:30.9771793Z decoder_outputs = self.decoder( 2025-12-04T08:54:30.9772077Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9772162Z layer_outputs = layer_module( 2025-12-04T08:54:30.9772402Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9772489Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9772777Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9772853Z return func(*args, **kwargs) 2025-12-04T08:54:30.9773117Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T08:54:30.9773231Z self_attention_outputs = self.layer[0]( 2025-12-04T08:54:30.9773499Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9773579Z return func(*args, **kwargs) 2025-12-04T08:54:30.9773826Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T08:54:30.9773910Z attention_output = self.SelfAttention( 2025-12-04T08:54:30.9774179Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9774251Z return func(*args, **kwargs) 2025-12-04T08:54:30.9774505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 367, in forward 2025-12-04T08:54:30.9774590Z query_states = self.q(hidden_states) 2025-12-04T08:54:30.9774594Z 2025-12-04T08:54:30.9774704Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9774928Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9774997Z res = mod(**inputs) 2025-12-04T08:54:30.9775269Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T08:54:30.9775348Z decoder_outputs = self.decoder( 2025-12-04T08:54:30.9775611Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9775696Z layer_outputs = layer_module( 2025-12-04T08:54:30.9775938Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9776023Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9776292Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9776368Z return func(*args, **kwargs) 2025-12-04T08:54:30.9776638Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T08:54:30.9776726Z self_attention_outputs = self.layer[0]( 2025-12-04T08:54:30.9776987Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9777069Z return func(*args, **kwargs) 2025-12-04T08:54:30.9777331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T08:54:30.9777422Z attention_output = self.SelfAttention( 2025-12-04T08:54:30.9777693Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9777766Z return func(*args, **kwargs) 2025-12-04T08:54:30.9778034Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 388, in forward 2025-12-04T08:54:30.9778117Z key_states = self.k(current_states) 2025-12-04T08:54:30.9778121Z 2025-12-04T08:54:30.9778257Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9778484Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9778551Z res = mod(**inputs) 2025-12-04T08:54:30.9778822Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T08:54:30.9778901Z decoder_outputs = self.decoder( 2025-12-04T08:54:30.9779188Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9779274Z layer_outputs = layer_module( 2025-12-04T08:54:30.9779516Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9779619Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9779890Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9779964Z return func(*args, **kwargs) 2025-12-04T08:54:30.9780236Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T08:54:30.9780323Z self_attention_outputs = self.layer[0]( 2025-12-04T08:54:30.9780585Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9780684Z return func(*args, **kwargs) 2025-12-04T08:54:30.9780950Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T08:54:30.9781037Z attention_output = self.SelfAttention( 2025-12-04T08:54:30.9781314Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9781387Z return func(*args, **kwargs) 2025-12-04T08:54:30.9781662Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 389, in forward 2025-12-04T08:54:30.9781745Z value_states = self.v(current_states) 2025-12-04T08:54:30.9781748Z 2025-12-04T08:54:30.9781835Z cudagraph partition due to non gpu ops 2025-12-04T08:54:30.9781928Z cudagraph partition due to non gpu ops 2025-12-04T08:54:30.9782042Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9782270Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9782339Z res = mod(**inputs) 2025-12-04T08:54:30.9782607Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T08:54:30.9782695Z decoder_outputs = self.decoder( 2025-12-04T08:54:30.9782963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9783048Z layer_outputs = layer_module( 2025-12-04T08:54:30.9783288Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9783368Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9783623Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9783692Z return func(*args, **kwargs) 2025-12-04T08:54:30.9783946Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T08:54:30.9784037Z self_attention_outputs = self.layer[0]( 2025-12-04T08:54:30.9784286Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9784357Z return func(*args, **kwargs) 2025-12-04T08:54:30.9784614Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T08:54:30.9784713Z attention_output = self.SelfAttention( 2025-12-04T08:54:30.9784968Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9785036Z return func(*args, **kwargs) 2025-12-04T08:54:30.9785282Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 447, in forward 2025-12-04T08:54:30.9785383Z attn_output = self.o(attn_output) 2025-12-04T08:54:30.9785388Z 2025-12-04T08:54:30.9785493Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9785704Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9785787Z res = mod(**inputs) 2025-12-04T08:54:30.9786048Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T08:54:30.9786127Z decoder_outputs = self.decoder( 2025-12-04T08:54:30.9786378Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9786451Z layer_outputs = layer_module( 2025-12-04T08:54:30.9786688Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9786786Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9787038Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9787105Z return func(*args, **kwargs) 2025-12-04T08:54:30.9787353Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 589, in forward 2025-12-04T08:54:30.9787445Z cross_attention_outputs = self.layer[1]( 2025-12-04T08:54:30.9787693Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9787762Z return func(*args, **kwargs) 2025-12-04T08:54:30.9788016Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 517, in forward 2025-12-04T08:54:30.9788103Z attention_output = self.EncDecAttention( 2025-12-04T08:54:30.9788367Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9788442Z return func(*args, **kwargs) 2025-12-04T08:54:30.9788703Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 367, in forward 2025-12-04T08:54:30.9788793Z query_states = self.q(hidden_states) 2025-12-04T08:54:30.9788799Z 2025-12-04T08:54:30.9788908Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9789130Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9789200Z res = mod(**inputs) 2025-12-04T08:54:30.9789464Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T08:54:30.9789551Z decoder_outputs = self.decoder( 2025-12-04T08:54:30.9789815Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9789892Z layer_outputs = layer_module( 2025-12-04T08:54:30.9790143Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9790227Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9790495Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9790570Z return func(*args, **kwargs) 2025-12-04T08:54:30.9790829Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 589, in forward 2025-12-04T08:54:30.9790943Z cross_attention_outputs = self.layer[1]( 2025-12-04T08:54:30.9791208Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9791281Z return func(*args, **kwargs) 2025-12-04T08:54:30.9791549Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 517, in forward 2025-12-04T08:54:30.9791661Z attention_output = self.EncDecAttention( 2025-12-04T08:54:30.9791933Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9792007Z return func(*args, **kwargs) 2025-12-04T08:54:30.9792290Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 388, in forward 2025-12-04T08:54:30.9792380Z key_states = self.k(current_states) 2025-12-04T08:54:30.9792384Z 2025-12-04T08:54:30.9792494Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9792721Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9792791Z res = mod(**inputs) 2025-12-04T08:54:30.9793053Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T08:54:30.9793162Z decoder_outputs = self.decoder( 2025-12-04T08:54:30.9793429Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9793505Z layer_outputs = layer_module( 2025-12-04T08:54:30.9793758Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9793841Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9794111Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9794184Z return func(*args, **kwargs) 2025-12-04T08:54:30.9794460Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 589, in forward 2025-12-04T08:54:30.9794555Z cross_attention_outputs = self.layer[1]( 2025-12-04T08:54:30.9794825Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9794900Z return func(*args, **kwargs) 2025-12-04T08:54:30.9795180Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 517, in forward 2025-12-04T08:54:30.9795270Z attention_output = self.EncDecAttention( 2025-12-04T08:54:30.9795556Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9795628Z return func(*args, **kwargs) 2025-12-04T08:54:30.9795898Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 389, in forward 2025-12-04T08:54:30.9795989Z value_states = self.v(current_states) 2025-12-04T08:54:30.9795992Z 2025-12-04T08:54:30.9796078Z cudagraph partition due to non gpu ops 2025-12-04T08:54:30.9796170Z cudagraph partition due to non gpu ops 2025-12-04T08:54:30.9796280Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9796497Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9796572Z res = mod(**inputs) 2025-12-04T08:54:30.9796846Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T08:54:30.9796927Z decoder_outputs = self.decoder( 2025-12-04T08:54:30.9797213Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9797289Z layer_outputs = layer_module( 2025-12-04T08:54:30.9797555Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9797640Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9797907Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9797990Z return func(*args, **kwargs) 2025-12-04T08:54:30.9798362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 589, in forward 2025-12-04T08:54:30.9798459Z cross_attention_outputs = self.layer[1]( 2025-12-04T08:54:30.9798750Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9798845Z return func(*args, **kwargs) 2025-12-04T08:54:30.9799139Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 517, in forward 2025-12-04T08:54:30.9799234Z attention_output = self.EncDecAttention( 2025-12-04T08:54:30.9799524Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9799606Z return func(*args, **kwargs) 2025-12-04T08:54:30.9799892Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 447, in forward 2025-12-04T08:54:30.9800491Z attn_output = self.o(attn_output) 2025-12-04T08:54:30.9800504Z 2025-12-04T08:54:30.9800611Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9800837Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9800915Z res = mod(**inputs) 2025-12-04T08:54:30.9801235Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T08:54:30.9801318Z decoder_outputs = self.decoder( 2025-12-04T08:54:30.9801612Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9801689Z layer_outputs = layer_module( 2025-12-04T08:54:30.9801945Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9802032Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9802320Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9802403Z return func(*args, **kwargs) 2025-12-04T08:54:30.9802678Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T08:54:30.9802780Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T08:54:30.9803064Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T08:54:30.9803194Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T08:54:30.9803479Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 184, in forward 2025-12-04T08:54:30.9803585Z hidden_gelu = self.act(self.wi_0(hidden_states)) 2025-12-04T08:54:30.9803591Z 2025-12-04T08:54:30.9803704Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9803944Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9804014Z res = mod(**inputs) 2025-12-04T08:54:30.9804300Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T08:54:30.9804382Z decoder_outputs = self.decoder( 2025-12-04T08:54:30.9804655Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9804741Z layer_outputs = layer_module( 2025-12-04T08:54:30.9805012Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9805100Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9805376Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9805455Z return func(*args, **kwargs) 2025-12-04T08:54:30.9805753Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T08:54:30.9805855Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T08:54:30.9806122Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T08:54:30.9806266Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T08:54:30.9806515Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 185, in forward 2025-12-04T08:54:30.9806594Z hidden_linear = self.wi_1(hidden_states) 2025-12-04T08:54:30.9806605Z 2025-12-04T08:54:30.9806711Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9806913Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9807006Z res = mod(**inputs) 2025-12-04T08:54:30.9807263Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T08:54:30.9807336Z decoder_outputs = self.decoder( 2025-12-04T08:54:30.9807600Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9807674Z layer_outputs = layer_module( 2025-12-04T08:54:30.9807917Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9807998Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9808251Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9808330Z return func(*args, **kwargs) 2025-12-04T08:54:30.9808583Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T08:54:30.9808678Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T08:54:30.9808938Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T08:54:30.9809052Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T08:54:30.9809310Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 186, in forward 2025-12-04T08:54:30.9809401Z hidden_states = hidden_gelu * hidden_linear 2025-12-04T08:54:30.9809405Z 2025-12-04T08:54:30.9809511Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9809724Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9809790Z res = mod(**inputs) 2025-12-04T08:54:30.9810052Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T08:54:30.9810129Z decoder_outputs = self.decoder( 2025-12-04T08:54:30.9810385Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9810464Z layer_outputs = layer_module( 2025-12-04T08:54:30.9810699Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9810780Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9811038Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9811133Z return func(*args, **kwargs) 2025-12-04T08:54:30.9811389Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T08:54:30.9811478Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T08:54:30.9811728Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T08:54:30.9811869Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T08:54:30.9812116Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 199, in forward 2025-12-04T08:54:30.9812217Z hidden_states = self.wo(hidden_states) 2025-12-04T08:54:30.9812229Z 2025-12-04T08:54:30.9812335Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9812540Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9812616Z res = mod(**inputs) 2025-12-04T08:54:30.9812866Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T08:54:30.9812940Z decoder_outputs = self.decoder( 2025-12-04T08:54:30.9813196Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9813288Z layer_outputs = layer_module( 2025-12-04T08:54:30.9813524Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9813603Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9813863Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9813942Z return func(*args, **kwargs) 2025-12-04T08:54:30.9814192Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T08:54:30.9814277Z self_attention_outputs = self.layer[0]( 2025-12-04T08:54:30.9814537Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9814605Z return func(*args, **kwargs) 2025-12-04T08:54:30.9814863Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T08:54:30.9814951Z attention_output = self.SelfAttention( 2025-12-04T08:54:30.9815200Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9815281Z return func(*args, **kwargs) 2025-12-04T08:54:30.9815526Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 367, in forward 2025-12-04T08:54:30.9815605Z query_states = self.q(hidden_states) 2025-12-04T08:54:30.9815616Z 2025-12-04T08:54:30.9815723Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9815930Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9816002Z res = mod(**inputs) 2025-12-04T08:54:30.9816255Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T08:54:30.9816329Z decoder_outputs = self.decoder( 2025-12-04T08:54:30.9816588Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9816662Z layer_outputs = layer_module( 2025-12-04T08:54:30.9816901Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9816981Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9817227Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9817319Z return func(*args, **kwargs) 2025-12-04T08:54:30.9817565Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T08:54:30.9817648Z self_attention_outputs = self.layer[0]( 2025-12-04T08:54:30.9817904Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9817987Z return func(*args, **kwargs) 2025-12-04T08:54:30.9818246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T08:54:30.9818348Z attention_output = self.SelfAttention( 2025-12-04T08:54:30.9818596Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9818673Z return func(*args, **kwargs) 2025-12-04T08:54:30.9818920Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 388, in forward 2025-12-04T08:54:30.9819004Z key_states = self.k(current_states) 2025-12-04T08:54:30.9819008Z 2025-12-04T08:54:30.9819113Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9819316Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9819407Z res = mod(**inputs) 2025-12-04T08:54:30.9819673Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T08:54:30.9819746Z decoder_outputs = self.decoder( 2025-12-04T08:54:30.9820015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9820087Z layer_outputs = layer_module( 2025-12-04T08:54:30.9820326Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9820419Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9820659Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9820966Z return func(*args, **kwargs) 2025-12-04T08:54:30.9821238Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T08:54:30.9821322Z self_attention_outputs = self.layer[0]( 2025-12-04T08:54:30.9821566Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9821635Z return func(*args, **kwargs) 2025-12-04T08:54:30.9821875Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T08:54:30.9821955Z attention_output = self.SelfAttention( 2025-12-04T08:54:30.9822192Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9822268Z return func(*args, **kwargs) 2025-12-04T08:54:30.9822500Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 389, in forward 2025-12-04T08:54:30.9822585Z value_states = self.v(current_states) 2025-12-04T08:54:30.9822588Z 2025-12-04T08:54:30.9822668Z cudagraph partition due to non gpu ops 2025-12-04T08:54:30.9822746Z cudagraph partition due to non gpu ops 2025-12-04T08:54:30.9822852Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9823045Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9823110Z res = mod(**inputs) 2025-12-04T08:54:30.9823358Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T08:54:30.9823428Z decoder_outputs = self.decoder( 2025-12-04T08:54:30.9823739Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9823809Z layer_outputs = layer_module( 2025-12-04T08:54:30.9824031Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9824121Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9824387Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9824457Z return func(*args, **kwargs) 2025-12-04T08:54:30.9824708Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T08:54:30.9824811Z self_attention_outputs = self.layer[0]( 2025-12-04T08:54:30.9825056Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9825125Z return func(*args, **kwargs) 2025-12-04T08:54:30.9825368Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T08:54:30.9825460Z attention_output = self.SelfAttention( 2025-12-04T08:54:30.9825734Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9825803Z return func(*args, **kwargs) 2025-12-04T08:54:30.9826057Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 447, in forward 2025-12-04T08:54:30.9826136Z attn_output = self.o(attn_output) 2025-12-04T08:54:30.9826140Z 2025-12-04T08:54:30.9826254Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9826463Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9826524Z res = mod(**inputs) 2025-12-04T08:54:30.9826772Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T08:54:30.9826842Z decoder_outputs = self.decoder( 2025-12-04T08:54:30.9827089Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9827159Z layer_outputs = layer_module( 2025-12-04T08:54:30.9827380Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9827461Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9827695Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9827762Z return func(*args, **kwargs) 2025-12-04T08:54:30.9828006Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T08:54:30.9828088Z self_attention_outputs = self.layer[0]( 2025-12-04T08:54:30.9828337Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9828403Z return func(*args, **kwargs) 2025-12-04T08:54:30.9828643Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 489, in forward 2025-12-04T08:54:30.9828787Z hidden_states = hidden_states + self.dropout(attention_output[0]) 2025-12-04T08:54:30.9828791Z 2025-12-04T08:54:30.9828893Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9829101Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9829167Z res = mod(**inputs) 2025-12-04T08:54:30.9829412Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T08:54:30.9829492Z decoder_outputs = self.decoder( 2025-12-04T08:54:30.9829754Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9829836Z layer_outputs = layer_module( 2025-12-04T08:54:30.9830062Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9830141Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9830402Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9830472Z return func(*args, **kwargs) 2025-12-04T08:54:30.9830728Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 589, in forward 2025-12-04T08:54:30.9830817Z cross_attention_outputs = self.layer[1]( 2025-12-04T08:54:30.9831060Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9831129Z return func(*args, **kwargs) 2025-12-04T08:54:30.9831375Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 517, in forward 2025-12-04T08:54:30.9831460Z attention_output = self.EncDecAttention( 2025-12-04T08:54:30.9831727Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9831794Z return func(*args, **kwargs) 2025-12-04T08:54:30.9832047Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 367, in forward 2025-12-04T08:54:30.9832133Z query_states = self.q(hidden_states) 2025-12-04T08:54:30.9832137Z 2025-12-04T08:54:30.9832237Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9832444Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9832508Z res = mod(**inputs) 2025-12-04T08:54:30.9832764Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T08:54:30.9832841Z decoder_outputs = self.decoder( 2025-12-04T08:54:30.9833079Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9833149Z layer_outputs = layer_module( 2025-12-04T08:54:30.9833375Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9833449Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9833692Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9833759Z return func(*args, **kwargs) 2025-12-04T08:54:30.9833994Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 589, in forward 2025-12-04T08:54:30.9834081Z cross_attention_outputs = self.layer[1]( 2025-12-04T08:54:30.9834314Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9834379Z return func(*args, **kwargs) 2025-12-04T08:54:30.9834627Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 517, in forward 2025-12-04T08:54:30.9834713Z attention_output = self.EncDecAttention( 2025-12-04T08:54:30.9834961Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9835030Z return func(*args, **kwargs) 2025-12-04T08:54:30.9835268Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 388, in forward 2025-12-04T08:54:30.9835351Z key_states = self.k(current_states) 2025-12-04T08:54:30.9835354Z 2025-12-04T08:54:30.9835477Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9835685Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9835749Z res = mod(**inputs) 2025-12-04T08:54:30.9835994Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T08:54:30.9836077Z decoder_outputs = self.decoder( 2025-12-04T08:54:30.9836338Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9836410Z layer_outputs = layer_module( 2025-12-04T08:54:30.9836655Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9836732Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9836984Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9837060Z return func(*args, **kwargs) 2025-12-04T08:54:30.9837310Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 589, in forward 2025-12-04T08:54:30.9837399Z cross_attention_outputs = self.layer[1]( 2025-12-04T08:54:30.9837670Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9837742Z return func(*args, **kwargs) 2025-12-04T08:54:30.9837997Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 517, in forward 2025-12-04T08:54:30.9838088Z attention_output = self.EncDecAttention( 2025-12-04T08:54:30.9838419Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9838499Z return func(*args, **kwargs) 2025-12-04T08:54:30.9838770Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 389, in forward 2025-12-04T08:54:30.9838863Z value_states = self.v(current_states) 2025-12-04T08:54:30.9838867Z 2025-12-04T08:54:30.9838957Z cudagraph partition due to non gpu ops 2025-12-04T08:54:30.9839051Z cudagraph partition due to non gpu ops 2025-12-04T08:54:30.9839169Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9839394Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9839473Z res = mod(**inputs) 2025-12-04T08:54:30.9839754Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T08:54:30.9839835Z decoder_outputs = self.decoder( 2025-12-04T08:54:30.9840108Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9840186Z layer_outputs = layer_module( 2025-12-04T08:54:30.9840439Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9840523Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9840784Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9840871Z return func(*args, **kwargs) 2025-12-04T08:54:30.9841136Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 589, in forward 2025-12-04T08:54:30.9841223Z cross_attention_outputs = self.layer[1]( 2025-12-04T08:54:30.9841496Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9841576Z return func(*args, **kwargs) 2025-12-04T08:54:30.9841833Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 517, in forward 2025-12-04T08:54:30.9841941Z attention_output = self.EncDecAttention( 2025-12-04T08:54:30.9842193Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9842268Z return func(*args, **kwargs) 2025-12-04T08:54:30.9842518Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 447, in forward 2025-12-04T08:54:30.9842615Z attn_output = self.o(attn_output) 2025-12-04T08:54:30.9842625Z 2025-12-04T08:54:30.9842730Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9842941Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9843034Z res = mod(**inputs) 2025-12-04T08:54:30.9843300Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T08:54:30.9843379Z decoder_outputs = self.decoder( 2025-12-04T08:54:30.9843660Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9843734Z layer_outputs = layer_module( 2025-12-04T08:54:30.9843974Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9844078Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9844325Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9844402Z return func(*args, **kwargs) 2025-12-04T08:54:30.9844650Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T08:54:30.9844745Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T08:54:30.9844999Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T08:54:30.9845119Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T08:54:30.9845373Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 184, in forward 2025-12-04T08:54:30.9845472Z hidden_gelu = self.act(self.wi_0(hidden_states)) 2025-12-04T08:54:30.9845477Z 2025-12-04T08:54:30.9845581Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9845793Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9845860Z res = mod(**inputs) 2025-12-04T08:54:30.9846116Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T08:54:30.9846192Z decoder_outputs = self.decoder( 2025-12-04T08:54:30.9846441Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9846523Z layer_outputs = layer_module( 2025-12-04T08:54:30.9846752Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9846830Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9847084Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9847159Z return func(*args, **kwargs) 2025-12-04T08:54:30.9847413Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T08:54:30.9847505Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T08:54:30.9847751Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T08:54:30.9847877Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T08:54:30.9848144Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 185, in forward 2025-12-04T08:54:30.9848226Z hidden_linear = self.wi_1(hidden_states) 2025-12-04T08:54:30.9848237Z 2025-12-04T08:54:30.9848340Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9848541Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9848614Z res = mod(**inputs) 2025-12-04T08:54:30.9848879Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T08:54:30.9848952Z decoder_outputs = self.decoder( 2025-12-04T08:54:30.9849226Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9849297Z layer_outputs = layer_module( 2025-12-04T08:54:30.9849532Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9849611Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9849860Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9849937Z return func(*args, **kwargs) 2025-12-04T08:54:30.9850200Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T08:54:30.9850294Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T08:54:30.9850548Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T08:54:30.9850665Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T08:54:30.9850916Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 186, in forward 2025-12-04T08:54:30.9851007Z hidden_states = hidden_gelu * hidden_linear 2025-12-04T08:54:30.9851010Z 2025-12-04T08:54:30.9851116Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9851328Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9851394Z res = mod(**inputs) 2025-12-04T08:54:30.9851659Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T08:54:30.9851738Z decoder_outputs = self.decoder( 2025-12-04T08:54:30.9851992Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9852070Z layer_outputs = layer_module( 2025-12-04T08:54:30.9852300Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9852381Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9852639Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9852709Z return func(*args, **kwargs) 2025-12-04T08:54:30.9852966Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T08:54:30.9853057Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T08:54:30.9853306Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T08:54:30.9853429Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T08:54:30.9853676Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 199, in forward 2025-12-04T08:54:30.9853759Z hidden_states = self.wo(hidden_states) 2025-12-04T08:54:30.9853771Z 2025-12-04T08:54:30.9853877Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9854082Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9854170Z res = mod(**inputs) 2025-12-04T08:54:30.9854422Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T08:54:30.9854497Z decoder_outputs = self.decoder( 2025-12-04T08:54:30.9854757Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9854873Z layer_outputs = layer_module( 2025-12-04T08:54:30.9855108Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9855186Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9855449Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9855526Z return func(*args, **kwargs) 2025-12-04T08:54:30.9855776Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T08:54:30.9855860Z self_attention_outputs = self.layer[0]( 2025-12-04T08:54:30.9856116Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9856205Z return func(*args, **kwargs) 2025-12-04T08:54:30.9856464Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T08:54:30.9856550Z attention_output = self.SelfAttention( 2025-12-04T08:54:30.9856801Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9856882Z return func(*args, **kwargs) 2025-12-04T08:54:30.9857133Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 367, in forward 2025-12-04T08:54:30.9857213Z query_states = self.q(hidden_states) 2025-12-04T08:54:30.9857225Z 2025-12-04T08:54:30.9857331Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9857537Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9857610Z res = mod(**inputs) 2025-12-04T08:54:30.9857862Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T08:54:30.9857938Z decoder_outputs = self.decoder( 2025-12-04T08:54:30.9858204Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9858278Z layer_outputs = layer_module( 2025-12-04T08:54:30.9858514Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9858595Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9858843Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9858920Z return func(*args, **kwargs) 2025-12-04T08:54:30.9859170Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T08:54:30.9859255Z self_attention_outputs = self.layer[0]( 2025-12-04T08:54:30.9859512Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9859581Z return func(*args, **kwargs) 2025-12-04T08:54:30.9859835Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T08:54:30.9859921Z attention_output = self.SelfAttention( 2025-12-04T08:54:30.9860171Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9860248Z return func(*args, **kwargs) 2025-12-04T08:54:30.9860521Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 388, in forward 2025-12-04T08:54:30.9860600Z key_states = self.k(current_states) 2025-12-04T08:54:30.9860611Z 2025-12-04T08:54:30.9860716Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9860921Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9861010Z res = mod(**inputs) 2025-12-04T08:54:30.9861266Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T08:54:30.9861339Z decoder_outputs = self.decoder( 2025-12-04T08:54:30.9861614Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9861686Z layer_outputs = layer_module( 2025-12-04T08:54:30.9861924Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9862005Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9862254Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9862350Z return func(*args, **kwargs) 2025-12-04T08:54:30.9862602Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T08:54:30.9862684Z self_attention_outputs = self.layer[0]( 2025-12-04T08:54:30.9862966Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9863039Z return func(*args, **kwargs) 2025-12-04T08:54:30.9863314Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T08:54:30.9863402Z attention_output = self.SelfAttention( 2025-12-04T08:54:30.9863678Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9863757Z return func(*args, **kwargs) 2025-12-04T08:54:30.9864028Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 389, in forward 2025-12-04T08:54:30.9864114Z value_states = self.v(current_states) 2025-12-04T08:54:30.9864118Z 2025-12-04T08:54:30.9864200Z cudagraph partition due to non gpu ops 2025-12-04T08:54:30.9864281Z cudagraph partition due to non gpu ops 2025-12-04T08:54:30.9864392Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9864596Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9864661Z res = mod(**inputs) 2025-12-04T08:54:30.9864920Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T08:54:30.9864994Z decoder_outputs = self.decoder( 2025-12-04T08:54:30.9865253Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9865326Z layer_outputs = layer_module( 2025-12-04T08:54:30.9865557Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9865647Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9865895Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9865964Z return func(*args, **kwargs) 2025-12-04T08:54:30.9866218Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 565, in forward 2025-12-04T08:54:30.9866301Z self_attention_outputs = self.layer[0]( 2025-12-04T08:54:30.9866580Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9866651Z return func(*args, **kwargs) 2025-12-04T08:54:30.9866897Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 479, in forward 2025-12-04T08:54:30.9866988Z attention_output = self.SelfAttention( 2025-12-04T08:54:30.9867253Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9867324Z return func(*args, **kwargs) 2025-12-04T08:54:30.9867579Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 447, in forward 2025-12-04T08:54:30.9867683Z attn_output = self.o(attn_output) 2025-12-04T08:54:30.9867687Z 2025-12-04T08:54:30.9867799Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9868003Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9868069Z res = mod(**inputs) 2025-12-04T08:54:30.9868326Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T08:54:30.9868401Z decoder_outputs = self.decoder( 2025-12-04T08:54:30.9868658Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9868751Z layer_outputs = layer_module( 2025-12-04T08:54:30.9868983Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9869070Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9869322Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9869391Z return func(*args, **kwargs) 2025-12-04T08:54:30.9869647Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 589, in forward 2025-12-04T08:54:30.9869729Z cross_attention_outputs = self.layer[1]( 2025-12-04T08:54:30.9869985Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9870054Z return func(*args, **kwargs) 2025-12-04T08:54:30.9870305Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 517, in forward 2025-12-04T08:54:30.9870401Z attention_output = self.EncDecAttention( 2025-12-04T08:54:30.9870648Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9870719Z return func(*args, **kwargs) 2025-12-04T08:54:30.9870973Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 367, in forward 2025-12-04T08:54:30.9871052Z query_states = self.q(hidden_states) 2025-12-04T08:54:30.9871055Z 2025-12-04T08:54:30.9871168Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9871372Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9871438Z res = mod(**inputs) 2025-12-04T08:54:30.9871699Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T08:54:30.9871776Z decoder_outputs = self.decoder( 2025-12-04T08:54:30.9872030Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9872102Z layer_outputs = layer_module( 2025-12-04T08:54:30.9872333Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9872420Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9872686Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9872756Z return func(*args, **kwargs) 2025-12-04T08:54:30.9873014Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 589, in forward 2025-12-04T08:54:30.9873096Z cross_attention_outputs = self.layer[1]( 2025-12-04T08:54:30.9873372Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9873442Z return func(*args, **kwargs) 2025-12-04T08:54:30.9873693Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 517, in forward 2025-12-04T08:54:30.9873806Z attention_output = self.EncDecAttention( 2025-12-04T08:54:30.9874054Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9874122Z return func(*args, **kwargs) 2025-12-04T08:54:30.9874378Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 388, in forward 2025-12-04T08:54:30.9874456Z key_states = self.k(current_states) 2025-12-04T08:54:30.9874460Z 2025-12-04T08:54:30.9874571Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9874793Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9874860Z res = mod(**inputs) 2025-12-04T08:54:30.9875116Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T08:54:30.9875189Z decoder_outputs = self.decoder( 2025-12-04T08:54:30.9875447Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9875520Z layer_outputs = layer_module( 2025-12-04T08:54:30.9875759Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9875846Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9876093Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9876161Z return func(*args, **kwargs) 2025-12-04T08:54:30.9876425Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 589, in forward 2025-12-04T08:54:30.9876512Z cross_attention_outputs = self.layer[1]( 2025-12-04T08:54:30.9876783Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9876858Z return func(*args, **kwargs) 2025-12-04T08:54:30.9877117Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 517, in forward 2025-12-04T08:54:30.9877217Z attention_output = self.EncDecAttention( 2025-12-04T08:54:30.9877480Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9877552Z return func(*args, **kwargs) 2025-12-04T08:54:30.9877821Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 389, in forward 2025-12-04T08:54:30.9877904Z value_states = self.v(current_states) 2025-12-04T08:54:30.9877911Z 2025-12-04T08:54:30.9878007Z cudagraph partition due to non gpu ops 2025-12-04T08:54:30.9878094Z cudagraph partition due to non gpu ops 2025-12-04T08:54:30.9878442Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9878684Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9878760Z res = mod(**inputs) 2025-12-04T08:54:30.9879050Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T08:54:30.9879154Z decoder_outputs = self.decoder( 2025-12-04T08:54:30.9879446Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9879529Z layer_outputs = layer_module( 2025-12-04T08:54:30.9879775Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9879861Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9880161Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9880236Z return func(*args, **kwargs) 2025-12-04T08:54:30.9880531Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 589, in forward 2025-12-04T08:54:30.9880621Z cross_attention_outputs = self.layer[1]( 2025-12-04T08:54:30.9880885Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9880966Z return func(*args, **kwargs) 2025-12-04T08:54:30.9881214Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 517, in forward 2025-12-04T08:54:30.9881300Z attention_output = self.EncDecAttention( 2025-12-04T08:54:30.9881576Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9881644Z return func(*args, **kwargs) 2025-12-04T08:54:30.9881899Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 447, in forward 2025-12-04T08:54:30.9881979Z attn_output = self.o(attn_output) 2025-12-04T08:54:30.9881983Z 2025-12-04T08:54:30.9882088Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9882301Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9882366Z res = mod(**inputs) 2025-12-04T08:54:30.9882625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T08:54:30.9882699Z decoder_outputs = self.decoder( 2025-12-04T08:54:30.9882951Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9883031Z layer_outputs = layer_module( 2025-12-04T08:54:30.9883269Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9883348Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9883605Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9883675Z return func(*args, **kwargs) 2025-12-04T08:54:30.9883930Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 589, in forward 2025-12-04T08:54:30.9884014Z cross_attention_outputs = self.layer[1]( 2025-12-04T08:54:30.9884264Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9884341Z return func(*args, **kwargs) 2025-12-04T08:54:30.9884588Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 529, in forward 2025-12-04T08:54:30.9884724Z layer_output = hidden_states + self.dropout(attention_output[0]) 2025-12-04T08:54:30.9884736Z 2025-12-04T08:54:30.9884841Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9885046Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9885118Z res = mod(**inputs) 2025-12-04T08:54:30.9885370Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T08:54:30.9885461Z decoder_outputs = self.decoder( 2025-12-04T08:54:30.9885718Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9885791Z layer_outputs = layer_module( 2025-12-04T08:54:30.9886028Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9886125Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9886390Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9886466Z return func(*args, **kwargs) 2025-12-04T08:54:30.9886733Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T08:54:30.9886827Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T08:54:30.9887084Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T08:54:30.9887205Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T08:54:30.9887465Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 184, in forward 2025-12-04T08:54:30.9887587Z hidden_gelu = self.act(self.wi_0(hidden_states)) 2025-12-04T08:54:30.9887590Z 2025-12-04T08:54:30.9887697Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9887909Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9887974Z res = mod(**inputs) 2025-12-04T08:54:30.9888234Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T08:54:30.9888306Z decoder_outputs = self.decoder( 2025-12-04T08:54:30.9888553Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9888635Z layer_outputs = layer_module( 2025-12-04T08:54:30.9888862Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9888941Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9889204Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9889280Z return func(*args, **kwargs) 2025-12-04T08:54:30.9889547Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T08:54:30.9889647Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T08:54:30.9889908Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T08:54:30.9890038Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T08:54:30.9890301Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 185, in forward 2025-12-04T08:54:30.9890398Z hidden_linear = self.wi_1(hidden_states) 2025-12-04T08:54:30.9890409Z 2025-12-04T08:54:30.9890511Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9890717Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9890791Z res = mod(**inputs) 2025-12-04T08:54:30.9891046Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T08:54:30.9891124Z decoder_outputs = self.decoder( 2025-12-04T08:54:30.9891395Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9891473Z layer_outputs = layer_module( 2025-12-04T08:54:30.9891743Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9891829Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9892088Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9892167Z return func(*args, **kwargs) 2025-12-04T08:54:30.9892458Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T08:54:30.9892556Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T08:54:30.9892834Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T08:54:30.9892970Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T08:54:30.9893224Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 186, in forward 2025-12-04T08:54:30.9893312Z hidden_states = hidden_gelu * hidden_linear 2025-12-04T08:54:30.9893317Z 2025-12-04T08:54:30.9893422Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9893633Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9893698Z res = mod(**inputs) 2025-12-04T08:54:30.9893980Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-12-04T08:54:30.9894054Z decoder_outputs = self.decoder( 2025-12-04T08:54:30.9894305Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1087, in forward 2025-12-04T08:54:30.9894383Z layer_outputs = layer_module( 2025-12-04T08:54:30.9894612Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:54:30.9894691Z return super().__call__(*args, **kwargs) 2025-12-04T08:54:30.9894951Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:54:30.9895021Z return func(*args, **kwargs) 2025-12-04T08:54:30.9895275Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 615, in forward 2025-12-04T08:54:30.9895366Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T08:54:30.9895616Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-12-04T08:54:30.9895737Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T08:54:30.9895985Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 199, in forward 2025-12-04T08:54:30.9896067Z hidden_states = self.wo(hidden_states) 2025-12-04T08:54:30.9896078Z 2025-12-04T08:54:30.9896183Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9896390Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9896463Z res = mod(**inputs) 2025-12-04T08:54:30.9896715Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1816, in forward 2025-12-04T08:54:30.9896803Z lm_logits = self.lm_head(sequence_output) 2025-12-04T08:54:30.9896809Z 2025-12-04T08:54:30.9896920Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:54:30.9897123Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:54:30.9897196Z res = mod(**inputs) 2025-12-04T08:54:30.9897447Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1823, in forward 2025-12-04T08:54:30.9897598Z loss = loss_fct(lm_logits.view(-1, lm_logits.size(-1)), labels.view(-1)) 2025-12-04T08:54:30.9897602Z 2025-12-04T08:54:43.3493088Z Compilation time (from dynamo_timed): 23.609408509 2025-12-04T08:54:43.3666210Z pass 2025-12-04T08:54:43.3667281Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T08:54:43.3668753Z TIMING: _recursive_pre_grad_passes:0.01586 _recursive_joint_graph_passes:0.8535 _recursive_post_grad_passes:0.09298 async_compile.wait:1.31117 code_gen:11.61944 inductor_compile:13.98868 backend_compile:19.39985 gc:0.0002 entire_frame_compile:23.60941 total_wall_time:23.60941 2025-12-04T08:54:43.3670413Z STATS: call_* op count: 1189 | FakeTensorMode.__torch_dispatch__:16428 | FakeTensor.__torch_dispatch__:6387 | ProxyTorchDispatchMode.__torch_dispatch__:4865 2025-12-04T08:54:43.3671306Z Dynamo produced 1 graphs covering 1189 ops with 0 graph breaks (0 unique) 2025-12-04T08:54:46.1903387Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/__init__.py:65: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-12-04T08:54:46.1904273Z import pynvml # type: ignore[import] 2025-12-04T08:54:49.8237101Z 2025-12-04T08:54:49.8401432Z loading model: 0it [00:00, ?it/s]If you want to use `MegatronBertForCausalLM` as a standalone, add `is_decoder=True.` 2025-12-04T08:54:49.8402669Z WARNING:transformers.models.megatron_bert.modeling_megatron_bert:If you want to use `MegatronBertForCausalLM` as a standalone, add `is_decoder=True.` 2025-12-04T08:54:53.4559074Z 2025-12-04T08:54:53.4560002Z loading model: 0it [00:03, ?it/s] 2025-12-04T08:54:53.4584616Z cpu eval MegatronBertForCausalLM 2025-12-04T08:54:57.8868196Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T08:54:59.8415436Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T08:55:01.8167744Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T08:55:16.9509961Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:16.9510618Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:16.9511040Z res = mod(**inputs) 2025-12-04T08:55:16.9511634Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:16.9512219Z outputs = self.bert( 2025-12-04T08:55:16.9512771Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:16.9513304Z encoder_outputs = self.encoder( 2025-12-04T08:55:16.9513845Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:16.9514339Z layer_outputs = layer_module( 2025-12-04T08:55:16.9514756Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:16.9515172Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:16.9515619Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:16.9516060Z return func(*args, **kwargs) 2025-12-04T08:55:16.9516520Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T08:55:16.9517019Z self_attention_outputs = self.attention( 2025-12-04T08:55:16.9517471Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:16.9517899Z return func(*args, **kwargs) 2025-12-04T08:55:16.9518591Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T08:55:16.9519403Z self_outputs = self.self( 2025-12-04T08:55:16.9519845Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:16.9520255Z return func(*args, **kwargs) 2025-12-04T08:55:16.9520853Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 221, in forward 2025-12-04T08:55:16.9521421Z query_layer = self.query(hidden_states) 2025-12-04T08:55:16.9521584Z 2025-12-04T08:55:16.9521717Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:16.9522126Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:16.9522541Z res = mod(**inputs) 2025-12-04T08:55:16.9522998Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:16.9523498Z outputs = self.bert( 2025-12-04T08:55:16.9523963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:16.9524451Z encoder_outputs = self.encoder( 2025-12-04T08:55:16.9524922Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:16.9525534Z layer_outputs = layer_module( 2025-12-04T08:55:16.9525923Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:16.9526333Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:16.9526753Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:16.9527160Z return func(*args, **kwargs) 2025-12-04T08:55:16.9527622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T08:55:16.9528100Z self_attention_outputs = self.attention( 2025-12-04T08:55:16.9528524Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:16.9528924Z return func(*args, **kwargs) 2025-12-04T08:55:16.9529379Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T08:55:16.9529845Z self_outputs = self.self( 2025-12-04T08:55:16.9530247Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:16.9530651Z return func(*args, **kwargs) 2025-12-04T08:55:16.9531096Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 245, in forward 2025-12-04T08:55:16.9531570Z key_layer = self.key(current_states) 2025-12-04T08:55:16.9531719Z 2025-12-04T08:55:16.9531838Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:16.9532251Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:16.9532610Z res = mod(**inputs) 2025-12-04T08:55:16.9533053Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:16.9533505Z outputs = self.bert( 2025-12-04T08:55:16.9533944Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:16.9534409Z encoder_outputs = self.encoder( 2025-12-04T08:55:16.9534856Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:16.9535319Z layer_outputs = layer_module( 2025-12-04T08:55:16.9535755Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:16.9536167Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:16.9536577Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:16.9536994Z return func(*args, **kwargs) 2025-12-04T08:55:16.9537476Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T08:55:16.9537967Z self_attention_outputs = self.attention( 2025-12-04T08:55:16.9538394Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:16.9538839Z return func(*args, **kwargs) 2025-12-04T08:55:16.9539306Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T08:55:16.9539770Z self_outputs = self.self( 2025-12-04T08:55:16.9540185Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:16.9540605Z return func(*args, **kwargs) 2025-12-04T08:55:16.9541087Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 249, in forward 2025-12-04T08:55:16.9541599Z value_layer = self.value(current_states) 2025-12-04T08:55:16.9541763Z 2025-12-04T08:55:16.9541858Z cudagraph partition due to non gpu ops 2025-12-04T08:55:16.9542108Z cudagraph partition due to non gpu ops 2025-12-04T08:55:16.9542382Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:16.9542783Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:16.9543149Z res = mod(**inputs) 2025-12-04T08:55:16.9543601Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:16.9544065Z outputs = self.bert( 2025-12-04T08:55:16.9544523Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:16.9545018Z encoder_outputs = self.encoder( 2025-12-04T08:55:16.9545500Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:16.9545973Z layer_outputs = layer_module( 2025-12-04T08:55:16.9546375Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:16.9546792Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:16.9547222Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:16.9547658Z return func(*args, **kwargs) 2025-12-04T08:55:16.9548126Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T08:55:16.9548623Z self_attention_outputs = self.attention( 2025-12-04T08:55:16.9549053Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:16.9549482Z return func(*args, **kwargs) 2025-12-04T08:55:16.9549949Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 375, in forward 2025-12-04T08:55:16.9550500Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T08:55:16.9551035Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 322, in forward 2025-12-04T08:55:16.9551531Z hidden_states = self.dense(hidden_states) 2025-12-04T08:55:16.9551710Z 2025-12-04T08:55:16.9551842Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:16.9552253Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:16.9552612Z res = mod(**inputs) 2025-12-04T08:55:16.9553096Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:16.9553562Z outputs = self.bert( 2025-12-04T08:55:16.9553991Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:16.9554478Z encoder_outputs = self.encoder( 2025-12-04T08:55:16.9554939Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:16.9555404Z layer_outputs = layer_module( 2025-12-04T08:55:16.9555798Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:16.9556195Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:16.9556625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:16.9557273Z return func(*args, **kwargs) 2025-12-04T08:55:16.9557795Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T08:55:16.9558393Z layer_output = apply_chunking_to_forward( 2025-12-04T08:55:16.9558866Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:55:16.9559310Z return forward_fn(*input_tensors) 2025-12-04T08:55:16.9559835Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-12-04T08:55:16.9560373Z intermediate_output = self.intermediate(ln_output) 2025-12-04T08:55:16.9560874Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 391, in forward 2025-12-04T08:55:16.9561363Z hidden_states = self.dense(hidden_states) 2025-12-04T08:55:16.9561527Z 2025-12-04T08:55:16.9561649Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:16.9562058Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:16.9562426Z res = mod(**inputs) 2025-12-04T08:55:16.9562872Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:16.9563353Z outputs = self.bert( 2025-12-04T08:55:16.9563799Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:16.9564259Z encoder_outputs = self.encoder( 2025-12-04T08:55:16.9564720Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:16.9565186Z layer_outputs = layer_module( 2025-12-04T08:55:16.9565585Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:16.9565988Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:16.9566411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:16.9566824Z return func(*args, **kwargs) 2025-12-04T08:55:16.9567270Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T08:55:16.9567764Z layer_output = apply_chunking_to_forward( 2025-12-04T08:55:16.9568316Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:55:16.9568765Z return forward_fn(*input_tensors) 2025-12-04T08:55:16.9569257Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-12-04T08:55:16.9569805Z intermediate_output = self.intermediate(ln_output) 2025-12-04T08:55:16.9570321Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 392, in forward 2025-12-04T08:55:16.9570874Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:55:16.9571295Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T08:55:16.9571683Z return self.act(input) 2025-12-04T08:55:16.9571807Z 2025-12-04T08:55:16.9571936Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:16.9572336Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:16.9572693Z res = mod(**inputs) 2025-12-04T08:55:16.9573146Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:16.9573624Z outputs = self.bert( 2025-12-04T08:55:16.9574063Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:16.9574539Z encoder_outputs = self.encoder( 2025-12-04T08:55:16.9575011Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:16.9575484Z layer_outputs = layer_module( 2025-12-04T08:55:16.9575864Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:16.9576267Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:16.9576692Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:16.9577114Z return func(*args, **kwargs) 2025-12-04T08:55:16.9577567Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T08:55:16.9578040Z layer_output = apply_chunking_to_forward( 2025-12-04T08:55:16.9578489Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:55:16.9578914Z return forward_fn(*input_tensors) 2025-12-04T08:55:16.9579408Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-12-04T08:55:16.9579970Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T08:55:16.9580507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 404, in forward 2025-12-04T08:55:16.9580979Z hidden_states = self.dense(hidden_states) 2025-12-04T08:55:16.9581141Z 2025-12-04T08:55:16.9581256Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:16.9581681Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:16.9582042Z res = mod(**inputs) 2025-12-04T08:55:16.9582476Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:16.9582937Z outputs = self.bert( 2025-12-04T08:55:16.9583378Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:16.9583848Z encoder_outputs = self.encoder( 2025-12-04T08:55:16.9584318Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:16.9584790Z layer_outputs = layer_module( 2025-12-04T08:55:16.9585179Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:16.9585604Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:16.9586040Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:16.9586590Z return func(*args, **kwargs) 2025-12-04T08:55:16.9587058Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T08:55:16.9587534Z self_attention_outputs = self.attention( 2025-12-04T08:55:16.9587967Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:16.9588377Z return func(*args, **kwargs) 2025-12-04T08:55:16.9588835Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T08:55:16.9589342Z self_outputs = self.self( 2025-12-04T08:55:16.9589758Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:16.9590169Z return func(*args, **kwargs) 2025-12-04T08:55:16.9590617Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 221, in forward 2025-12-04T08:55:16.9591097Z query_layer = self.query(hidden_states) 2025-12-04T08:55:16.9591248Z 2025-12-04T08:55:16.9591988Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:16.9592402Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:16.9592764Z res = mod(**inputs) 2025-12-04T08:55:16.9593219Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:16.9593697Z outputs = self.bert( 2025-12-04T08:55:16.9594141Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:16.9594626Z encoder_outputs = self.encoder( 2025-12-04T08:55:16.9595103Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:16.9595582Z layer_outputs = layer_module( 2025-12-04T08:55:16.9595991Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:16.9596401Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:16.9596843Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:16.9597265Z return func(*args, **kwargs) 2025-12-04T08:55:16.9597720Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T08:55:16.9598273Z self_attention_outputs = self.attention( 2025-12-04T08:55:16.9598725Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:16.9599141Z return func(*args, **kwargs) 2025-12-04T08:55:16.9599620Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T08:55:16.9600098Z self_outputs = self.self( 2025-12-04T08:55:16.9600533Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:16.9600948Z return func(*args, **kwargs) 2025-12-04T08:55:16.9601416Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 245, in forward 2025-12-04T08:55:16.9601911Z key_layer = self.key(current_states) 2025-12-04T08:55:16.9602068Z 2025-12-04T08:55:16.9602216Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:16.9602620Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:16.9602984Z res = mod(**inputs) 2025-12-04T08:55:16.9603439Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:16.9603942Z outputs = self.bert( 2025-12-04T08:55:16.9604396Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:16.9604883Z encoder_outputs = self.encoder( 2025-12-04T08:55:16.9605356Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:16.9605834Z layer_outputs = layer_module( 2025-12-04T08:55:16.9606258Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:16.9606667Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:16.9607094Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:16.9607518Z return func(*args, **kwargs) 2025-12-04T08:55:16.9607994Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T08:55:16.9608498Z self_attention_outputs = self.attention( 2025-12-04T08:55:16.9608943Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:16.9609368Z return func(*args, **kwargs) 2025-12-04T08:55:16.9609840Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T08:55:16.9610312Z self_outputs = self.self( 2025-12-04T08:55:16.9610702Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:16.9611130Z return func(*args, **kwargs) 2025-12-04T08:55:16.9611602Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 249, in forward 2025-12-04T08:55:16.9612112Z value_layer = self.value(current_states) 2025-12-04T08:55:16.9612273Z 2025-12-04T08:55:16.9612367Z cudagraph partition due to non gpu ops 2025-12-04T08:55:16.9612611Z cudagraph partition due to non gpu ops 2025-12-04T08:55:16.9612881Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:16.9613286Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:16.9613651Z res = mod(**inputs) 2025-12-04T08:55:16.9614118Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:16.9614588Z outputs = self.bert( 2025-12-04T08:55:16.9615053Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:16.9615544Z encoder_outputs = self.encoder( 2025-12-04T08:55:16.9616041Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:16.9616538Z layer_outputs = layer_module( 2025-12-04T08:55:16.9616954Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:16.9617362Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:16.9617787Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:16.9618206Z return func(*args, **kwargs) 2025-12-04T08:55:16.9618701Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T08:55:16.9619210Z self_attention_outputs = self.attention( 2025-12-04T08:55:16.9619643Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:16.9620094Z return func(*args, **kwargs) 2025-12-04T08:55:16.9620555Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 375, in forward 2025-12-04T08:55:16.9621246Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T08:55:16.9621803Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 322, in forward 2025-12-04T08:55:16.9622400Z hidden_states = self.dense(hidden_states) 2025-12-04T08:55:16.9622560Z 2025-12-04T08:55:16.9622675Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:16.9623082Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:16.9623428Z res = mod(**inputs) 2025-12-04T08:55:16.9623878Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:16.9624356Z outputs = self.bert( 2025-12-04T08:55:16.9624800Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:16.9625279Z encoder_outputs = self.encoder( 2025-12-04T08:55:16.9625762Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:16.9626312Z layer_outputs = layer_module( 2025-12-04T08:55:16.9626705Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:16.9627113Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:16.9627543Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:16.9627968Z return func(*args, **kwargs) 2025-12-04T08:55:16.9628425Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T08:55:16.9628920Z layer_output = apply_chunking_to_forward( 2025-12-04T08:55:16.9629386Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:55:16.9629831Z return forward_fn(*input_tensors) 2025-12-04T08:55:16.9630343Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-12-04T08:55:16.9630894Z intermediate_output = self.intermediate(ln_output) 2025-12-04T08:55:16.9631427Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 391, in forward 2025-12-04T08:55:16.9631912Z hidden_states = self.dense(hidden_states) 2025-12-04T08:55:16.9632077Z 2025-12-04T08:55:16.9632194Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:16.9632602Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:16.9632966Z res = mod(**inputs) 2025-12-04T08:55:16.9633449Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:16.9633930Z outputs = self.bert( 2025-12-04T08:55:16.9634379Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:16.9634887Z encoder_outputs = self.encoder( 2025-12-04T08:55:16.9635363Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:16.9635839Z layer_outputs = layer_module( 2025-12-04T08:55:16.9636273Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:16.9636679Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:16.9637114Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:16.9637542Z return func(*args, **kwargs) 2025-12-04T08:55:16.9637999Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T08:55:16.9638580Z layer_output = apply_chunking_to_forward( 2025-12-04T08:55:16.9639048Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:55:16.9639496Z return forward_fn(*input_tensors) 2025-12-04T08:55:16.9640006Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-12-04T08:55:16.9640556Z intermediate_output = self.intermediate(ln_output) 2025-12-04T08:55:16.9641070Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 392, in forward 2025-12-04T08:55:16.9641598Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:55:16.9642033Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T08:55:16.9642429Z return self.act(input) 2025-12-04T08:55:16.9642558Z 2025-12-04T08:55:16.9642688Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:16.9664842Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:16.9665255Z res = mod(**inputs) 2025-12-04T08:55:16.9665764Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:16.9666268Z outputs = self.bert( 2025-12-04T08:55:16.9666723Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:16.9667211Z encoder_outputs = self.encoder( 2025-12-04T08:55:16.9667691Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:16.9668178Z layer_outputs = layer_module( 2025-12-04T08:55:16.9668588Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:16.9669012Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:16.9669454Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:16.9669881Z return func(*args, **kwargs) 2025-12-04T08:55:16.9670340Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T08:55:16.9670832Z layer_output = apply_chunking_to_forward( 2025-12-04T08:55:16.9671422Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:55:16.9671862Z return forward_fn(*input_tensors) 2025-12-04T08:55:16.9672380Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-12-04T08:55:16.9672970Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T08:55:16.9673554Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 404, in forward 2025-12-04T08:55:16.9674031Z hidden_states = self.dense(hidden_states) 2025-12-04T08:55:16.9674244Z 2025-12-04T08:55:16.9674369Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:16.9674776Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:16.9675135Z res = mod(**inputs) 2025-12-04T08:55:16.9675574Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:16.9676037Z outputs = self.bert( 2025-12-04T08:55:16.9676482Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:16.9676982Z encoder_outputs = self.encoder( 2025-12-04T08:55:16.9677462Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:16.9677943Z layer_outputs = layer_module( 2025-12-04T08:55:16.9678450Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:16.9678875Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:16.9679318Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:16.9679756Z return func(*args, **kwargs) 2025-12-04T08:55:16.9680224Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T08:55:16.9680737Z layer_output = apply_chunking_to_forward( 2025-12-04T08:55:16.9681208Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:55:16.9681668Z return forward_fn(*input_tensors) 2025-12-04T08:55:16.9682178Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-12-04T08:55:16.9682776Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T08:55:16.9683335Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 406, in forward 2025-12-04T08:55:16.9683839Z return input_tensor + hidden_states 2025-12-04T08:55:16.9683993Z 2025-12-04T08:55:16.9684117Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:16.9684544Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:16.9684942Z res = mod(**inputs) 2025-12-04T08:55:16.9685415Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:16.9685888Z outputs = self.bert( 2025-12-04T08:55:16.9686368Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:16.9686864Z encoder_outputs = self.encoder( 2025-12-04T08:55:16.9687347Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:16.9687843Z layer_outputs = layer_module( 2025-12-04T08:55:16.9688272Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:16.9688689Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:16.9689131Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:16.9689568Z return func(*args, **kwargs) 2025-12-04T08:55:16.9690054Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T08:55:16.9690554Z self_attention_outputs = self.attention( 2025-12-04T08:55:16.9691009Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:16.9691434Z return func(*args, **kwargs) 2025-12-04T08:55:16.9691908Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T08:55:16.9692380Z self_outputs = self.self( 2025-12-04T08:55:16.9692795Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:16.9693219Z return func(*args, **kwargs) 2025-12-04T08:55:16.9693702Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 221, in forward 2025-12-04T08:55:16.9694187Z query_layer = self.query(hidden_states) 2025-12-04T08:55:16.9694355Z 2025-12-04T08:55:16.9694486Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:16.9694894Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:16.9695250Z res = mod(**inputs) 2025-12-04T08:55:16.9695695Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:16.9696159Z outputs = self.bert( 2025-12-04T08:55:16.9696605Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:16.9697091Z encoder_outputs = self.encoder( 2025-12-04T08:55:16.9697557Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:16.9698029Z layer_outputs = layer_module( 2025-12-04T08:55:16.9698424Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:16.9698833Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:16.9699264Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:16.9699700Z return func(*args, **kwargs) 2025-12-04T08:55:16.9700143Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T08:55:16.9700621Z self_attention_outputs = self.attention( 2025-12-04T08:55:16.9701044Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:16.9701453Z return func(*args, **kwargs) 2025-12-04T08:55:16.9701920Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T08:55:16.9702398Z self_outputs = self.self( 2025-12-04T08:55:16.9702818Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:16.9703233Z return func(*args, **kwargs) 2025-12-04T08:55:16.9703722Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 245, in forward 2025-12-04T08:55:16.9704235Z key_layer = self.key(current_states) 2025-12-04T08:55:16.9704386Z 2025-12-04T08:55:16.9704510Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:16.9704901Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:16.9705258Z res = mod(**inputs) 2025-12-04T08:55:16.9705735Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:16.9706224Z outputs = self.bert( 2025-12-04T08:55:16.9706668Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:16.9707177Z encoder_outputs = self.encoder( 2025-12-04T08:55:16.9707651Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:16.9708109Z layer_outputs = layer_module( 2025-12-04T08:55:16.9708496Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:16.9708893Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:16.9709304Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:16.9709737Z return func(*args, **kwargs) 2025-12-04T08:55:16.9710189Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T08:55:16.9710676Z self_attention_outputs = self.attention( 2025-12-04T08:55:16.9711092Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:16.9711499Z return func(*args, **kwargs) 2025-12-04T08:55:16.9711958Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T08:55:16.9712421Z self_outputs = self.self( 2025-12-04T08:55:16.9712824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:16.9713242Z return func(*args, **kwargs) 2025-12-04T08:55:16.9713708Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 249, in forward 2025-12-04T08:55:16.9714195Z value_layer = self.value(current_states) 2025-12-04T08:55:16.9714358Z 2025-12-04T08:55:16.9714450Z cudagraph partition due to non gpu ops 2025-12-04T08:55:16.9714698Z cudagraph partition due to non gpu ops 2025-12-04T08:55:16.9714967Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:16.9715367Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:16.9715731Z res = mod(**inputs) 2025-12-04T08:55:16.9716189Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:16.9716656Z outputs = self.bert( 2025-12-04T08:55:16.9717105Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:16.9717585Z encoder_outputs = self.encoder( 2025-12-04T08:55:16.9718060Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:16.9718625Z layer_outputs = layer_module( 2025-12-04T08:55:16.9719030Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:16.9719445Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:16.9719897Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:16.9720324Z return func(*args, **kwargs) 2025-12-04T08:55:16.9720977Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T08:55:16.9721481Z self_attention_outputs = self.attention( 2025-12-04T08:55:16.9722023Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:16.9722444Z return func(*args, **kwargs) 2025-12-04T08:55:16.9722914Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 375, in forward 2025-12-04T08:55:16.9723500Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T08:55:16.9724037Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 322, in forward 2025-12-04T08:55:16.9724533Z hidden_states = self.dense(hidden_states) 2025-12-04T08:55:16.9724694Z 2025-12-04T08:55:16.9724823Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:16.9725239Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:16.9725633Z res = mod(**inputs) 2025-12-04T08:55:16.9726097Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:16.9726568Z outputs = self.bert( 2025-12-04T08:55:16.9727020Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:16.9727501Z encoder_outputs = self.encoder( 2025-12-04T08:55:16.9727975Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:16.9728455Z layer_outputs = layer_module( 2025-12-04T08:55:16.9728846Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:16.9729258Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:16.9729692Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:16.9730114Z return func(*args, **kwargs) 2025-12-04T08:55:16.9730576Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T08:55:16.9731075Z layer_output = apply_chunking_to_forward( 2025-12-04T08:55:16.9731537Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:55:16.9731986Z return forward_fn(*input_tensors) 2025-12-04T08:55:16.9732499Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-12-04T08:55:16.9733059Z intermediate_output = self.intermediate(ln_output) 2025-12-04T08:55:16.9733582Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 391, in forward 2025-12-04T08:55:16.9734074Z hidden_states = self.dense(hidden_states) 2025-12-04T08:55:16.9734242Z 2025-12-04T08:55:16.9734360Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:16.9734772Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:16.9735134Z res = mod(**inputs) 2025-12-04T08:55:16.9735585Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:16.9736056Z outputs = self.bert( 2025-12-04T08:55:16.9736540Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:16.9737017Z encoder_outputs = self.encoder( 2025-12-04T08:55:16.9737494Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:16.9737974Z layer_outputs = layer_module( 2025-12-04T08:55:16.9738391Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:16.9738794Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:16.9739226Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:16.9739679Z return func(*args, **kwargs) 2025-12-04T08:55:16.9740149Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T08:55:16.9740660Z layer_output = apply_chunking_to_forward( 2025-12-04T08:55:16.9741121Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:55:16.9741569Z return forward_fn(*input_tensors) 2025-12-04T08:55:16.9742095Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-12-04T08:55:16.9742626Z intermediate_output = self.intermediate(ln_output) 2025-12-04T08:55:16.9743139Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 392, in forward 2025-12-04T08:55:16.9743663Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:55:16.9744102Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T08:55:16.9744504Z return self.act(input) 2025-12-04T08:55:16.9744629Z 2025-12-04T08:55:16.9744753Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:16.9745148Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:16.9745525Z res = mod(**inputs) 2025-12-04T08:55:16.9745968Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:16.9746452Z outputs = self.bert( 2025-12-04T08:55:16.9746895Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:16.9747390Z encoder_outputs = self.encoder( 2025-12-04T08:55:16.9747866Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:16.9748352Z layer_outputs = layer_module( 2025-12-04T08:55:16.9748743Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:16.9749155Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:16.9749591Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:16.9750018Z return func(*args, **kwargs) 2025-12-04T08:55:16.9750480Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T08:55:16.9750957Z layer_output = apply_chunking_to_forward( 2025-12-04T08:55:16.9751401Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:55:16.9751831Z return forward_fn(*input_tensors) 2025-12-04T08:55:16.9752346Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-12-04T08:55:16.9752920Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T08:55:16.9753466Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 404, in forward 2025-12-04T08:55:16.9753954Z hidden_states = self.dense(hidden_states) 2025-12-04T08:55:16.9754120Z 2025-12-04T08:55:16.9754287Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:16.9754700Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:16.9755060Z res = mod(**inputs) 2025-12-04T08:55:16.9755548Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:16.9756034Z outputs = self.bert( 2025-12-04T08:55:16.9756495Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:16.9756973Z encoder_outputs = self.encoder( 2025-12-04T08:55:16.9757458Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:16.9757975Z layer_outputs = layer_module( 2025-12-04T08:55:16.9758454Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:16.9758872Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:16.9759308Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:16.9759736Z return func(*args, **kwargs) 2025-12-04T08:55:16.9760197Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T08:55:16.9760689Z self_attention_outputs = self.attention( 2025-12-04T08:55:16.9761127Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:16.9761549Z return func(*args, **kwargs) 2025-12-04T08:55:16.9762009Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T08:55:16.9762489Z self_outputs = self.self( 2025-12-04T08:55:16.9762906Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:16.9763330Z return func(*args, **kwargs) 2025-12-04T08:55:16.9763792Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 221, in forward 2025-12-04T08:55:16.9764281Z query_layer = self.query(hidden_states) 2025-12-04T08:55:16.9764437Z 2025-12-04T08:55:16.9764556Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:16.9764965Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:16.9765330Z res = mod(**inputs) 2025-12-04T08:55:16.9765782Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:16.9766250Z outputs = self.bert( 2025-12-04T08:55:16.9766710Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:16.9767187Z encoder_outputs = self.encoder( 2025-12-04T08:55:16.9767668Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:16.9768141Z layer_outputs = layer_module( 2025-12-04T08:55:16.9768533Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:16.9768977Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:16.9769408Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:16.9769837Z return func(*args, **kwargs) 2025-12-04T08:55:16.9770349Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T08:55:16.9770846Z self_attention_outputs = self.attention( 2025-12-04T08:55:16.9771266Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:16.9771697Z return func(*args, **kwargs) 2025-12-04T08:55:16.9772146Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T08:55:16.9772613Z self_outputs = self.self( 2025-12-04T08:55:16.9773014Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:16.9773421Z return func(*args, **kwargs) 2025-12-04T08:55:16.9773867Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 245, in forward 2025-12-04T08:55:16.9774355Z key_layer = self.key(current_states) 2025-12-04T08:55:16.9774510Z 2025-12-04T08:55:16.9774629Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:16.9775039Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:16.9775406Z res = mod(**inputs) 2025-12-04T08:55:16.9775843Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:16.9776310Z outputs = self.bert( 2025-12-04T08:55:16.9776754Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:16.9777215Z encoder_outputs = self.encoder( 2025-12-04T08:55:16.9777680Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:16.9778148Z layer_outputs = layer_module( 2025-12-04T08:55:16.9778541Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:16.9778941Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:16.9779369Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:16.9779788Z return func(*args, **kwargs) 2025-12-04T08:55:16.9780236Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T08:55:16.9780722Z self_attention_outputs = self.attention( 2025-12-04T08:55:16.9781153Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:16.9781570Z return func(*args, **kwargs) 2025-12-04T08:55:16.9782022Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T08:55:16.9782499Z self_outputs = self.self( 2025-12-04T08:55:16.9782905Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:16.9783319Z return func(*args, **kwargs) 2025-12-04T08:55:16.9783770Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 249, in forward 2025-12-04T08:55:16.9784257Z value_layer = self.value(current_states) 2025-12-04T08:55:16.9784410Z 2025-12-04T08:55:16.9784537Z cudagraph partition due to non gpu ops 2025-12-04T08:55:16.9784769Z cudagraph partition due to non gpu ops 2025-12-04T08:55:16.9785027Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:16.9785423Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:16.9785783Z res = mod(**inputs) 2025-12-04T08:55:16.9786250Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:16.9786724Z outputs = self.bert( 2025-12-04T08:55:16.9787172Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:16.9787660Z encoder_outputs = self.encoder( 2025-12-04T08:55:16.9788131Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:16.9788608Z layer_outputs = layer_module( 2025-12-04T08:55:16.9788999Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:16.9789399Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:16.9789828Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:16.9790282Z return func(*args, **kwargs) 2025-12-04T08:55:16.9790754Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T08:55:16.9791257Z self_attention_outputs = self.attention( 2025-12-04T08:55:16.9791705Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:16.9792140Z return func(*args, **kwargs) 2025-12-04T08:55:16.9792615Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 375, in forward 2025-12-04T08:55:16.9793174Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T08:55:16.9793726Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 322, in forward 2025-12-04T08:55:16.9794227Z hidden_states = self.dense(hidden_states) 2025-12-04T08:55:16.9794389Z 2025-12-04T08:55:16.9794515Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:16.9794936Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:16.9795315Z res = mod(**inputs) 2025-12-04T08:55:16.9795782Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:16.9796260Z outputs = self.bert( 2025-12-04T08:55:16.9796723Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:16.9797209Z encoder_outputs = self.encoder( 2025-12-04T08:55:16.9797689Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:16.9798177Z layer_outputs = layer_module( 2025-12-04T08:55:16.9798673Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:16.9799092Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:16.9799521Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:16.9799949Z return func(*args, **kwargs) 2025-12-04T08:55:16.9800413Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T08:55:16.9800944Z layer_output = apply_chunking_to_forward( 2025-12-04T08:55:16.9801400Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:55:16.9801852Z return forward_fn(*input_tensors) 2025-12-04T08:55:16.9802363Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-12-04T08:55:16.9802929Z intermediate_output = self.intermediate(ln_output) 2025-12-04T08:55:16.9803437Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 391, in forward 2025-12-04T08:55:16.9803969Z hidden_states = self.dense(hidden_states) 2025-12-04T08:55:16.9804125Z 2025-12-04T08:55:16.9804250Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:16.9804663Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:16.9805024Z res = mod(**inputs) 2025-12-04T08:55:16.9805475Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:16.9805962Z outputs = self.bert( 2025-12-04T08:55:16.9806430Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:16.9806925Z encoder_outputs = self.encoder( 2025-12-04T08:55:16.9807407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:16.9807888Z layer_outputs = layer_module( 2025-12-04T08:55:16.9808284Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:16.9808705Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:16.9809171Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:16.9809602Z return func(*args, **kwargs) 2025-12-04T08:55:16.9810073Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T08:55:16.9810574Z layer_output = apply_chunking_to_forward( 2025-12-04T08:55:16.9811037Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:55:16.9811468Z return forward_fn(*input_tensors) 2025-12-04T08:55:16.9811973Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-12-04T08:55:16.9812510Z intermediate_output = self.intermediate(ln_output) 2025-12-04T08:55:16.9813018Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 392, in forward 2025-12-04T08:55:16.9813520Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:55:16.9813951Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T08:55:16.9814344Z return self.act(input) 2025-12-04T08:55:16.9814475Z 2025-12-04T08:55:16.9814603Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:16.9815028Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:16.9815391Z res = mod(**inputs) 2025-12-04T08:55:16.9815834Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:16.9816293Z outputs = self.bert( 2025-12-04T08:55:16.9816742Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:16.9817244Z encoder_outputs = self.encoder( 2025-12-04T08:55:16.9817703Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:16.9818156Z layer_outputs = layer_module( 2025-12-04T08:55:16.9818564Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:16.9818968Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:16.9819383Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:16.9819817Z return func(*args, **kwargs) 2025-12-04T08:55:16.9820265Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T08:55:16.9821078Z layer_output = apply_chunking_to_forward( 2025-12-04T08:55:16.9821529Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:55:16.9821968Z return forward_fn(*input_tensors) 2025-12-04T08:55:16.9822464Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-12-04T08:55:16.9823126Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T08:55:16.9823674Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 404, in forward 2025-12-04T08:55:16.9824173Z hidden_states = self.dense(hidden_states) 2025-12-04T08:55:16.9824330Z 2025-12-04T08:55:16.9824460Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:16.9824874Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:16.9825232Z res = mod(**inputs) 2025-12-04T08:55:16.9825689Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:16.9826163Z outputs = self.bert( 2025-12-04T08:55:16.9826611Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:16.9827098Z encoder_outputs = self.encoder( 2025-12-04T08:55:16.9827584Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:16.9828066Z layer_outputs = layer_module( 2025-12-04T08:55:16.9828452Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:16.9828857Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:16.9829292Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:16.9829703Z return func(*args, **kwargs) 2025-12-04T08:55:16.9830159Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T08:55:16.9830643Z layer_output = apply_chunking_to_forward( 2025-12-04T08:55:16.9831095Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:55:16.9831541Z return forward_fn(*input_tensors) 2025-12-04T08:55:16.9832060Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-12-04T08:55:16.9832638Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T08:55:16.9833168Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 406, in forward 2025-12-04T08:55:16.9833759Z return input_tensor + hidden_states 2025-12-04T08:55:16.9833918Z 2025-12-04T08:55:16.9834035Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:16.9834442Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:16.9834810Z res = mod(**inputs) 2025-12-04T08:55:16.9835295Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:16.9835771Z outputs = self.bert( 2025-12-04T08:55:16.9836224Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:16.9836730Z encoder_outputs = self.encoder( 2025-12-04T08:55:16.9837208Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:16.9837684Z layer_outputs = layer_module( 2025-12-04T08:55:16.9838079Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:16.9838550Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:16.9838991Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:16.9839490Z return func(*args, **kwargs) 2025-12-04T08:55:16.9839951Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T08:55:16.9840444Z self_attention_outputs = self.attention( 2025-12-04T08:55:16.9840885Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:16.9841308Z return func(*args, **kwargs) 2025-12-04T08:55:16.9841771Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T08:55:16.9842251Z self_outputs = self.self( 2025-12-04T08:55:16.9842664Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:16.9843089Z return func(*args, **kwargs) 2025-12-04T08:55:16.9843553Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 221, in forward 2025-12-04T08:55:16.9844044Z query_layer = self.query(hidden_states) 2025-12-04T08:55:16.9844204Z 2025-12-04T08:55:16.9844333Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:16.9844738Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:16.9845108Z res = mod(**inputs) 2025-12-04T08:55:16.9845564Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:16.9846039Z outputs = self.bert( 2025-12-04T08:55:16.9846478Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:16.9846980Z encoder_outputs = self.encoder( 2025-12-04T08:55:16.9847453Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:16.9847933Z layer_outputs = layer_module( 2025-12-04T08:55:16.9848319Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:16.9848730Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:16.9849169Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:16.9849571Z return func(*args, **kwargs) 2025-12-04T08:55:16.9850053Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T08:55:16.9850529Z self_attention_outputs = self.attention( 2025-12-04T08:55:16.9850956Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:16.9851366Z return func(*args, **kwargs) 2025-12-04T08:55:16.9851835Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T08:55:16.9852302Z self_outputs = self.self( 2025-12-04T08:55:16.9852728Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:16.9853145Z return func(*args, **kwargs) 2025-12-04T08:55:16.9853605Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 245, in forward 2025-12-04T08:55:16.9854082Z key_layer = self.key(current_states) 2025-12-04T08:55:16.9854242Z 2025-12-04T08:55:16.9854356Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:16.9854751Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:16.9855124Z res = mod(**inputs) 2025-12-04T08:55:16.9855571Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:16.9856018Z outputs = self.bert( 2025-12-04T08:55:16.9856458Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:16.9856920Z encoder_outputs = self.encoder( 2025-12-04T08:55:16.9857369Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:16.9857826Z layer_outputs = layer_module( 2025-12-04T08:55:16.9858213Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:16.9858623Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:16.9859046Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:16.9859455Z return func(*args, **kwargs) 2025-12-04T08:55:16.9859902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T08:55:16.9860377Z self_attention_outputs = self.attention( 2025-12-04T08:55:16.9860788Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:16.9861193Z return func(*args, **kwargs) 2025-12-04T08:55:16.9861641Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T08:55:16.9862094Z self_outputs = self.self( 2025-12-04T08:55:16.9862491Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:16.9862895Z return func(*args, **kwargs) 2025-12-04T08:55:16.9863508Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 249, in forward 2025-12-04T08:55:16.9863989Z value_layer = self.value(current_states) 2025-12-04T08:55:16.9864146Z 2025-12-04T08:55:16.9864256Z cudagraph partition due to non gpu ops 2025-12-04T08:55:16.9864494Z cudagraph partition due to non gpu ops 2025-12-04T08:55:16.9864745Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:16.9865138Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:16.9865518Z res = mod(**inputs) 2025-12-04T08:55:16.9865956Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:16.9866408Z outputs = self.bert( 2025-12-04T08:55:16.9866865Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:16.9867329Z encoder_outputs = self.encoder( 2025-12-04T08:55:16.9867780Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:16.9868259Z layer_outputs = layer_module( 2025-12-04T08:55:16.9868641Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:16.9869042Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:16.9869454Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:16.9869862Z return func(*args, **kwargs) 2025-12-04T08:55:16.9870308Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T08:55:16.9870799Z self_attention_outputs = self.attention( 2025-12-04T08:55:16.9871215Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:16.9871619Z return func(*args, **kwargs) 2025-12-04T08:55:16.9872068Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 375, in forward 2025-12-04T08:55:16.9872582Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T08:55:16.9873103Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 322, in forward 2025-12-04T08:55:16.9873572Z hidden_states = self.dense(hidden_states) 2025-12-04T08:55:16.9873721Z 2025-12-04T08:55:16.9873840Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:16.9874224Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:16.9874585Z res = mod(**inputs) 2025-12-04T08:55:16.9875041Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:16.9875512Z outputs = self.bert( 2025-12-04T08:55:16.9875961Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:16.9876434Z encoder_outputs = self.encoder( 2025-12-04T08:55:16.9876906Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:16.9877371Z layer_outputs = layer_module( 2025-12-04T08:55:16.9877761Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:16.9878169Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:16.9878687Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:16.9879107Z return func(*args, **kwargs) 2025-12-04T08:55:16.9879574Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T08:55:16.9880070Z layer_output = apply_chunking_to_forward( 2025-12-04T08:55:16.9880539Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:55:16.9880985Z return forward_fn(*input_tensors) 2025-12-04T08:55:16.9881526Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-12-04T08:55:16.9882077Z intermediate_output = self.intermediate(ln_output) 2025-12-04T08:55:16.9882579Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 391, in forward 2025-12-04T08:55:16.9883091Z hidden_states = self.dense(hidden_states) 2025-12-04T08:55:16.9883255Z 2025-12-04T08:55:16.9883373Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:16.9883778Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:16.9884158Z res = mod(**inputs) 2025-12-04T08:55:16.9884609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:16.9885086Z outputs = self.bert( 2025-12-04T08:55:16.9885538Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:16.9886010Z encoder_outputs = self.encoder( 2025-12-04T08:55:16.9886485Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:16.9887003Z layer_outputs = layer_module( 2025-12-04T08:55:16.9887392Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:16.9887803Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:16.9888236Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:16.9888671Z return func(*args, **kwargs) 2025-12-04T08:55:16.9889123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T08:55:16.9889608Z layer_output = apply_chunking_to_forward( 2025-12-04T08:55:16.9890065Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:55:16.9890506Z return forward_fn(*input_tensors) 2025-12-04T08:55:16.9891014Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-12-04T08:55:16.9891567Z intermediate_output = self.intermediate(ln_output) 2025-12-04T08:55:16.9892058Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 392, in forward 2025-12-04T08:55:16.9892553Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:55:16.9892974Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T08:55:16.9893356Z return self.act(input) 2025-12-04T08:55:16.9893477Z 2025-12-04T08:55:16.9893595Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:16.9893983Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:16.9894340Z res = mod(**inputs) 2025-12-04T08:55:16.9894777Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:16.9895229Z outputs = self.bert( 2025-12-04T08:55:16.9895661Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:16.9896128Z encoder_outputs = self.encoder( 2025-12-04T08:55:16.9896584Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:16.9897057Z layer_outputs = layer_module( 2025-12-04T08:55:16.9897448Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:16.9897843Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:16.9898269Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:16.9898680Z return func(*args, **kwargs) 2025-12-04T08:55:16.9899155Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T08:55:16.9899634Z layer_output = apply_chunking_to_forward( 2025-12-04T08:55:16.9900094Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:55:16.9900526Z return forward_fn(*input_tensors) 2025-12-04T08:55:16.9901020Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-12-04T08:55:16.9901569Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T08:55:16.9902082Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 404, in forward 2025-12-04T08:55:16.9902574Z hidden_states = self.dense(hidden_states) 2025-12-04T08:55:16.9902736Z 2025-12-04T08:55:16.9902851Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:16.9903249Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:16.9903600Z res = mod(**inputs) 2025-12-04T08:55:16.9904035Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:16.9904491Z outputs = self.bert( 2025-12-04T08:55:16.9904916Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:16.9905378Z encoder_outputs = self.encoder( 2025-12-04T08:55:16.9905834Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:16.9906294Z layer_outputs = layer_module( 2025-12-04T08:55:16.9906670Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:16.9907071Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:16.9907489Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:16.9907899Z return func(*args, **kwargs) 2025-12-04T08:55:16.9908338Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T08:55:16.9908810Z self_attention_outputs = self.attention( 2025-12-04T08:55:16.9909242Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:16.9909621Z return func(*args, **kwargs) 2025-12-04T08:55:16.9910061Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T08:55:16.9910521Z self_outputs = self.self( 2025-12-04T08:55:16.9910917Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:16.9911312Z return func(*args, **kwargs) 2025-12-04T08:55:16.9911756Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 221, in forward 2025-12-04T08:55:16.9912227Z query_layer = self.query(hidden_states) 2025-12-04T08:55:16.9912376Z 2025-12-04T08:55:16.9912522Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:16.9912915Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:16.9913279Z res = mod(**inputs) 2025-12-04T08:55:16.9913740Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:16.9914214Z outputs = self.bert( 2025-12-04T08:55:16.9914651Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:16.9915113Z encoder_outputs = self.encoder( 2025-12-04T08:55:16.9915592Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:16.9916054Z layer_outputs = layer_module( 2025-12-04T08:55:16.9916448Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:16.9916855Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:16.9917276Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:16.9917715Z return func(*args, **kwargs) 2025-12-04T08:55:16.9918180Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T08:55:16.9918731Z self_attention_outputs = self.attention( 2025-12-04T08:55:16.9919158Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:16.9919584Z return func(*args, **kwargs) 2025-12-04T08:55:16.9920047Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T08:55:16.9920533Z self_outputs = self.self( 2025-12-04T08:55:16.9921185Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:16.9921600Z return func(*args, **kwargs) 2025-12-04T08:55:16.9922049Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 245, in forward 2025-12-04T08:55:16.9922519Z key_layer = self.key(current_states) 2025-12-04T08:55:16.9922677Z 2025-12-04T08:55:16.9922791Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:16.9923187Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:16.9923560Z res = mod(**inputs) 2025-12-04T08:55:16.9923997Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:16.9924467Z outputs = self.bert( 2025-12-04T08:55:16.9924910Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:16.9925380Z encoder_outputs = self.encoder( 2025-12-04T08:55:16.9925836Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:16.9926311Z layer_outputs = layer_module( 2025-12-04T08:55:16.9926690Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:16.9927078Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:16.9927496Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:16.9927898Z return func(*args, **kwargs) 2025-12-04T08:55:16.9928417Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T08:55:16.9928881Z self_attention_outputs = self.attention( 2025-12-04T08:55:16.9929300Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:16.9929722Z return func(*args, **kwargs) 2025-12-04T08:55:16.9930195Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T08:55:16.9930659Z self_outputs = self.self( 2025-12-04T08:55:16.9931059Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:16.9931503Z return func(*args, **kwargs) 2025-12-04T08:55:16.9931941Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 249, in forward 2025-12-04T08:55:16.9932412Z value_layer = self.value(current_states) 2025-12-04T08:55:16.9932559Z 2025-12-04T08:55:16.9932657Z cudagraph partition due to non gpu ops 2025-12-04T08:55:16.9932895Z cudagraph partition due to non gpu ops 2025-12-04T08:55:16.9933147Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:16.9933540Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:16.9933956Z res = mod(**inputs) 2025-12-04T08:55:16.9934390Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:16.9934850Z outputs = self.bert( 2025-12-04T08:55:16.9935289Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:16.9935754Z encoder_outputs = self.encoder( 2025-12-04T08:55:16.9936213Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:16.9936674Z layer_outputs = layer_module( 2025-12-04T08:55:16.9937065Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:16.9937453Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:16.9937873Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:16.9938280Z return func(*args, **kwargs) 2025-12-04T08:55:16.9938732Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T08:55:16.9939197Z self_attention_outputs = self.attention( 2025-12-04T08:55:16.9939623Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:16.9940040Z return func(*args, **kwargs) 2025-12-04T08:55:16.9940492Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 375, in forward 2025-12-04T08:55:16.9941009Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T08:55:16.9941529Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 322, in forward 2025-12-04T08:55:16.9942004Z hidden_states = self.dense(hidden_states) 2025-12-04T08:55:16.9942156Z 2025-12-04T08:55:16.9942269Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:16.9942666Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:16.9943024Z res = mod(**inputs) 2025-12-04T08:55:16.9943462Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:16.9943913Z outputs = self.bert( 2025-12-04T08:55:16.9944378Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:16.9944847Z encoder_outputs = self.encoder( 2025-12-04T08:55:16.9945301Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:16.9945752Z layer_outputs = layer_module( 2025-12-04T08:55:16.9946153Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:16.9946552Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:16.9946987Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:16.9947393Z return func(*args, **kwargs) 2025-12-04T08:55:16.9947840Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T08:55:16.9948310Z layer_output = apply_chunking_to_forward( 2025-12-04T08:55:16.9948745Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:55:16.9949178Z return forward_fn(*input_tensors) 2025-12-04T08:55:16.9949695Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-12-04T08:55:16.9950222Z intermediate_output = self.intermediate(ln_output) 2025-12-04T08:55:16.9950706Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 391, in forward 2025-12-04T08:55:16.9951179Z hidden_states = self.dense(hidden_states) 2025-12-04T08:55:16.9951335Z 2025-12-04T08:55:16.9951460Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:16.9951868Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:16.9952219Z res = mod(**inputs) 2025-12-04T08:55:16.9952654Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:16.9953114Z outputs = self.bert( 2025-12-04T08:55:16.9953557Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:16.9954033Z encoder_outputs = self.encoder( 2025-12-04T08:55:16.9954503Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:16.9954976Z layer_outputs = layer_module( 2025-12-04T08:55:16.9955364Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:16.9955774Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:16.9956207Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:16.9956618Z return func(*args, **kwargs) 2025-12-04T08:55:16.9957079Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T08:55:16.9957571Z layer_output = apply_chunking_to_forward( 2025-12-04T08:55:16.9958025Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:55:16.9958533Z return forward_fn(*input_tensors) 2025-12-04T08:55:16.9959052Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-12-04T08:55:16.9959595Z intermediate_output = self.intermediate(ln_output) 2025-12-04T08:55:16.9960129Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 392, in forward 2025-12-04T08:55:16.9960649Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:55:16.9961083Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T08:55:16.9961474Z return self.act(input) 2025-12-04T08:55:16.9961600Z 2025-12-04T08:55:16.9961745Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:16.9962157Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:16.9962523Z res = mod(**inputs) 2025-12-04T08:55:16.9963008Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:16.9963491Z outputs = self.bert( 2025-12-04T08:55:16.9963954Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:16.9964443Z encoder_outputs = self.encoder( 2025-12-04T08:55:16.9964913Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:16.9965407Z layer_outputs = layer_module( 2025-12-04T08:55:16.9965801Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:16.9966219Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:16.9966654Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:16.9967154Z return func(*args, **kwargs) 2025-12-04T08:55:16.9967605Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T08:55:16.9968080Z layer_output = apply_chunking_to_forward( 2025-12-04T08:55:16.9968525Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:55:16.9968958Z return forward_fn(*input_tensors) 2025-12-04T08:55:16.9969448Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-12-04T08:55:16.9970007Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T08:55:16.9970548Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 404, in forward 2025-12-04T08:55:16.9971023Z hidden_states = self.dense(hidden_states) 2025-12-04T08:55:16.9971172Z 2025-12-04T08:55:16.9971292Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:16.9971688Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:16.9972063Z res = mod(**inputs) 2025-12-04T08:55:16.9972498Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:16.9972966Z outputs = self.bert( 2025-12-04T08:55:16.9973398Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:16.9973859Z encoder_outputs = self.encoder( 2025-12-04T08:55:16.9974315Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:16.9974768Z layer_outputs = layer_module( 2025-12-04T08:55:16.9975151Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:16.9975551Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:16.9975995Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:16.9976400Z return func(*args, **kwargs) 2025-12-04T08:55:16.9976856Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T08:55:16.9977340Z layer_output = apply_chunking_to_forward( 2025-12-04T08:55:16.9977803Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:55:16.9978230Z return forward_fn(*input_tensors) 2025-12-04T08:55:16.9978726Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-12-04T08:55:16.9979295Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T08:55:16.9979820Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 406, in forward 2025-12-04T08:55:16.9980286Z return input_tensor + hidden_states 2025-12-04T08:55:16.9980436Z 2025-12-04T08:55:16.9980550Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:16.9980944Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:16.9981313Z res = mod(**inputs) 2025-12-04T08:55:16.9981762Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:16.9982225Z outputs = self.bert( 2025-12-04T08:55:16.9982660Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:16.9983114Z encoder_outputs = self.encoder( 2025-12-04T08:55:16.9983571Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:16.9984035Z layer_outputs = layer_module( 2025-12-04T08:55:16.9984410Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:16.9984810Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:16.9985235Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:16.9985646Z return func(*args, **kwargs) 2025-12-04T08:55:16.9986090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T08:55:16.9986570Z self_attention_outputs = self.attention( 2025-12-04T08:55:16.9986993Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:16.9987400Z return func(*args, **kwargs) 2025-12-04T08:55:16.9987845Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T08:55:16.9988302Z self_outputs = self.self( 2025-12-04T08:55:16.9988698Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:16.9988775Z return func(*args, **kwargs) 2025-12-04T08:55:16.9989093Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 221, in forward 2025-12-04T08:55:16.9989191Z query_layer = self.query(hidden_states) 2025-12-04T08:55:16.9989196Z 2025-12-04T08:55:16.9989312Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:16.9989538Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:16.9989609Z res = mod(**inputs) 2025-12-04T08:55:16.9989945Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:16.9990028Z outputs = self.bert( 2025-12-04T08:55:16.9990342Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:16.9990432Z encoder_outputs = self.encoder( 2025-12-04T08:55:16.9990765Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:16.9990845Z layer_outputs = layer_module( 2025-12-04T08:55:16.9991100Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:16.9991205Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:16.9991470Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:16.9991555Z return func(*args, **kwargs) 2025-12-04T08:55:16.9991872Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T08:55:16.9991969Z self_attention_outputs = self.attention( 2025-12-04T08:55:16.9992251Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:16.9992327Z return func(*args, **kwargs) 2025-12-04T08:55:16.9992656Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T08:55:16.9992735Z self_outputs = self.self( 2025-12-04T08:55:16.9993017Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:16.9993098Z return func(*args, **kwargs) 2025-12-04T08:55:16.9993423Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 245, in forward 2025-12-04T08:55:16.9993517Z key_layer = self.key(current_states) 2025-12-04T08:55:16.9993521Z 2025-12-04T08:55:16.9993635Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:16.9993857Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:16.9993938Z res = mod(**inputs) 2025-12-04T08:55:16.9994260Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:16.9994340Z outputs = self.bert( 2025-12-04T08:55:16.9994661Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:16.9994741Z encoder_outputs = self.encoder( 2025-12-04T08:55:16.9995072Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:16.9995150Z layer_outputs = layer_module( 2025-12-04T08:55:16.9995400Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:16.9995496Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:16.9995766Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:16.9995845Z return func(*args, **kwargs) 2025-12-04T08:55:16.9996180Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T08:55:16.9996271Z self_attention_outputs = self.attention( 2025-12-04T08:55:16.9996547Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:16.9996638Z return func(*args, **kwargs) 2025-12-04T08:55:16.9996960Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T08:55:16.9997034Z self_outputs = self.self( 2025-12-04T08:55:16.9997298Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:16.9997413Z return func(*args, **kwargs) 2025-12-04T08:55:16.9997744Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 249, in forward 2025-12-04T08:55:16.9997834Z value_layer = self.value(current_states) 2025-12-04T08:55:16.9997884Z 2025-12-04T08:55:16.9997975Z cudagraph partition due to non gpu ops 2025-12-04T08:55:16.9998063Z cudagraph partition due to non gpu ops 2025-12-04T08:55:16.9998183Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:16.9998501Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:16.9998579Z res = mod(**inputs) 2025-12-04T08:55:16.9998913Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:16.9999015Z outputs = self.bert( 2025-12-04T08:55:16.9999346Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:16.9999446Z encoder_outputs = self.encoder( 2025-12-04T08:55:16.9999762Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:16.9999849Z layer_outputs = layer_module( 2025-12-04T08:55:17.0000094Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:17.0000181Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:17.0000455Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0000530Z return func(*args, **kwargs) 2025-12-04T08:55:17.0000852Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T08:55:17.0000944Z self_attention_outputs = self.attention( 2025-12-04T08:55:17.0001213Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0001297Z return func(*args, **kwargs) 2025-12-04T08:55:17.0001616Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 375, in forward 2025-12-04T08:55:17.0001756Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T08:55:17.0002077Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 322, in forward 2025-12-04T08:55:17.0002169Z hidden_states = self.dense(hidden_states) 2025-12-04T08:55:17.0002173Z 2025-12-04T08:55:17.0002292Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:17.0002512Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:17.0002583Z res = mod(**inputs) 2025-12-04T08:55:17.0002920Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:17.0002994Z outputs = self.bert( 2025-12-04T08:55:17.0003321Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:17.0003400Z encoder_outputs = self.encoder( 2025-12-04T08:55:17.0003737Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:17.0003825Z layer_outputs = layer_module( 2025-12-04T08:55:17.0004067Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:17.0004154Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:17.0004503Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0004579Z return func(*args, **kwargs) 2025-12-04T08:55:17.0004904Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T08:55:17.0005018Z layer_output = apply_chunking_to_forward( 2025-12-04T08:55:17.0005306Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:55:17.0005400Z return forward_fn(*input_tensors) 2025-12-04T08:55:17.0005750Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-12-04T08:55:17.0005870Z intermediate_output = self.intermediate(ln_output) 2025-12-04T08:55:17.0006202Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 391, in forward 2025-12-04T08:55:17.0006292Z hidden_states = self.dense(hidden_states) 2025-12-04T08:55:17.0006296Z 2025-12-04T08:55:17.0006414Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:17.0006633Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:17.0006704Z res = mod(**inputs) 2025-12-04T08:55:17.0007035Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:17.0007109Z outputs = self.bert( 2025-12-04T08:55:17.0007426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:17.0007505Z encoder_outputs = self.encoder( 2025-12-04T08:55:17.0007821Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:17.0007907Z layer_outputs = layer_module( 2025-12-04T08:55:17.0008149Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:17.0008242Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:17.0008514Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0008587Z return func(*args, **kwargs) 2025-12-04T08:55:17.0008910Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T08:55:17.0008999Z layer_output = apply_chunking_to_forward( 2025-12-04T08:55:17.0009285Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:55:17.0009377Z return forward_fn(*input_tensors) 2025-12-04T08:55:17.0009724Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-12-04T08:55:17.0009844Z intermediate_output = self.intermediate(ln_output) 2025-12-04T08:55:17.0010155Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 392, in forward 2025-12-04T08:55:17.0010272Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:55:17.0010527Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T08:55:17.0010606Z return self.act(input) 2025-12-04T08:55:17.0010610Z 2025-12-04T08:55:17.0010728Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:17.0010946Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:17.0011019Z res = mod(**inputs) 2025-12-04T08:55:17.0011373Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:17.0011447Z outputs = self.bert( 2025-12-04T08:55:17.0011769Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:17.0011875Z encoder_outputs = self.encoder( 2025-12-04T08:55:17.0012195Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:17.0012279Z layer_outputs = layer_module( 2025-12-04T08:55:17.0012528Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:17.0012612Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:17.0012912Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0012988Z return func(*args, **kwargs) 2025-12-04T08:55:17.0013309Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T08:55:17.0013400Z layer_output = apply_chunking_to_forward( 2025-12-04T08:55:17.0013685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:55:17.0013774Z return forward_fn(*input_tensors) 2025-12-04T08:55:17.0014119Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-12-04T08:55:17.0014261Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T08:55:17.0014579Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 404, in forward 2025-12-04T08:55:17.0014671Z hidden_states = self.dense(hidden_states) 2025-12-04T08:55:17.0014675Z 2025-12-04T08:55:17.0014793Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:17.0015009Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:17.0015080Z res = mod(**inputs) 2025-12-04T08:55:17.0015403Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:17.0015473Z outputs = self.bert( 2025-12-04T08:55:17.0015810Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:17.0015888Z encoder_outputs = self.encoder( 2025-12-04T08:55:17.0016201Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:17.0016289Z layer_outputs = layer_module( 2025-12-04T08:55:17.0016534Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:17.0016618Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:17.0016890Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0016963Z return func(*args, **kwargs) 2025-12-04T08:55:17.0017304Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T08:55:17.0017396Z self_attention_outputs = self.attention( 2025-12-04T08:55:17.0017659Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0017742Z return func(*args, **kwargs) 2025-12-04T08:55:17.0018089Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T08:55:17.0018175Z self_outputs = self.self( 2025-12-04T08:55:17.0018436Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0018533Z return func(*args, **kwargs) 2025-12-04T08:55:17.0018857Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 221, in forward 2025-12-04T08:55:17.0018945Z query_layer = self.query(hidden_states) 2025-12-04T08:55:17.0018951Z 2025-12-04T08:55:17.0019063Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:17.0019286Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:17.0019355Z res = mod(**inputs) 2025-12-04T08:55:17.0019701Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:17.0019774Z outputs = self.bert( 2025-12-04T08:55:17.0020088Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:17.0020178Z encoder_outputs = self.encoder( 2025-12-04T08:55:17.0020489Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:17.0020573Z layer_outputs = layer_module( 2025-12-04T08:55:17.0021055Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:17.0021146Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:17.0021420Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0021498Z return func(*args, **kwargs) 2025-12-04T08:55:17.0021815Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T08:55:17.0021913Z self_attention_outputs = self.attention( 2025-12-04T08:55:17.0022181Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0022264Z return func(*args, **kwargs) 2025-12-04T08:55:17.0022587Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T08:55:17.0022668Z self_outputs = self.self( 2025-12-04T08:55:17.0022952Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0023027Z return func(*args, **kwargs) 2025-12-04T08:55:17.0023365Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 245, in forward 2025-12-04T08:55:17.0023453Z key_layer = self.key(current_states) 2025-12-04T08:55:17.0023458Z 2025-12-04T08:55:17.0023573Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:17.0023810Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:17.0023882Z res = mod(**inputs) 2025-12-04T08:55:17.0024204Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:17.0024360Z outputs = self.bert( 2025-12-04T08:55:17.0024674Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:17.0024760Z encoder_outputs = self.encoder( 2025-12-04T08:55:17.0025072Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:17.0028062Z layer_outputs = layer_module( 2025-12-04T08:55:17.0028335Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:17.0028424Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:17.0028712Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0028796Z return func(*args, **kwargs) 2025-12-04T08:55:17.0029135Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T08:55:17.0029229Z self_attention_outputs = self.attention( 2025-12-04T08:55:17.0029518Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0029633Z return func(*args, **kwargs) 2025-12-04T08:55:17.0029977Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T08:55:17.0030089Z self_outputs = self.self( 2025-12-04T08:55:17.0030360Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0030443Z return func(*args, **kwargs) 2025-12-04T08:55:17.0030769Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 249, in forward 2025-12-04T08:55:17.0030858Z value_layer = self.value(current_states) 2025-12-04T08:55:17.0030862Z 2025-12-04T08:55:17.0030960Z cudagraph partition due to non gpu ops 2025-12-04T08:55:17.0031045Z cudagraph partition due to non gpu ops 2025-12-04T08:55:17.0031165Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:17.0031386Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:17.0031459Z res = mod(**inputs) 2025-12-04T08:55:17.0031781Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:17.0031853Z outputs = self.bert( 2025-12-04T08:55:17.0032166Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:17.0032253Z encoder_outputs = self.encoder( 2025-12-04T08:55:17.0032573Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:17.0032659Z layer_outputs = layer_module( 2025-12-04T08:55:17.0032908Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:17.0032995Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:17.0033277Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0033354Z return func(*args, **kwargs) 2025-12-04T08:55:17.0033678Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T08:55:17.0033776Z self_attention_outputs = self.attention( 2025-12-04T08:55:17.0034047Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0034158Z return func(*args, **kwargs) 2025-12-04T08:55:17.0034485Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 375, in forward 2025-12-04T08:55:17.0034628Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T08:55:17.0034986Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 322, in forward 2025-12-04T08:55:17.0035196Z hidden_states = self.dense(hidden_states) 2025-12-04T08:55:17.0035200Z 2025-12-04T08:55:17.0035319Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:17.0035543Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:17.0035615Z res = mod(**inputs) 2025-12-04T08:55:17.0035956Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:17.0036028Z outputs = self.bert( 2025-12-04T08:55:17.0036354Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:17.0036442Z encoder_outputs = self.encoder( 2025-12-04T08:55:17.0036766Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:17.0036876Z layer_outputs = layer_module( 2025-12-04T08:55:17.0037129Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:17.0037215Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:17.0037496Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0037574Z return func(*args, **kwargs) 2025-12-04T08:55:17.0037909Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T08:55:17.0038003Z layer_output = apply_chunking_to_forward( 2025-12-04T08:55:17.0038368Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:55:17.0038472Z return forward_fn(*input_tensors) 2025-12-04T08:55:17.0038834Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-12-04T08:55:17.0038952Z intermediate_output = self.intermediate(ln_output) 2025-12-04T08:55:17.0039287Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 391, in forward 2025-12-04T08:55:17.0039382Z hidden_states = self.dense(hidden_states) 2025-12-04T08:55:17.0039386Z 2025-12-04T08:55:17.0039508Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:17.0039734Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:17.0039806Z res = mod(**inputs) 2025-12-04T08:55:17.0040144Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:17.0040222Z outputs = self.bert( 2025-12-04T08:55:17.0040570Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:17.0040651Z encoder_outputs = self.encoder( 2025-12-04T08:55:17.0040966Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:17.0041052Z layer_outputs = layer_module( 2025-12-04T08:55:17.0041298Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:17.0041407Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:17.0041685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0041761Z return func(*args, **kwargs) 2025-12-04T08:55:17.0042101Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T08:55:17.0042228Z layer_output = apply_chunking_to_forward( 2025-12-04T08:55:17.0042523Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:55:17.0042616Z return forward_fn(*input_tensors) 2025-12-04T08:55:17.0042975Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-12-04T08:55:17.0043098Z intermediate_output = self.intermediate(ln_output) 2025-12-04T08:55:17.0043421Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 392, in forward 2025-12-04T08:55:17.0043546Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:55:17.0043796Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T08:55:17.0043899Z return self.act(input) 2025-12-04T08:55:17.0043905Z 2025-12-04T08:55:17.0044027Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:17.0044266Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:17.0044338Z res = mod(**inputs) 2025-12-04T08:55:17.0044678Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:17.0044752Z outputs = self.bert( 2025-12-04T08:55:17.0045081Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:17.0045170Z encoder_outputs = self.encoder( 2025-12-04T08:55:17.0045499Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:17.0045586Z layer_outputs = layer_module( 2025-12-04T08:55:17.0045840Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:17.0045928Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:17.0046210Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0046287Z return func(*args, **kwargs) 2025-12-04T08:55:17.0046625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T08:55:17.0046726Z layer_output = apply_chunking_to_forward( 2025-12-04T08:55:17.0047026Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:55:17.0047116Z return forward_fn(*input_tensors) 2025-12-04T08:55:17.0047483Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-12-04T08:55:17.0047633Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T08:55:17.0047970Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 404, in forward 2025-12-04T08:55:17.0048062Z hidden_states = self.dense(hidden_states) 2025-12-04T08:55:17.0048066Z 2025-12-04T08:55:17.0048188Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:17.0048438Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:17.0048512Z res = mod(**inputs) 2025-12-04T08:55:17.0048849Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:17.0048925Z outputs = self.bert( 2025-12-04T08:55:17.0049280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:17.0049385Z encoder_outputs = self.encoder( 2025-12-04T08:55:17.0049706Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:17.0049792Z layer_outputs = layer_module( 2025-12-04T08:55:17.0050041Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:17.0050128Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:17.0050419Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0050495Z return func(*args, **kwargs) 2025-12-04T08:55:17.0050826Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T08:55:17.0050957Z layer_output = apply_chunking_to_forward( 2025-12-04T08:55:17.0051250Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:55:17.0051341Z return forward_fn(*input_tensors) 2025-12-04T08:55:17.0051696Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-12-04T08:55:17.0051851Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T08:55:17.0052178Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 406, in forward 2025-12-04T08:55:17.0052264Z return input_tensor + hidden_states 2025-12-04T08:55:17.0052268Z 2025-12-04T08:55:17.0052388Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:17.0052614Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:17.0052688Z res = mod(**inputs) 2025-12-04T08:55:17.0053031Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:17.0053104Z outputs = self.bert( 2025-12-04T08:55:17.0053434Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:17.0053514Z encoder_outputs = self.encoder( 2025-12-04T08:55:17.0053839Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:17.0053925Z layer_outputs = layer_module( 2025-12-04T08:55:17.0054175Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:17.0054270Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:17.0054545Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0054623Z return func(*args, **kwargs) 2025-12-04T08:55:17.0054954Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T08:55:17.0055046Z self_attention_outputs = self.attention( 2025-12-04T08:55:17.0055314Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0055422Z return func(*args, **kwargs) 2025-12-04T08:55:17.0055754Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T08:55:17.0055840Z self_outputs = self.self( 2025-12-04T08:55:17.0056118Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0056214Z return func(*args, **kwargs) 2025-12-04T08:55:17.0056565Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 221, in forward 2025-12-04T08:55:17.0056657Z query_layer = self.query(hidden_states) 2025-12-04T08:55:17.0056661Z 2025-12-04T08:55:17.0056781Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:17.0057001Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:17.0057072Z res = mod(**inputs) 2025-12-04T08:55:17.0057412Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:17.0057485Z outputs = self.bert( 2025-12-04T08:55:17.0057808Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:17.0057924Z encoder_outputs = self.encoder( 2025-12-04T08:55:17.0058250Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:17.0058338Z layer_outputs = layer_module( 2025-12-04T08:55:17.0058590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:17.0058676Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:17.0058959Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0059037Z return func(*args, **kwargs) 2025-12-04T08:55:17.0059363Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T08:55:17.0059465Z self_attention_outputs = self.attention( 2025-12-04T08:55:17.0059741Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0059826Z return func(*args, **kwargs) 2025-12-04T08:55:17.0060152Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T08:55:17.0060231Z self_outputs = self.self( 2025-12-04T08:55:17.0060512Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0060598Z return func(*args, **kwargs) 2025-12-04T08:55:17.0060925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 245, in forward 2025-12-04T08:55:17.0061013Z key_layer = self.key(current_states) 2025-12-04T08:55:17.0061017Z 2025-12-04T08:55:17.0061130Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:17.0061358Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:17.0061429Z res = mod(**inputs) 2025-12-04T08:55:17.0061759Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:17.0061839Z outputs = self.bert( 2025-12-04T08:55:17.0062161Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:17.0062249Z encoder_outputs = self.encoder( 2025-12-04T08:55:17.0062598Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:17.0062678Z layer_outputs = layer_module( 2025-12-04T08:55:17.0062933Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:17.0063021Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:17.0063316Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0063410Z return func(*args, **kwargs) 2025-12-04T08:55:17.0063737Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T08:55:17.0063836Z self_attention_outputs = self.attention( 2025-12-04T08:55:17.0064115Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0064188Z return func(*args, **kwargs) 2025-12-04T08:55:17.0064512Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T08:55:17.0064589Z self_outputs = self.self( 2025-12-04T08:55:17.0064896Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0064974Z return func(*args, **kwargs) 2025-12-04T08:55:17.0065301Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 249, in forward 2025-12-04T08:55:17.0065397Z value_layer = self.value(current_states) 2025-12-04T08:55:17.0065401Z 2025-12-04T08:55:17.0065492Z cudagraph partition due to non gpu ops 2025-12-04T08:55:17.0065588Z cudagraph partition due to non gpu ops 2025-12-04T08:55:17.0065703Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:17.0065927Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:17.0066005Z res = mod(**inputs) 2025-12-04T08:55:17.0066335Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:17.0066422Z outputs = self.bert( 2025-12-04T08:55:17.0066745Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:17.0066826Z encoder_outputs = self.encoder( 2025-12-04T08:55:17.0067148Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:17.0067227Z layer_outputs = layer_module( 2025-12-04T08:55:17.0067477Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:17.0067573Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:17.0067845Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0067920Z return func(*args, **kwargs) 2025-12-04T08:55:17.0068258Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T08:55:17.0068360Z self_attention_outputs = self.attention( 2025-12-04T08:55:17.0068630Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0068703Z return func(*args, **kwargs) 2025-12-04T08:55:17.0069015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 375, in forward 2025-12-04T08:55:17.0069161Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T08:55:17.0069496Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 322, in forward 2025-12-04T08:55:17.0069596Z hidden_states = self.dense(hidden_states) 2025-12-04T08:55:17.0069601Z 2025-12-04T08:55:17.0069717Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:17.0069958Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:17.0070059Z res = mod(**inputs) 2025-12-04T08:55:17.0070388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:17.0070461Z outputs = self.bert( 2025-12-04T08:55:17.0070792Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:17.0070873Z encoder_outputs = self.encoder( 2025-12-04T08:55:17.0071204Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:17.0071283Z layer_outputs = layer_module( 2025-12-04T08:55:17.0071534Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:17.0071647Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:17.0071920Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0072005Z return func(*args, **kwargs) 2025-12-04T08:55:17.0072328Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T08:55:17.0072422Z layer_output = apply_chunking_to_forward( 2025-12-04T08:55:17.0072727Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:55:17.0072812Z return forward_fn(*input_tensors) 2025-12-04T08:55:17.0073170Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-12-04T08:55:17.0073295Z intermediate_output = self.intermediate(ln_output) 2025-12-04T08:55:17.0073620Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 391, in forward 2025-12-04T08:55:17.0073722Z hidden_states = self.dense(hidden_states) 2025-12-04T08:55:17.0073726Z 2025-12-04T08:55:17.0073839Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:17.0074063Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:17.0074145Z res = mod(**inputs) 2025-12-04T08:55:17.0074472Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:17.0074553Z outputs = self.bert( 2025-12-04T08:55:17.0074889Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:17.0074972Z encoder_outputs = self.encoder( 2025-12-04T08:55:17.0075304Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:17.0075384Z layer_outputs = layer_module( 2025-12-04T08:55:17.0075631Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:17.0075725Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:17.0075996Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0076079Z return func(*args, **kwargs) 2025-12-04T08:55:17.0076425Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T08:55:17.0076520Z layer_output = apply_chunking_to_forward( 2025-12-04T08:55:17.0076828Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:55:17.0076943Z return forward_fn(*input_tensors) 2025-12-04T08:55:17.0077345Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-12-04T08:55:17.0077462Z intermediate_output = self.intermediate(ln_output) 2025-12-04T08:55:17.0077790Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 392, in forward 2025-12-04T08:55:17.0077922Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:55:17.0078166Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T08:55:17.0078508Z return self.act(input) 2025-12-04T08:55:17.0078533Z 2025-12-04T08:55:17.0078660Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:17.0078919Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:17.0079004Z res = mod(**inputs) 2025-12-04T08:55:17.0079335Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:17.0079411Z outputs = self.bert( 2025-12-04T08:55:17.0079746Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:17.0079827Z encoder_outputs = self.encoder( 2025-12-04T08:55:17.0080162Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:17.0080244Z layer_outputs = layer_module( 2025-12-04T08:55:17.0080493Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:17.0080593Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:17.0080867Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0080946Z return func(*args, **kwargs) 2025-12-04T08:55:17.0081280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T08:55:17.0081373Z layer_output = apply_chunking_to_forward( 2025-12-04T08:55:17.0081674Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:55:17.0081758Z return forward_fn(*input_tensors) 2025-12-04T08:55:17.0082115Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-12-04T08:55:17.0082272Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T08:55:17.0082603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 404, in forward 2025-12-04T08:55:17.0082703Z hidden_states = self.dense(hidden_states) 2025-12-04T08:55:17.0082707Z 2025-12-04T08:55:17.0082822Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:17.0083045Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:17.0083125Z res = mod(**inputs) 2025-12-04T08:55:17.0083475Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:17.0083572Z outputs = self.bert( 2025-12-04T08:55:17.0083908Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:17.0083990Z encoder_outputs = self.encoder( 2025-12-04T08:55:17.0084346Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:17.0084446Z layer_outputs = layer_module( 2025-12-04T08:55:17.0084698Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:17.0084792Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:17.0085066Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0085152Z return func(*args, **kwargs) 2025-12-04T08:55:17.0085479Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T08:55:17.0085571Z self_attention_outputs = self.attention( 2025-12-04T08:55:17.0085853Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0085949Z return func(*args, **kwargs) 2025-12-04T08:55:17.0086298Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T08:55:17.0086387Z self_outputs = self.self( 2025-12-04T08:55:17.0086674Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0086756Z return func(*args, **kwargs) 2025-12-04T08:55:17.0087094Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 221, in forward 2025-12-04T08:55:17.0087186Z query_layer = self.query(hidden_states) 2025-12-04T08:55:17.0087191Z 2025-12-04T08:55:17.0087312Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:17.0087569Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:17.0087649Z res = mod(**inputs) 2025-12-04T08:55:17.0087997Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:17.0088070Z outputs = self.bert( 2025-12-04T08:55:17.0088415Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:17.0088495Z encoder_outputs = self.encoder( 2025-12-04T08:55:17.0088829Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:17.0088916Z layer_outputs = layer_module( 2025-12-04T08:55:17.0089165Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:17.0089258Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:17.0089549Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0089627Z return func(*args, **kwargs) 2025-12-04T08:55:17.0089972Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T08:55:17.0090061Z self_attention_outputs = self.attention( 2025-12-04T08:55:17.0090363Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0090440Z return func(*args, **kwargs) 2025-12-04T08:55:17.0090802Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T08:55:17.0090889Z self_outputs = self.self( 2025-12-04T08:55:17.0091187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0091264Z return func(*args, **kwargs) 2025-12-04T08:55:17.0091617Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 245, in forward 2025-12-04T08:55:17.0091725Z key_layer = self.key(current_states) 2025-12-04T08:55:17.0091729Z 2025-12-04T08:55:17.0091850Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:17.0092074Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:17.0092146Z res = mod(**inputs) 2025-12-04T08:55:17.0092490Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:17.0092564Z outputs = self.bert( 2025-12-04T08:55:17.0092892Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:17.0092980Z encoder_outputs = self.encoder( 2025-12-04T08:55:17.0093328Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:17.0093416Z layer_outputs = layer_module( 2025-12-04T08:55:17.0093666Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:17.0093752Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:17.0094041Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0094116Z return func(*args, **kwargs) 2025-12-04T08:55:17.0094456Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T08:55:17.0094546Z self_attention_outputs = self.attention( 2025-12-04T08:55:17.0094818Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0094905Z return func(*args, **kwargs) 2025-12-04T08:55:17.0095234Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T08:55:17.0095313Z self_outputs = self.self( 2025-12-04T08:55:17.0095594Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0095670Z return func(*args, **kwargs) 2025-12-04T08:55:17.0096004Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 249, in forward 2025-12-04T08:55:17.0096091Z value_layer = self.value(current_states) 2025-12-04T08:55:17.0096094Z 2025-12-04T08:55:17.0096185Z cudagraph partition due to non gpu ops 2025-12-04T08:55:17.0096282Z cudagraph partition due to non gpu ops 2025-12-04T08:55:17.0096395Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:17.0096628Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:17.0096702Z res = mod(**inputs) 2025-12-04T08:55:17.0097040Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:17.0097121Z outputs = self.bert( 2025-12-04T08:55:17.0097447Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:17.0097529Z encoder_outputs = self.encoder( 2025-12-04T08:55:17.0097882Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:17.0097962Z layer_outputs = layer_module( 2025-12-04T08:55:17.0098218Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:17.0098306Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:17.0098596Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0098704Z return func(*args, **kwargs) 2025-12-04T08:55:17.0099028Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T08:55:17.0099118Z self_attention_outputs = self.attention( 2025-12-04T08:55:17.0099400Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0099475Z return func(*args, **kwargs) 2025-12-04T08:55:17.0099805Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 375, in forward 2025-12-04T08:55:17.0099947Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T08:55:17.0100298Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 322, in forward 2025-12-04T08:55:17.0100401Z hidden_states = self.dense(hidden_states) 2025-12-04T08:55:17.0100405Z 2025-12-04T08:55:17.0100518Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:17.0100748Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:17.0100820Z res = mod(**inputs) 2025-12-04T08:55:17.0101150Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:17.0101231Z outputs = self.bert( 2025-12-04T08:55:17.0101555Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:17.0101637Z encoder_outputs = self.encoder( 2025-12-04T08:55:17.0101974Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:17.0102055Z layer_outputs = layer_module( 2025-12-04T08:55:17.0102311Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:17.0102397Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:17.0102667Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0102753Z return func(*args, **kwargs) 2025-12-04T08:55:17.0103078Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T08:55:17.0103179Z layer_output = apply_chunking_to_forward( 2025-12-04T08:55:17.0103474Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:55:17.0103561Z return forward_fn(*input_tensors) 2025-12-04T08:55:17.0103934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-12-04T08:55:17.0104048Z intermediate_output = self.intermediate(ln_output) 2025-12-04T08:55:17.0104372Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 391, in forward 2025-12-04T08:55:17.0104472Z hidden_states = self.dense(hidden_states) 2025-12-04T08:55:17.0104476Z 2025-12-04T08:55:17.0104612Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:17.0104853Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:17.0104925Z res = mod(**inputs) 2025-12-04T08:55:17.0105263Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:17.0105372Z outputs = self.bert( 2025-12-04T08:55:17.0105714Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:17.0105805Z encoder_outputs = self.encoder( 2025-12-04T08:55:17.0106127Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:17.0106204Z layer_outputs = layer_module( 2025-12-04T08:55:17.0106461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:17.0106545Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:17.0106822Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0106926Z return func(*args, **kwargs) 2025-12-04T08:55:17.0107244Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T08:55:17.0107342Z layer_output = apply_chunking_to_forward( 2025-12-04T08:55:17.0107627Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:55:17.0107710Z return forward_fn(*input_tensors) 2025-12-04T08:55:17.0108067Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-12-04T08:55:17.0108179Z intermediate_output = self.intermediate(ln_output) 2025-12-04T08:55:17.0108503Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 392, in forward 2025-12-04T08:55:17.0108625Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:55:17.0108867Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T08:55:17.0108953Z return self.act(input) 2025-12-04T08:55:17.0108957Z 2025-12-04T08:55:17.0109067Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:17.0109292Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:17.0109362Z res = mod(**inputs) 2025-12-04T08:55:17.0109684Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:17.0109763Z outputs = self.bert( 2025-12-04T08:55:17.0110080Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:17.0110159Z encoder_outputs = self.encoder( 2025-12-04T08:55:17.0110481Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:17.0110560Z layer_outputs = layer_module( 2025-12-04T08:55:17.0110814Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:17.0110897Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:17.0111161Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0111245Z return func(*args, **kwargs) 2025-12-04T08:55:17.0111582Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T08:55:17.0111674Z layer_output = apply_chunking_to_forward( 2025-12-04T08:55:17.0111967Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:55:17.0112050Z return forward_fn(*input_tensors) 2025-12-04T08:55:17.0112427Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-12-04T08:55:17.0112593Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T08:55:17.0112911Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 404, in forward 2025-12-04T08:55:17.0113009Z hidden_states = self.dense(hidden_states) 2025-12-04T08:55:17.0113013Z 2025-12-04T08:55:17.0113124Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:17.0113352Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:17.0113423Z res = mod(**inputs) 2025-12-04T08:55:17.0113741Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:17.0113843Z outputs = self.bert( 2025-12-04T08:55:17.0114158Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:17.0114245Z encoder_outputs = self.encoder( 2025-12-04T08:55:17.0114559Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:17.0114636Z layer_outputs = layer_module( 2025-12-04T08:55:17.0114886Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:17.0114973Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:17.0115245Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0115331Z return func(*args, **kwargs) 2025-12-04T08:55:17.0115660Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T08:55:17.0115770Z layer_output = apply_chunking_to_forward( 2025-12-04T08:55:17.0116065Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:55:17.0116148Z return forward_fn(*input_tensors) 2025-12-04T08:55:17.0116508Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-12-04T08:55:17.0116655Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T08:55:17.0116987Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 406, in forward 2025-12-04T08:55:17.0117072Z return input_tensor + hidden_states 2025-12-04T08:55:17.0117076Z 2025-12-04T08:55:17.0117192Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:17.0117425Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:17.0117500Z res = mod(**inputs) 2025-12-04T08:55:17.0117829Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:17.0117912Z outputs = self.bert( 2025-12-04T08:55:17.0118318Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:17.0118418Z encoder_outputs = self.encoder( 2025-12-04T08:55:17.0118777Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:17.0118860Z layer_outputs = layer_module( 2025-12-04T08:55:17.0119121Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:17.0119229Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:17.0119539Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0119617Z return func(*args, **kwargs) 2025-12-04T08:55:17.0119956Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T08:55:17.0120055Z self_attention_outputs = self.attention( 2025-12-04T08:55:17.0120323Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0120398Z return func(*args, **kwargs) 2025-12-04T08:55:17.0121045Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T08:55:17.0121207Z self_outputs = self.self( 2025-12-04T08:55:17.0121484Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0121560Z return func(*args, **kwargs) 2025-12-04T08:55:17.0121877Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 221, in forward 2025-12-04T08:55:17.0121977Z query_layer = self.query(hidden_states) 2025-12-04T08:55:17.0121981Z 2025-12-04T08:55:17.0122093Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:17.0122321Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:17.0122391Z res = mod(**inputs) 2025-12-04T08:55:17.0122710Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:17.0122790Z outputs = self.bert( 2025-12-04T08:55:17.0123108Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:17.0123188Z encoder_outputs = self.encoder( 2025-12-04T08:55:17.0123512Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:17.0123588Z layer_outputs = layer_module( 2025-12-04T08:55:17.0123844Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:17.0123927Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:17.0124195Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0124273Z return func(*args, **kwargs) 2025-12-04T08:55:17.0124575Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T08:55:17.0124662Z self_attention_outputs = self.attention( 2025-12-04T08:55:17.0124934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0125009Z return func(*args, **kwargs) 2025-12-04T08:55:17.0125334Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T08:55:17.0125410Z self_outputs = self.self( 2025-12-04T08:55:17.0125673Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0125811Z return func(*args, **kwargs) 2025-12-04T08:55:17.0126126Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 245, in forward 2025-12-04T08:55:17.0126218Z key_layer = self.key(current_states) 2025-12-04T08:55:17.0126224Z 2025-12-04T08:55:17.0126336Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:17.0126582Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:17.0126702Z res = mod(**inputs) 2025-12-04T08:55:17.0127026Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:17.0127097Z outputs = self.bert( 2025-12-04T08:55:17.0127425Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:17.0127506Z encoder_outputs = self.encoder( 2025-12-04T08:55:17.0127835Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:17.0127910Z layer_outputs = layer_module( 2025-12-04T08:55:17.0128177Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:17.0128270Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:17.0128554Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0128634Z return func(*args, **kwargs) 2025-12-04T08:55:17.0128955Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T08:55:17.0129044Z self_attention_outputs = self.attention( 2025-12-04T08:55:17.0129318Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0129392Z return func(*args, **kwargs) 2025-12-04T08:55:17.0129710Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T08:55:17.0129798Z self_outputs = self.self( 2025-12-04T08:55:17.0130066Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0130150Z return func(*args, **kwargs) 2025-12-04T08:55:17.0130469Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 249, in forward 2025-12-04T08:55:17.0130554Z value_layer = self.value(current_states) 2025-12-04T08:55:17.0130558Z 2025-12-04T08:55:17.0130657Z cudagraph partition due to non gpu ops 2025-12-04T08:55:17.0130745Z cudagraph partition due to non gpu ops 2025-12-04T08:55:17.0130857Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:17.0131082Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:17.0131151Z res = mod(**inputs) 2025-12-04T08:55:17.0131488Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:17.0131562Z outputs = self.bert( 2025-12-04T08:55:17.0131881Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:17.0131969Z encoder_outputs = self.encoder( 2025-12-04T08:55:17.0132287Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:17.0132370Z layer_outputs = layer_module( 2025-12-04T08:55:17.0132652Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:17.0132738Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:17.0133010Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0133085Z return func(*args, **kwargs) 2025-12-04T08:55:17.0133416Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T08:55:17.0133532Z self_attention_outputs = self.attention( 2025-12-04T08:55:17.0133792Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0133872Z return func(*args, **kwargs) 2025-12-04T08:55:17.0134188Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 375, in forward 2025-12-04T08:55:17.0134329Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T08:55:17.0134652Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 322, in forward 2025-12-04T08:55:17.0134741Z hidden_states = self.dense(hidden_states) 2025-12-04T08:55:17.0134766Z 2025-12-04T08:55:17.0134884Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:17.0135102Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:17.0135173Z res = mod(**inputs) 2025-12-04T08:55:17.0135521Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:17.0135594Z outputs = self.bert( 2025-12-04T08:55:17.0135907Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:17.0135995Z encoder_outputs = self.encoder( 2025-12-04T08:55:17.0136306Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:17.0136388Z layer_outputs = layer_module( 2025-12-04T08:55:17.0136634Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:17.0136718Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:17.0136990Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0137064Z return func(*args, **kwargs) 2025-12-04T08:55:17.0137385Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T08:55:17.0137475Z layer_output = apply_chunking_to_forward( 2025-12-04T08:55:17.0137761Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:55:17.0137852Z return forward_fn(*input_tensors) 2025-12-04T08:55:17.0138199Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-12-04T08:55:17.0138314Z intermediate_output = self.intermediate(ln_output) 2025-12-04T08:55:17.0138638Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 391, in forward 2025-12-04T08:55:17.0138727Z hidden_states = self.dense(hidden_states) 2025-12-04T08:55:17.0138731Z 2025-12-04T08:55:17.0138847Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:17.0139059Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:17.0139128Z res = mod(**inputs) 2025-12-04T08:55:17.0139472Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:17.0139545Z outputs = self.bert( 2025-12-04T08:55:17.0139868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:17.0139948Z encoder_outputs = self.encoder( 2025-12-04T08:55:17.0140277Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:17.0140380Z layer_outputs = layer_module( 2025-12-04T08:55:17.0140625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:17.0140708Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:17.0140978Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0141054Z return func(*args, **kwargs) 2025-12-04T08:55:17.0141376Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T08:55:17.0141465Z layer_output = apply_chunking_to_forward( 2025-12-04T08:55:17.0141780Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:55:17.0141870Z return forward_fn(*input_tensors) 2025-12-04T08:55:17.0142213Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-12-04T08:55:17.0142331Z intermediate_output = self.intermediate(ln_output) 2025-12-04T08:55:17.0142657Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 392, in forward 2025-12-04T08:55:17.0142781Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:55:17.0143028Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T08:55:17.0143106Z return self.act(input) 2025-12-04T08:55:17.0143110Z 2025-12-04T08:55:17.0143220Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:17.0143451Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:17.0143523Z res = mod(**inputs) 2025-12-04T08:55:17.0143862Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:17.0143934Z outputs = self.bert( 2025-12-04T08:55:17.0144253Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:17.0144341Z encoder_outputs = self.encoder( 2025-12-04T08:55:17.0144655Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:17.0144738Z layer_outputs = layer_module( 2025-12-04T08:55:17.0144983Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:17.0145070Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:17.0145351Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0145430Z return func(*args, **kwargs) 2025-12-04T08:55:17.0145763Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T08:55:17.0145860Z layer_output = apply_chunking_to_forward( 2025-12-04T08:55:17.0146167Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:55:17.0146258Z return forward_fn(*input_tensors) 2025-12-04T08:55:17.0146614Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-12-04T08:55:17.0146758Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T08:55:17.0147152Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 404, in forward 2025-12-04T08:55:17.0147286Z hidden_states = self.dense(hidden_states) 2025-12-04T08:55:17.0147290Z 2025-12-04T08:55:17.0147408Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:17.0147626Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:17.0147695Z res = mod(**inputs) 2025-12-04T08:55:17.0148025Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:17.0148098Z outputs = self.bert( 2025-12-04T08:55:17.0148412Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:17.0148519Z encoder_outputs = self.encoder( 2025-12-04T08:55:17.0148833Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:17.0148921Z layer_outputs = layer_module( 2025-12-04T08:55:17.0149165Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:17.0149251Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:17.0149526Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0149601Z return func(*args, **kwargs) 2025-12-04T08:55:17.0149924Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T08:55:17.0150015Z self_attention_outputs = self.attention( 2025-12-04T08:55:17.0150280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0150367Z return func(*args, **kwargs) 2025-12-04T08:55:17.0150681Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T08:55:17.0150760Z self_outputs = self.self( 2025-12-04T08:55:17.0151030Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0151102Z return func(*args, **kwargs) 2025-12-04T08:55:17.0151423Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 221, in forward 2025-12-04T08:55:17.0151511Z query_layer = self.query(hidden_states) 2025-12-04T08:55:17.0151515Z 2025-12-04T08:55:17.0151627Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:17.0151849Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:17.0151921Z res = mod(**inputs) 2025-12-04T08:55:17.0152245Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:17.0152317Z outputs = self.bert( 2025-12-04T08:55:17.0152631Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:17.0152716Z encoder_outputs = self.encoder( 2025-12-04T08:55:17.0153050Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:17.0153128Z layer_outputs = layer_module( 2025-12-04T08:55:17.0153385Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:17.0153470Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:17.0153769Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0153864Z return func(*args, **kwargs) 2025-12-04T08:55:17.0154194Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T08:55:17.0154291Z self_attention_outputs = self.attention( 2025-12-04T08:55:17.0154563Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0154637Z return func(*args, **kwargs) 2025-12-04T08:55:17.0154970Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T08:55:17.0155049Z self_outputs = self.self( 2025-12-04T08:55:17.0155325Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0155432Z return func(*args, **kwargs) 2025-12-04T08:55:17.0155761Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 245, in forward 2025-12-04T08:55:17.0155858Z key_layer = self.key(current_states) 2025-12-04T08:55:17.0155862Z 2025-12-04T08:55:17.0155975Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:17.0156203Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:17.0156276Z res = mod(**inputs) 2025-12-04T08:55:17.0156604Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:17.0156686Z outputs = self.bert( 2025-12-04T08:55:17.0157008Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:17.0157090Z encoder_outputs = self.encoder( 2025-12-04T08:55:17.0157419Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:17.0157499Z layer_outputs = layer_module( 2025-12-04T08:55:17.0157757Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:17.0157840Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:17.0158112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0158195Z return func(*args, **kwargs) 2025-12-04T08:55:17.0158677Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T08:55:17.0158791Z self_attention_outputs = self.attention( 2025-12-04T08:55:17.0159071Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0159150Z return func(*args, **kwargs) 2025-12-04T08:55:17.0159487Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T08:55:17.0159567Z self_outputs = self.self( 2025-12-04T08:55:17.0159838Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0159921Z return func(*args, **kwargs) 2025-12-04T08:55:17.0160279Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 249, in forward 2025-12-04T08:55:17.0160378Z value_layer = self.value(current_states) 2025-12-04T08:55:17.0160383Z 2025-12-04T08:55:17.0160475Z cudagraph partition due to non gpu ops 2025-12-04T08:55:17.0160564Z cudagraph partition due to non gpu ops 2025-12-04T08:55:17.0160692Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:17.0160941Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:17.0161031Z res = mod(**inputs) 2025-12-04T08:55:17.0161365Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:17.0161438Z outputs = self.bert( 2025-12-04T08:55:17.0161768Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:17.0161850Z encoder_outputs = self.encoder( 2025-12-04T08:55:17.0162173Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:17.0162260Z layer_outputs = layer_module( 2025-12-04T08:55:17.0162531Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:17.0162625Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:17.0162899Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0162975Z return func(*args, **kwargs) 2025-12-04T08:55:17.0163305Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T08:55:17.0163395Z self_attention_outputs = self.attention( 2025-12-04T08:55:17.0163667Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0163750Z return func(*args, **kwargs) 2025-12-04T08:55:17.0164076Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 375, in forward 2025-12-04T08:55:17.0164229Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T08:55:17.0164558Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 322, in forward 2025-12-04T08:55:17.0164652Z hidden_states = self.dense(hidden_states) 2025-12-04T08:55:17.0164656Z 2025-12-04T08:55:17.0164779Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:17.0165002Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:17.0165083Z res = mod(**inputs) 2025-12-04T08:55:17.0165413Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:17.0165487Z outputs = self.bert( 2025-12-04T08:55:17.0165821Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:17.0165903Z encoder_outputs = self.encoder( 2025-12-04T08:55:17.0166229Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:17.0166317Z layer_outputs = layer_module( 2025-12-04T08:55:17.0166570Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:17.0166663Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:17.0166936Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0167031Z return func(*args, **kwargs) 2025-12-04T08:55:17.0167362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T08:55:17.0167456Z layer_output = apply_chunking_to_forward( 2025-12-04T08:55:17.0167758Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:55:17.0167879Z return forward_fn(*input_tensors) 2025-12-04T08:55:17.0168244Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-12-04T08:55:17.0168358Z intermediate_output = self.intermediate(ln_output) 2025-12-04T08:55:17.0168664Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 391, in forward 2025-12-04T08:55:17.0168748Z hidden_states = self.dense(hidden_states) 2025-12-04T08:55:17.0168760Z 2025-12-04T08:55:17.0168870Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:17.0169087Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:17.0169161Z res = mod(**inputs) 2025-12-04T08:55:17.0169497Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:17.0169571Z outputs = self.bert( 2025-12-04T08:55:17.0169903Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:17.0169980Z encoder_outputs = self.encoder( 2025-12-04T08:55:17.0170302Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:17.0170378Z layer_outputs = layer_module( 2025-12-04T08:55:17.0170621Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:17.0170712Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:17.0170974Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0171045Z return func(*args, **kwargs) 2025-12-04T08:55:17.0171352Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T08:55:17.0171436Z layer_output = apply_chunking_to_forward( 2025-12-04T08:55:17.0171711Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:55:17.0171788Z return forward_fn(*input_tensors) 2025-12-04T08:55:17.0172115Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-12-04T08:55:17.0172227Z intermediate_output = self.intermediate(ln_output) 2025-12-04T08:55:17.0172522Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 392, in forward 2025-12-04T08:55:17.0172654Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:55:17.0172890Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T08:55:17.0172966Z return self.act(input) 2025-12-04T08:55:17.0172970Z 2025-12-04T08:55:17.0173088Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:17.0173305Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:17.0173375Z res = mod(**inputs) 2025-12-04T08:55:17.0173717Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:17.0173802Z outputs = self.bert( 2025-12-04T08:55:17.0174106Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:17.0174179Z encoder_outputs = self.encoder( 2025-12-04T08:55:17.0174495Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:17.0174594Z layer_outputs = layer_module( 2025-12-04T08:55:17.0174821Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:17.0174905Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:17.0175153Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0175222Z return func(*args, **kwargs) 2025-12-04T08:55:17.0175526Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T08:55:17.0175608Z layer_output = apply_chunking_to_forward( 2025-12-04T08:55:17.0175876Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:55:17.0175982Z return forward_fn(*input_tensors) 2025-12-04T08:55:17.0176313Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-12-04T08:55:17.0176457Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T08:55:17.0176757Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 404, in forward 2025-12-04T08:55:17.0176840Z hidden_states = self.dense(hidden_states) 2025-12-04T08:55:17.0176843Z 2025-12-04T08:55:17.0176956Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:17.0177160Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:17.0177230Z res = mod(**inputs) 2025-12-04T08:55:17.0177530Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:17.0177600Z outputs = self.bert( 2025-12-04T08:55:17.0177902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:17.0177976Z encoder_outputs = self.encoder( 2025-12-04T08:55:17.0178272Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:17.0178352Z layer_outputs = layer_module( 2025-12-04T08:55:17.0178581Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:17.0178669Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:17.0178919Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0178991Z return func(*args, **kwargs) 2025-12-04T08:55:17.0179296Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T08:55:17.0179381Z layer_output = apply_chunking_to_forward( 2025-12-04T08:55:17.0179655Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:55:17.0179732Z return forward_fn(*input_tensors) 2025-12-04T08:55:17.0180058Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-12-04T08:55:17.0180215Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T08:55:17.0180514Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 406, in forward 2025-12-04T08:55:17.0180604Z return input_tensor + hidden_states 2025-12-04T08:55:17.0180610Z 2025-12-04T08:55:17.0180719Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:17.0180985Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:17.0181064Z res = mod(**inputs) 2025-12-04T08:55:17.0181388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:17.0181461Z outputs = self.bert( 2025-12-04T08:55:17.0181783Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:17.0181865Z encoder_outputs = self.encoder( 2025-12-04T08:55:17.0182188Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:17.0182264Z layer_outputs = layer_module( 2025-12-04T08:55:17.0182539Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:17.0182630Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:17.0182882Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0182953Z return func(*args, **kwargs) 2025-12-04T08:55:17.0183258Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T08:55:17.0183342Z self_attention_outputs = self.attention( 2025-12-04T08:55:17.0183602Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0183673Z return func(*args, **kwargs) 2025-12-04T08:55:17.0183969Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T08:55:17.0184051Z self_outputs = self.self( 2025-12-04T08:55:17.0184300Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0184381Z return func(*args, **kwargs) 2025-12-04T08:55:17.0184676Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 221, in forward 2025-12-04T08:55:17.0184760Z query_layer = self.query(hidden_states) 2025-12-04T08:55:17.0184764Z 2025-12-04T08:55:17.0184877Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:17.0185083Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:17.0185149Z res = mod(**inputs) 2025-12-04T08:55:17.0185471Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:17.0185542Z outputs = self.bert( 2025-12-04T08:55:17.0185866Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:17.0185944Z encoder_outputs = self.encoder( 2025-12-04T08:55:17.0186261Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:17.0186341Z layer_outputs = layer_module( 2025-12-04T08:55:17.0186571Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:17.0186682Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:17.0186951Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0187024Z return func(*args, **kwargs) 2025-12-04T08:55:17.0187356Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T08:55:17.0187464Z self_attention_outputs = self.attention( 2025-12-04T08:55:17.0187732Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0187808Z return func(*args, **kwargs) 2025-12-04T08:55:17.0188119Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T08:55:17.0188203Z self_outputs = self.self( 2025-12-04T08:55:17.0188468Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0188541Z return func(*args, **kwargs) 2025-12-04T08:55:17.0188868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 245, in forward 2025-12-04T08:55:17.0188975Z key_layer = self.key(current_states) 2025-12-04T08:55:17.0188979Z 2025-12-04T08:55:17.0189097Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:17.0189316Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:17.0189386Z res = mod(**inputs) 2025-12-04T08:55:17.0189713Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:17.0189785Z outputs = self.bert( 2025-12-04T08:55:17.0190101Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:17.0190188Z encoder_outputs = self.encoder( 2025-12-04T08:55:17.0190505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:17.0190590Z layer_outputs = layer_module( 2025-12-04T08:55:17.0190834Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:17.0190919Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:17.0191192Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0191266Z return func(*args, **kwargs) 2025-12-04T08:55:17.0191591Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T08:55:17.0191678Z self_attention_outputs = self.attention( 2025-12-04T08:55:17.0191943Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0192026Z return func(*args, **kwargs) 2025-12-04T08:55:17.0192346Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T08:55:17.0192422Z self_outputs = self.self( 2025-12-04T08:55:17.0192702Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0192778Z return func(*args, **kwargs) 2025-12-04T08:55:17.0193107Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 249, in forward 2025-12-04T08:55:17.0193194Z value_layer = self.value(current_states) 2025-12-04T08:55:17.0193197Z 2025-12-04T08:55:17.0193287Z cudagraph partition due to non gpu ops 2025-12-04T08:55:17.0193404Z cudagraph partition due to non gpu ops 2025-12-04T08:55:17.0193520Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:17.0193741Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:17.0193817Z res = mod(**inputs) 2025-12-04T08:55:17.0194165Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:17.0194263Z outputs = self.bert( 2025-12-04T08:55:17.0194583Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:17.0194665Z encoder_outputs = self.encoder( 2025-12-04T08:55:17.0195000Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:17.0195077Z layer_outputs = layer_module( 2025-12-04T08:55:17.0195337Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:17.0195422Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:17.0195692Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0195799Z return func(*args, **kwargs) 2025-12-04T08:55:17.0196130Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T08:55:17.0196221Z self_attention_outputs = self.attention( 2025-12-04T08:55:17.0196506Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0196581Z return func(*args, **kwargs) 2025-12-04T08:55:17.0196923Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 375, in forward 2025-12-04T08:55:17.0197065Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T08:55:17.0197396Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 322, in forward 2025-12-04T08:55:17.0197499Z hidden_states = self.dense(hidden_states) 2025-12-04T08:55:17.0197503Z 2025-12-04T08:55:17.0197616Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:17.0197849Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:17.0197921Z res = mod(**inputs) 2025-12-04T08:55:17.0198322Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:17.0198413Z outputs = self.bert( 2025-12-04T08:55:17.0198740Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:17.0198820Z encoder_outputs = self.encoder( 2025-12-04T08:55:17.0199154Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:17.0199236Z layer_outputs = layer_module( 2025-12-04T08:55:17.0199496Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:17.0199583Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:17.0199855Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0199942Z return func(*args, **kwargs) 2025-12-04T08:55:17.0200266Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T08:55:17.0200368Z layer_output = apply_chunking_to_forward( 2025-12-04T08:55:17.0200697Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:55:17.0200783Z return forward_fn(*input_tensors) 2025-12-04T08:55:17.0201147Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-12-04T08:55:17.0201281Z intermediate_output = self.intermediate(ln_output) 2025-12-04T08:55:17.0201611Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 391, in forward 2025-12-04T08:55:17.0201709Z hidden_states = self.dense(hidden_states) 2025-12-04T08:55:17.0201713Z 2025-12-04T08:55:17.0201824Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:17.0202048Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:17.0202118Z res = mod(**inputs) 2025-12-04T08:55:17.0202437Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:17.0202518Z outputs = self.bert( 2025-12-04T08:55:17.0202832Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:17.0202946Z encoder_outputs = self.encoder( 2025-12-04T08:55:17.0203260Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:17.0203336Z layer_outputs = layer_module( 2025-12-04T08:55:17.0203585Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:17.0203668Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:17.0203931Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0204011Z return func(*args, **kwargs) 2025-12-04T08:55:17.0204328Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T08:55:17.0204426Z layer_output = apply_chunking_to_forward( 2025-12-04T08:55:17.0204724Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:55:17.0204806Z return forward_fn(*input_tensors) 2025-12-04T08:55:17.0205155Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-12-04T08:55:17.0205266Z intermediate_output = self.intermediate(ln_output) 2025-12-04T08:55:17.0205584Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 392, in forward 2025-12-04T08:55:17.0205705Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:55:17.0205940Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T08:55:17.0206026Z return self.act(input) 2025-12-04T08:55:17.0206032Z 2025-12-04T08:55:17.0206142Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:17.0206358Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:17.0206440Z res = mod(**inputs) 2025-12-04T08:55:17.0206763Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:17.0206843Z outputs = self.bert( 2025-12-04T08:55:17.0207166Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:17.0207268Z encoder_outputs = self.encoder( 2025-12-04T08:55:17.0207612Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:17.0207691Z layer_outputs = layer_module( 2025-12-04T08:55:17.0207948Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:17.0208065Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:17.0208370Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0208453Z return func(*args, **kwargs) 2025-12-04T08:55:17.0208769Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T08:55:17.0208857Z layer_output = apply_chunking_to_forward( 2025-12-04T08:55:17.0209151Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:55:17.0209231Z return forward_fn(*input_tensors) 2025-12-04T08:55:17.0209586Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-12-04T08:55:17.0209753Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T08:55:17.0210080Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 404, in forward 2025-12-04T08:55:17.0210176Z hidden_states = self.dense(hidden_states) 2025-12-04T08:55:17.0210180Z 2025-12-04T08:55:17.0210288Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:17.0210514Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:17.0210583Z res = mod(**inputs) 2025-12-04T08:55:17.0210901Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:17.0210980Z outputs = self.bert( 2025-12-04T08:55:17.0211296Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:17.0211377Z encoder_outputs = self.encoder( 2025-12-04T08:55:17.0211713Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:17.0211792Z layer_outputs = layer_module( 2025-12-04T08:55:17.0212050Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:17.0212143Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:17.0212410Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0212494Z return func(*args, **kwargs) 2025-12-04T08:55:17.0212809Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T08:55:17.0212905Z self_attention_outputs = self.attention( 2025-12-04T08:55:17.0213174Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0213250Z return func(*args, **kwargs) 2025-12-04T08:55:17.0213575Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T08:55:17.0213651Z self_outputs = self.self( 2025-12-04T08:55:17.0213915Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0213997Z return func(*args, **kwargs) 2025-12-04T08:55:17.0214340Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 221, in forward 2025-12-04T08:55:17.0214436Z query_layer = self.query(hidden_states) 2025-12-04T08:55:17.0214440Z 2025-12-04T08:55:17.0214550Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:17.0214768Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:17.0214862Z res = mod(**inputs) 2025-12-04T08:55:17.0215215Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:17.0215296Z outputs = self.bert( 2025-12-04T08:55:17.0215609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:17.0215688Z encoder_outputs = self.encoder( 2025-12-04T08:55:17.0216012Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:17.0216087Z layer_outputs = layer_module( 2025-12-04T08:55:17.0216328Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:17.0216443Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:17.0216708Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0216789Z return func(*args, **kwargs) 2025-12-04T08:55:17.0217108Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T08:55:17.0217195Z self_attention_outputs = self.attention( 2025-12-04T08:55:17.0217464Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0217540Z return func(*args, **kwargs) 2025-12-04T08:55:17.0217856Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T08:55:17.0217943Z self_outputs = self.self( 2025-12-04T08:55:17.0218211Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0218297Z return func(*args, **kwargs) 2025-12-04T08:55:17.0218620Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 245, in forward 2025-12-04T08:55:17.0218707Z key_layer = self.key(current_states) 2025-12-04T08:55:17.0218711Z 2025-12-04T08:55:17.0218831Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:17.0219052Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:17.0219133Z res = mod(**inputs) 2025-12-04T08:55:17.0219468Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:17.0219540Z outputs = self.bert( 2025-12-04T08:55:17.0219861Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:17.0219943Z encoder_outputs = self.encoder( 2025-12-04T08:55:17.0220260Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:17.0220343Z layer_outputs = layer_module( 2025-12-04T08:55:17.0220585Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:17.0220673Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:17.0221235Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0221315Z return func(*args, **kwargs) 2025-12-04T08:55:17.0221642Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T08:55:17.0221731Z self_attention_outputs = self.attention( 2025-12-04T08:55:17.0222037Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0222141Z return func(*args, **kwargs) 2025-12-04T08:55:17.0222456Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T08:55:17.0222541Z self_outputs = self.self( 2025-12-04T08:55:17.0222808Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0222882Z return func(*args, **kwargs) 2025-12-04T08:55:17.0223221Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 249, in forward 2025-12-04T08:55:17.0223310Z value_layer = self.value(current_states) 2025-12-04T08:55:17.0223314Z 2025-12-04T08:55:17.0223413Z cudagraph partition due to non gpu ops 2025-12-04T08:55:17.0223531Z cudagraph partition due to non gpu ops 2025-12-04T08:55:17.0223664Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:17.0223895Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:17.0223965Z res = mod(**inputs) 2025-12-04T08:55:17.0224294Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:17.0224374Z outputs = self.bert( 2025-12-04T08:55:17.0224695Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:17.0224782Z encoder_outputs = self.encoder( 2025-12-04T08:55:17.0225118Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:17.0225197Z layer_outputs = layer_module( 2025-12-04T08:55:17.0225489Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:17.0225577Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:17.0225866Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0225942Z return func(*args, **kwargs) 2025-12-04T08:55:17.0226275Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T08:55:17.0226372Z self_attention_outputs = self.attention( 2025-12-04T08:55:17.0226647Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0226721Z return func(*args, **kwargs) 2025-12-04T08:55:17.0227043Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 375, in forward 2025-12-04T08:55:17.0227187Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T08:55:17.0227508Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 322, in forward 2025-12-04T08:55:17.0227599Z hidden_states = self.dense(hidden_states) 2025-12-04T08:55:17.0227602Z 2025-12-04T08:55:17.0227712Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:17.0227949Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:17.0228018Z res = mod(**inputs) 2025-12-04T08:55:17.0228371Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:17.0228443Z outputs = self.bert( 2025-12-04T08:55:17.0228761Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:17.0228867Z encoder_outputs = self.encoder( 2025-12-04T08:55:17.0229196Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:17.0229273Z layer_outputs = layer_module( 2025-12-04T08:55:17.0229523Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:17.0229605Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:17.0229882Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0229956Z return func(*args, **kwargs) 2025-12-04T08:55:17.0230270Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T08:55:17.0230386Z layer_output = apply_chunking_to_forward( 2025-12-04T08:55:17.0230675Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:55:17.0230767Z return forward_fn(*input_tensors) 2025-12-04T08:55:17.0231114Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-12-04T08:55:17.0231226Z intermediate_output = self.intermediate(ln_output) 2025-12-04T08:55:17.0231548Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 391, in forward 2025-12-04T08:55:17.0231638Z hidden_states = self.dense(hidden_states) 2025-12-04T08:55:17.0231641Z 2025-12-04T08:55:17.0231761Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:17.0231985Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:17.0232059Z res = mod(**inputs) 2025-12-04T08:55:17.0232394Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:17.0232467Z outputs = self.bert( 2025-12-04T08:55:17.0232790Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:17.0232880Z encoder_outputs = self.encoder( 2025-12-04T08:55:17.0233207Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:17.0233292Z layer_outputs = layer_module( 2025-12-04T08:55:17.0233536Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:17.0233620Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:17.0233892Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0233969Z return func(*args, **kwargs) 2025-12-04T08:55:17.0234286Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T08:55:17.0234383Z layer_output = apply_chunking_to_forward( 2025-12-04T08:55:17.0234670Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:55:17.0234760Z return forward_fn(*input_tensors) 2025-12-04T08:55:17.0235140Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-12-04T08:55:17.0235254Z intermediate_output = self.intermediate(ln_output) 2025-12-04T08:55:17.0235576Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 392, in forward 2025-12-04T08:55:17.0235700Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:55:17.0235987Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T08:55:17.0236064Z return self.act(input) 2025-12-04T08:55:17.0236068Z 2025-12-04T08:55:17.0236179Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:17.0236402Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:17.0236473Z res = mod(**inputs) 2025-12-04T08:55:17.0236798Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:17.0236879Z outputs = self.bert( 2025-12-04T08:55:17.0237201Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:17.0237311Z encoder_outputs = self.encoder( 2025-12-04T08:55:17.0237636Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:17.0237716Z layer_outputs = layer_module( 2025-12-04T08:55:17.0237972Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:17.0238059Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:17.0238403Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0238492Z return func(*args, **kwargs) 2025-12-04T08:55:17.0238846Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T08:55:17.0238948Z layer_output = apply_chunking_to_forward( 2025-12-04T08:55:17.0239247Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:55:17.0239333Z return forward_fn(*input_tensors) 2025-12-04T08:55:17.0239699Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-12-04T08:55:17.0239847Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T08:55:17.0240181Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 404, in forward 2025-12-04T08:55:17.0240272Z hidden_states = self.dense(hidden_states) 2025-12-04T08:55:17.0240279Z 2025-12-04T08:55:17.0240393Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:17.0240631Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:17.0240704Z res = mod(**inputs) 2025-12-04T08:55:17.0241056Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:17.0241133Z outputs = self.bert( 2025-12-04T08:55:17.0241453Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:17.0241543Z encoder_outputs = self.encoder( 2025-12-04T08:55:17.0241865Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:17.0241945Z layer_outputs = layer_module( 2025-12-04T08:55:17.0242231Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:17.0242318Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:17.0242601Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0242680Z return func(*args, **kwargs) 2025-12-04T08:55:17.0243023Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T08:55:17.0243142Z layer_output = apply_chunking_to_forward( 2025-12-04T08:55:17.0243436Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:55:17.0243527Z return forward_fn(*input_tensors) 2025-12-04T08:55:17.0243884Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-12-04T08:55:17.0244034Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T08:55:17.0244365Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 406, in forward 2025-12-04T08:55:17.0244471Z return input_tensor + hidden_states 2025-12-04T08:55:17.0244475Z 2025-12-04T08:55:17.0244597Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:17.0244820Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:17.0244891Z res = mod(**inputs) 2025-12-04T08:55:17.0245222Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:17.0245296Z outputs = self.bert( 2025-12-04T08:55:17.0245623Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:17.0245712Z encoder_outputs = self.encoder( 2025-12-04T08:55:17.0246033Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:17.0246118Z layer_outputs = layer_module( 2025-12-04T08:55:17.0246372Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:17.0246460Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:17.0246741Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0246818Z return func(*args, **kwargs) 2025-12-04T08:55:17.0247141Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T08:55:17.0247242Z self_attention_outputs = self.attention( 2025-12-04T08:55:17.0247512Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0247595Z return func(*args, **kwargs) 2025-12-04T08:55:17.0247919Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T08:55:17.0248001Z self_outputs = self.self( 2025-12-04T08:55:17.0248280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0248354Z return func(*args, **kwargs) 2025-12-04T08:55:17.0248686Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 221, in forward 2025-12-04T08:55:17.0248769Z query_layer = self.query(hidden_states) 2025-12-04T08:55:17.0248773Z 2025-12-04T08:55:17.0248875Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:17.0249107Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:17.0249174Z res = mod(**inputs) 2025-12-04T08:55:17.0249478Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:17.0249553Z outputs = self.bert( 2025-12-04T08:55:17.0249869Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:17.0249971Z encoder_outputs = self.encoder( 2025-12-04T08:55:17.0250267Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:17.0250339Z layer_outputs = layer_module( 2025-12-04T08:55:17.0250578Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:17.0250658Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:17.0250914Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0250986Z return func(*args, **kwargs) 2025-12-04T08:55:17.0251299Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T08:55:17.0251394Z self_attention_outputs = self.attention( 2025-12-04T08:55:17.0251644Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0251716Z return func(*args, **kwargs) 2025-12-04T08:55:17.0252021Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T08:55:17.0252094Z self_outputs = self.self( 2025-12-04T08:55:17.0252351Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0252420Z return func(*args, **kwargs) 2025-12-04T08:55:17.0252717Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 245, in forward 2025-12-04T08:55:17.0252810Z key_layer = self.key(current_states) 2025-12-04T08:55:17.0252815Z 2025-12-04T08:55:17.0252923Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:17.0253134Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:17.0253201Z res = mod(**inputs) 2025-12-04T08:55:17.0253501Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:17.0253578Z outputs = self.bert( 2025-12-04T08:55:17.0253875Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:17.0253949Z encoder_outputs = self.encoder( 2025-12-04T08:55:17.0254253Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:17.0254328Z layer_outputs = layer_module( 2025-12-04T08:55:17.0254567Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:17.0254647Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:17.0254897Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0254974Z return func(*args, **kwargs) 2025-12-04T08:55:17.0255286Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T08:55:17.0255460Z self_attention_outputs = self.attention( 2025-12-04T08:55:17.0255712Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0255782Z return func(*args, **kwargs) 2025-12-04T08:55:17.0256085Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T08:55:17.0256202Z self_outputs = self.self( 2025-12-04T08:55:17.0256454Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0256533Z return func(*args, **kwargs) 2025-12-04T08:55:17.0256845Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 249, in forward 2025-12-04T08:55:17.0256935Z value_layer = self.value(current_states) 2025-12-04T08:55:17.0256938Z 2025-12-04T08:55:17.0257026Z cudagraph partition due to non gpu ops 2025-12-04T08:55:17.0257110Z cudagraph partition due to non gpu ops 2025-12-04T08:55:17.0257224Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:17.0257430Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:17.0257525Z res = mod(**inputs) 2025-12-04T08:55:17.0257838Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:17.0257909Z outputs = self.bert( 2025-12-04T08:55:17.0258215Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:17.0258290Z encoder_outputs = self.encoder( 2025-12-04T08:55:17.0258588Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:17.0258670Z layer_outputs = layer_module( 2025-12-04T08:55:17.0258900Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:17.0258986Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:17.0259235Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0259307Z return func(*args, **kwargs) 2025-12-04T08:55:17.0259614Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T08:55:17.0259701Z self_attention_outputs = self.attention( 2025-12-04T08:55:17.0259950Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0260027Z return func(*args, **kwargs) 2025-12-04T08:55:17.0260329Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 375, in forward 2025-12-04T08:55:17.0260469Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T08:55:17.0260775Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 322, in forward 2025-12-04T08:55:17.0260867Z hidden_states = self.dense(hidden_states) 2025-12-04T08:55:17.0260872Z 2025-12-04T08:55:17.0260991Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:17.0261209Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:17.0261285Z res = mod(**inputs) 2025-12-04T08:55:17.0261604Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:17.0261676Z outputs = self.bert( 2025-12-04T08:55:17.0262016Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:17.0262099Z encoder_outputs = self.encoder( 2025-12-04T08:55:17.0262417Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:17.0262503Z layer_outputs = layer_module( 2025-12-04T08:55:17.0262764Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:17.0262872Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:17.0263138Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0263211Z return func(*args, **kwargs) 2025-12-04T08:55:17.0263534Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T08:55:17.0263627Z layer_output = apply_chunking_to_forward( 2025-12-04T08:55:17.0263921Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:55:17.0264007Z return forward_fn(*input_tensors) 2025-12-04T08:55:17.0264382Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-12-04T08:55:17.0264504Z intermediate_output = self.intermediate(ln_output) 2025-12-04T08:55:17.0264820Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 391, in forward 2025-12-04T08:55:17.0264907Z hidden_states = self.dense(hidden_states) 2025-12-04T08:55:17.0264918Z 2025-12-04T08:55:17.0265028Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:17.0265246Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:17.0265322Z res = mod(**inputs) 2025-12-04T08:55:17.0265639Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:17.0265711Z outputs = self.bert( 2025-12-04T08:55:17.0266040Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:17.0266118Z encoder_outputs = self.encoder( 2025-12-04T08:55:17.0266443Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:17.0266520Z layer_outputs = layer_module( 2025-12-04T08:55:17.0266764Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:17.0266855Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:17.0267122Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0267196Z return func(*args, **kwargs) 2025-12-04T08:55:17.0267520Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T08:55:17.0267612Z layer_output = apply_chunking_to_forward( 2025-12-04T08:55:17.0267909Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:55:17.0267993Z return forward_fn(*input_tensors) 2025-12-04T08:55:17.0268340Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-12-04T08:55:17.0268460Z intermediate_output = self.intermediate(ln_output) 2025-12-04T08:55:17.0268798Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 392, in forward 2025-12-04T08:55:17.0268930Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:55:17.0269167Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T08:55:17.0269246Z return self.act(input) 2025-12-04T08:55:17.0269251Z 2025-12-04T08:55:17.0269390Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:17.0269627Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:17.0269698Z res = mod(**inputs) 2025-12-04T08:55:17.0270034Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:17.0270106Z outputs = self.bert( 2025-12-04T08:55:17.0270434Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:17.0270515Z encoder_outputs = self.encoder( 2025-12-04T08:55:17.0270832Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:17.0270916Z layer_outputs = layer_module( 2025-12-04T08:55:17.0271183Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:17.0271278Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:17.0271543Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0271618Z return func(*args, **kwargs) 2025-12-04T08:55:17.0271940Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T08:55:17.0272030Z layer_output = apply_chunking_to_forward( 2025-12-04T08:55:17.0272320Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:55:17.0272410Z return forward_fn(*input_tensors) 2025-12-04T08:55:17.0272762Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-12-04T08:55:17.0272918Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T08:55:17.0273238Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 404, in forward 2025-12-04T08:55:17.0273327Z hidden_states = self.dense(hidden_states) 2025-12-04T08:55:17.0273331Z 2025-12-04T08:55:17.0273448Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:17.0273666Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:17.0273742Z res = mod(**inputs) 2025-12-04T08:55:17.0274069Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:17.0274141Z outputs = self.bert( 2025-12-04T08:55:17.0274463Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:17.0274545Z encoder_outputs = self.encoder( 2025-12-04T08:55:17.0274862Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:17.0274946Z layer_outputs = layer_module( 2025-12-04T08:55:17.0275191Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:17.0275282Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:17.0275569Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0275646Z return func(*args, **kwargs) 2025-12-04T08:55:17.0275969Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T08:55:17.0276061Z self_attention_outputs = self.attention( 2025-12-04T08:55:17.0276348Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0276441Z return func(*args, **kwargs) 2025-12-04T08:55:17.0276759Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T08:55:17.0276845Z self_outputs = self.self( 2025-12-04T08:55:17.0277114Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0277187Z return func(*args, **kwargs) 2025-12-04T08:55:17.0277513Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 221, in forward 2025-12-04T08:55:17.0277602Z query_layer = self.query(hidden_states) 2025-12-04T08:55:17.0277606Z 2025-12-04T08:55:17.0277747Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:17.0277967Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:17.0278037Z res = mod(**inputs) 2025-12-04T08:55:17.0278440Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:17.0278518Z outputs = self.bert( 2025-12-04T08:55:17.0278844Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:17.0278924Z encoder_outputs = self.encoder( 2025-12-04T08:55:17.0279244Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:17.0279330Z layer_outputs = layer_module( 2025-12-04T08:55:17.0279574Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:17.0279662Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:17.0279940Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0280014Z return func(*args, **kwargs) 2025-12-04T08:55:17.0280340Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T08:55:17.0280429Z self_attention_outputs = self.attention( 2025-12-04T08:55:17.0280705Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0280789Z return func(*args, **kwargs) 2025-12-04T08:55:17.0281109Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T08:55:17.0281185Z self_outputs = self.self( 2025-12-04T08:55:17.0281461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0281536Z return func(*args, **kwargs) 2025-12-04T08:55:17.0281859Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 245, in forward 2025-12-04T08:55:17.0281942Z key_layer = self.key(current_states) 2025-12-04T08:55:17.0281946Z 2025-12-04T08:55:17.0282055Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:17.0282282Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:17.0282373Z res = mod(**inputs) 2025-12-04T08:55:17.0282700Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:17.0282771Z outputs = self.bert( 2025-12-04T08:55:17.0283089Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:17.0283224Z encoder_outputs = self.encoder( 2025-12-04T08:55:17.0283539Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:17.0283615Z layer_outputs = layer_module( 2025-12-04T08:55:17.0283865Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:17.0283948Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:17.0284220Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0284293Z return func(*args, **kwargs) 2025-12-04T08:55:17.0284607Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T08:55:17.0284727Z self_attention_outputs = self.attention( 2025-12-04T08:55:17.0284996Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0285079Z return func(*args, **kwargs) 2025-12-04T08:55:17.0285399Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T08:55:17.0285473Z self_outputs = self.self( 2025-12-04T08:55:17.0285744Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0285819Z return func(*args, **kwargs) 2025-12-04T08:55:17.0286135Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 249, in forward 2025-12-04T08:55:17.0286230Z value_layer = self.value(current_states) 2025-12-04T08:55:17.0286235Z 2025-12-04T08:55:17.0286323Z cudagraph partition due to non gpu ops 2025-12-04T08:55:17.0286419Z cudagraph partition due to non gpu ops 2025-12-04T08:55:17.0286532Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:17.0286750Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:17.0286830Z res = mod(**inputs) 2025-12-04T08:55:17.0287160Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:17.0287232Z outputs = self.bert( 2025-12-04T08:55:17.0287555Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:17.0287637Z encoder_outputs = self.encoder( 2025-12-04T08:55:17.0287973Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:17.0288054Z layer_outputs = layer_module( 2025-12-04T08:55:17.0288306Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:17.0288403Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:17.0288675Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0288759Z return func(*args, **kwargs) 2025-12-04T08:55:17.0289083Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T08:55:17.0289196Z self_attention_outputs = self.attention( 2025-12-04T08:55:17.0289478Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0289565Z return func(*args, **kwargs) 2025-12-04T08:55:17.0289886Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 375, in forward 2025-12-04T08:55:17.0290054Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T08:55:17.0290393Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 322, in forward 2025-12-04T08:55:17.0290491Z hidden_states = self.dense(hidden_states) 2025-12-04T08:55:17.0290495Z 2025-12-04T08:55:17.0290609Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:17.0290829Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:17.0290909Z res = mod(**inputs) 2025-12-04T08:55:17.0291237Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:17.0291316Z outputs = self.bert( 2025-12-04T08:55:17.0291659Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:17.0291744Z encoder_outputs = self.encoder( 2025-12-04T08:55:17.0292076Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:17.0292155Z layer_outputs = layer_module( 2025-12-04T08:55:17.0292406Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:17.0292498Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:17.0292773Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0292857Z return func(*args, **kwargs) 2025-12-04T08:55:17.0293183Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T08:55:17.0293278Z layer_output = apply_chunking_to_forward( 2025-12-04T08:55:17.0293584Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:55:17.0293671Z return forward_fn(*input_tensors) 2025-12-04T08:55:17.0294039Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-12-04T08:55:17.0294155Z intermediate_output = self.intermediate(ln_output) 2025-12-04T08:55:17.0294482Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 391, in forward 2025-12-04T08:55:17.0294582Z hidden_states = self.dense(hidden_states) 2025-12-04T08:55:17.0294586Z 2025-12-04T08:55:17.0294697Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:17.0294918Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:17.0295000Z res = mod(**inputs) 2025-12-04T08:55:17.0295337Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:17.0295419Z outputs = self.bert( 2025-12-04T08:55:17.0295743Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:17.0295824Z encoder_outputs = self.encoder( 2025-12-04T08:55:17.0296180Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:17.0296259Z layer_outputs = layer_module( 2025-12-04T08:55:17.0296515Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:17.0296603Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:17.0296919Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0297024Z return func(*args, **kwargs) 2025-12-04T08:55:17.0297351Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T08:55:17.0297444Z layer_output = apply_chunking_to_forward( 2025-12-04T08:55:17.0297747Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:55:17.0297831Z return forward_fn(*input_tensors) 2025-12-04T08:55:17.0298200Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-12-04T08:55:17.0298315Z intermediate_output = self.intermediate(ln_output) 2025-12-04T08:55:17.0298645Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 392, in forward 2025-12-04T08:55:17.0298805Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:55:17.0299049Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T08:55:17.0299133Z return self.act(input) 2025-12-04T08:55:17.0299137Z 2025-12-04T08:55:17.0299249Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:17.0299473Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:17.0299551Z res = mod(**inputs) 2025-12-04T08:55:17.0299893Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:17.0299966Z outputs = self.bert( 2025-12-04T08:55:17.0300297Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:17.0300381Z encoder_outputs = self.encoder( 2025-12-04T08:55:17.0300715Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:17.0300793Z layer_outputs = layer_module( 2025-12-04T08:55:17.0301043Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:17.0301137Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:17.0301425Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0301508Z return func(*args, **kwargs) 2025-12-04T08:55:17.0301834Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T08:55:17.0301927Z layer_output = apply_chunking_to_forward( 2025-12-04T08:55:17.0302234Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:55:17.0302319Z return forward_fn(*input_tensors) 2025-12-04T08:55:17.0302679Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-12-04T08:55:17.0302837Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T08:55:17.0303164Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 404, in forward 2025-12-04T08:55:17.0303293Z hidden_states = self.dense(hidden_states) 2025-12-04T08:55:17.0303297Z 2025-12-04T08:55:17.0303411Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:17.0303633Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:17.0303714Z res = mod(**inputs) 2025-12-04T08:55:17.0304060Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:17.0304161Z outputs = self.bert( 2025-12-04T08:55:17.0304487Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:17.0304568Z encoder_outputs = self.encoder( 2025-12-04T08:55:17.0304903Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:17.0304985Z layer_outputs = layer_module( 2025-12-04T08:55:17.0305237Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:17.0305332Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:17.0305605Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0305715Z return func(*args, **kwargs) 2025-12-04T08:55:17.0306039Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T08:55:17.0306129Z layer_output = apply_chunking_to_forward( 2025-12-04T08:55:17.0306426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:55:17.0306510Z return forward_fn(*input_tensors) 2025-12-04T08:55:17.0306874Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-12-04T08:55:17.0307020Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T08:55:17.0307345Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 406, in forward 2025-12-04T08:55:17.0307439Z return input_tensor + hidden_states 2025-12-04T08:55:17.0307445Z 2025-12-04T08:55:17.0307558Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:17.0307787Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:17.0307858Z res = mod(**inputs) 2025-12-04T08:55:17.0308191Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:17.0308271Z outputs = self.bert( 2025-12-04T08:55:17.0308640Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:17.0308717Z encoder_outputs = self.encoder( 2025-12-04T08:55:17.0309038Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:17.0309116Z layer_outputs = layer_module( 2025-12-04T08:55:17.0309370Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:17.0309455Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:17.0309717Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0309800Z return func(*args, **kwargs) 2025-12-04T08:55:17.0310113Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T08:55:17.0310226Z self_attention_outputs = self.attention( 2025-12-04T08:55:17.0310504Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0310580Z return func(*args, **kwargs) 2025-12-04T08:55:17.0310933Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T08:55:17.0311029Z self_outputs = self.self( 2025-12-04T08:55:17.0311292Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0311375Z return func(*args, **kwargs) 2025-12-04T08:55:17.0311690Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 221, in forward 2025-12-04T08:55:17.0311785Z query_layer = self.query(hidden_states) 2025-12-04T08:55:17.0311789Z 2025-12-04T08:55:17.0311902Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:17.0312118Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:17.0312195Z res = mod(**inputs) 2025-12-04T08:55:17.0312531Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:17.0312625Z outputs = self.bert( 2025-12-04T08:55:17.0312951Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:17.0313029Z encoder_outputs = self.encoder( 2025-12-04T08:55:17.0313353Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:17.0313428Z layer_outputs = layer_module( 2025-12-04T08:55:17.0313676Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:17.0313766Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:17.0314035Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0314116Z return func(*args, **kwargs) 2025-12-04T08:55:17.0314438Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T08:55:17.0314528Z self_attention_outputs = self.attention( 2025-12-04T08:55:17.0314801Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0314874Z return func(*args, **kwargs) 2025-12-04T08:55:17.0315203Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T08:55:17.0315289Z self_outputs = self.self( 2025-12-04T08:55:17.0315566Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0315649Z return func(*args, **kwargs) 2025-12-04T08:55:17.0315977Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 245, in forward 2025-12-04T08:55:17.0316066Z key_layer = self.key(current_states) 2025-12-04T08:55:17.0316072Z 2025-12-04T08:55:17.0316194Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:17.0316416Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:17.0316494Z res = mod(**inputs) 2025-12-04T08:55:17.0316828Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:17.0316900Z outputs = self.bert( 2025-12-04T08:55:17.0317256Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:17.0317338Z encoder_outputs = self.encoder( 2025-12-04T08:55:17.0317667Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:17.0317771Z layer_outputs = layer_module( 2025-12-04T08:55:17.0318040Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:17.0318132Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:17.0318498Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0318581Z return func(*args, **kwargs) 2025-12-04T08:55:17.0318918Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T08:55:17.0319009Z self_attention_outputs = self.attention( 2025-12-04T08:55:17.0319282Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0319366Z return func(*args, **kwargs) 2025-12-04T08:55:17.0319715Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T08:55:17.0319803Z self_outputs = self.self( 2025-12-04T08:55:17.0320077Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0320152Z return func(*args, **kwargs) 2025-12-04T08:55:17.0320491Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 249, in forward 2025-12-04T08:55:17.0320580Z value_layer = self.value(current_states) 2025-12-04T08:55:17.0320586Z 2025-12-04T08:55:17.0320891Z cudagraph partition due to non gpu ops 2025-12-04T08:55:17.0320992Z cudagraph partition due to non gpu ops 2025-12-04T08:55:17.0321104Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:17.0321328Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:17.0321404Z res = mod(**inputs) 2025-12-04T08:55:17.0321726Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:17.0321810Z outputs = self.bert( 2025-12-04T08:55:17.0322126Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:17.0322214Z encoder_outputs = self.encoder( 2025-12-04T08:55:17.0322531Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:17.0322607Z layer_outputs = layer_module( 2025-12-04T08:55:17.0322861Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:17.0322946Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:17.0323213Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0323299Z return func(*args, **kwargs) 2025-12-04T08:55:17.0323611Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T08:55:17.0323709Z self_attention_outputs = self.attention( 2025-12-04T08:55:17.0323978Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0324055Z return func(*args, **kwargs) 2025-12-04T08:55:17.0324457Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 375, in forward 2025-12-04T08:55:17.0324604Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T08:55:17.0324940Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 322, in forward 2025-12-04T08:55:17.0325057Z hidden_states = self.dense(hidden_states) 2025-12-04T08:55:17.0325090Z 2025-12-04T08:55:17.0325203Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:17.0325447Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:17.0325521Z res = mod(**inputs) 2025-12-04T08:55:17.0325856Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:17.0325932Z outputs = self.bert( 2025-12-04T08:55:17.0326258Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:17.0326348Z encoder_outputs = self.encoder( 2025-12-04T08:55:17.0326670Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:17.0326788Z layer_outputs = layer_module( 2025-12-04T08:55:17.0327048Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:17.0327133Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:17.0327411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0327487Z return func(*args, **kwargs) 2025-12-04T08:55:17.0327811Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T08:55:17.0327913Z layer_output = apply_chunking_to_forward( 2025-12-04T08:55:17.0328206Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:55:17.0328293Z return forward_fn(*input_tensors) 2025-12-04T08:55:17.0328657Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-12-04T08:55:17.0328773Z intermediate_output = self.intermediate(ln_output) 2025-12-04T08:55:17.0329101Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 391, in forward 2025-12-04T08:55:17.0329192Z hidden_states = self.dense(hidden_states) 2025-12-04T08:55:17.0329196Z 2025-12-04T08:55:17.0329308Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:17.0329539Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:17.0329610Z res = mod(**inputs) 2025-12-04T08:55:17.0329942Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:17.0330019Z outputs = self.bert( 2025-12-04T08:55:17.0330340Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:17.0330432Z encoder_outputs = self.encoder( 2025-12-04T08:55:17.0330753Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:17.0330839Z layer_outputs = layer_module( 2025-12-04T08:55:17.0331088Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:17.0331195Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:17.0331477Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0331553Z return func(*args, **kwargs) 2025-12-04T08:55:17.0331875Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T08:55:17.0332011Z layer_output = apply_chunking_to_forward( 2025-12-04T08:55:17.0332325Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:55:17.0332417Z return forward_fn(*input_tensors) 2025-12-04T08:55:17.0332771Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-12-04T08:55:17.0332885Z intermediate_output = self.intermediate(ln_output) 2025-12-04T08:55:17.0333214Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 392, in forward 2025-12-04T08:55:17.0333338Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:55:17.0333584Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T08:55:17.0333689Z return self.act(input) 2025-12-04T08:55:17.0333693Z 2025-12-04T08:55:17.0333807Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:17.0334034Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:17.0334105Z res = mod(**inputs) 2025-12-04T08:55:17.0334428Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:17.0334510Z outputs = self.bert( 2025-12-04T08:55:17.0334832Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:17.0334920Z encoder_outputs = self.encoder( 2025-12-04T08:55:17.0335239Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:17.0335319Z layer_outputs = layer_module( 2025-12-04T08:55:17.0335578Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:17.0335666Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:17.0335944Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0336020Z return func(*args, **kwargs) 2025-12-04T08:55:17.0336342Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T08:55:17.0336442Z layer_output = apply_chunking_to_forward( 2025-12-04T08:55:17.0336732Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:55:17.0336815Z return forward_fn(*input_tensors) 2025-12-04T08:55:17.0337179Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-12-04T08:55:17.0337327Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T08:55:17.0337654Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 404, in forward 2025-12-04T08:55:17.0337745Z hidden_states = self.dense(hidden_states) 2025-12-04T08:55:17.0337749Z 2025-12-04T08:55:17.0337864Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:17.0338124Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:17.0338199Z res = mod(**inputs) 2025-12-04T08:55:17.0338536Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:17.0338611Z outputs = self.bert( 2025-12-04T08:55:17.0338963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:17.0339072Z encoder_outputs = self.encoder( 2025-12-04T08:55:17.0339393Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:17.0339469Z layer_outputs = layer_module( 2025-12-04T08:55:17.0339723Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:17.0339808Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:17.0340083Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0340158Z return func(*args, **kwargs) 2025-12-04T08:55:17.0340474Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T08:55:17.0340595Z self_attention_outputs = self.attention( 2025-12-04T08:55:17.0340859Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0340942Z return func(*args, **kwargs) 2025-12-04T08:55:17.0341257Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T08:55:17.0341334Z self_outputs = self.self( 2025-12-04T08:55:17.0341602Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0341678Z return func(*args, **kwargs) 2025-12-04T08:55:17.0341991Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 221, in forward 2025-12-04T08:55:17.0342087Z query_layer = self.query(hidden_states) 2025-12-04T08:55:17.0342093Z 2025-12-04T08:55:17.0342203Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:17.0342424Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:17.0342496Z res = mod(**inputs) 2025-12-04T08:55:17.0342814Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:17.0342893Z outputs = self.bert( 2025-12-04T08:55:17.0343206Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:17.0343286Z encoder_outputs = self.encoder( 2025-12-04T08:55:17.0343605Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:17.0343682Z layer_outputs = layer_module( 2025-12-04T08:55:17.0343932Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:17.0344016Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:17.0344280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0344361Z return func(*args, **kwargs) 2025-12-04T08:55:17.0344672Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T08:55:17.0344766Z self_attention_outputs = self.attention( 2025-12-04T08:55:17.0345049Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0345124Z return func(*args, **kwargs) 2025-12-04T08:55:17.0345447Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T08:55:17.0345524Z self_outputs = self.self( 2025-12-04T08:55:17.0345808Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0345907Z return func(*args, **kwargs) 2025-12-04T08:55:17.0346222Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 245, in forward 2025-12-04T08:55:17.0346311Z key_layer = self.key(current_states) 2025-12-04T08:55:17.0346315Z 2025-12-04T08:55:17.0346427Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:17.0346648Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:17.0346725Z res = mod(**inputs) 2025-12-04T08:55:17.0347048Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:17.0347145Z outputs = self.bert( 2025-12-04T08:55:17.0347458Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:17.0347538Z encoder_outputs = self.encoder( 2025-12-04T08:55:17.0347856Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:17.0347933Z layer_outputs = layer_module( 2025-12-04T08:55:17.0348174Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:17.0348266Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:17.0348528Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0348609Z return func(*args, **kwargs) 2025-12-04T08:55:17.0348920Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T08:55:17.0349013Z self_attention_outputs = self.attention( 2025-12-04T08:55:17.0349284Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0349356Z return func(*args, **kwargs) 2025-12-04T08:55:17.0349677Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T08:55:17.0349752Z self_outputs = self.self( 2025-12-04T08:55:17.0350017Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0350096Z return func(*args, **kwargs) 2025-12-04T08:55:17.0350407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 249, in forward 2025-12-04T08:55:17.0350493Z value_layer = self.value(current_states) 2025-12-04T08:55:17.0350497Z 2025-12-04T08:55:17.0350589Z cudagraph partition due to non gpu ops 2025-12-04T08:55:17.0350678Z cudagraph partition due to non gpu ops 2025-12-04T08:55:17.0350794Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:17.0351009Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:17.0351077Z res = mod(**inputs) 2025-12-04T08:55:17.0351401Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:17.0351472Z outputs = self.bert( 2025-12-04T08:55:17.0351800Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:17.0351889Z encoder_outputs = self.encoder( 2025-12-04T08:55:17.0352205Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:17.0352307Z layer_outputs = layer_module( 2025-12-04T08:55:17.0352570Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:17.0352654Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:17.0352924Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0352997Z return func(*args, **kwargs) 2025-12-04T08:55:17.0353322Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T08:55:17.0353411Z self_attention_outputs = self.attention( 2025-12-04T08:55:17.0353675Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0353755Z return func(*args, **kwargs) 2025-12-04T08:55:17.0354089Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 375, in forward 2025-12-04T08:55:17.0354229Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T08:55:17.0354551Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 322, in forward 2025-12-04T08:55:17.0354639Z hidden_states = self.dense(hidden_states) 2025-12-04T08:55:17.0354643Z 2025-12-04T08:55:17.0354760Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:17.0354974Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:17.0355045Z res = mod(**inputs) 2025-12-04T08:55:17.0355368Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:17.0355441Z outputs = self.bert( 2025-12-04T08:55:17.0355766Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:17.0355847Z encoder_outputs = self.encoder( 2025-12-04T08:55:17.0356169Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:17.0356257Z layer_outputs = layer_module( 2025-12-04T08:55:17.0356509Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:17.0356594Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:17.0356876Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0356952Z return func(*args, **kwargs) 2025-12-04T08:55:17.0357299Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T08:55:17.0357396Z layer_output = apply_chunking_to_forward( 2025-12-04T08:55:17.0357688Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:55:17.0357780Z return forward_fn(*input_tensors) 2025-12-04T08:55:17.0358135Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-12-04T08:55:17.0358327Z intermediate_output = self.intermediate(ln_output) 2025-12-04T08:55:17.0358686Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 391, in forward 2025-12-04T08:55:17.0358783Z hidden_states = self.dense(hidden_states) 2025-12-04T08:55:17.0358789Z 2025-12-04T08:55:17.0358913Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:17.0359138Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:17.0359229Z res = mod(**inputs) 2025-12-04T08:55:17.0359578Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:17.0359654Z outputs = self.bert( 2025-12-04T08:55:17.0359986Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:17.0360067Z encoder_outputs = self.encoder( 2025-12-04T08:55:17.0360398Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:17.0360489Z layer_outputs = layer_module( 2025-12-04T08:55:17.0360741Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:17.0360915Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:17.0361187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0361265Z return func(*args, **kwargs) 2025-12-04T08:55:17.0361595Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T08:55:17.0361686Z layer_output = apply_chunking_to_forward( 2025-12-04T08:55:17.0361981Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:55:17.0362073Z return forward_fn(*input_tensors) 2025-12-04T08:55:17.0362428Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-12-04T08:55:17.0362549Z intermediate_output = self.intermediate(ln_output) 2025-12-04T08:55:17.0362871Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 392, in forward 2025-12-04T08:55:17.0362998Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:55:17.0363247Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T08:55:17.0363325Z return self.act(input) 2025-12-04T08:55:17.0363329Z 2025-12-04T08:55:17.0363449Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:17.0363671Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:17.0363747Z res = mod(**inputs) 2025-12-04T08:55:17.0364080Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:17.0364154Z outputs = self.bert( 2025-12-04T08:55:17.0364475Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:17.0364570Z encoder_outputs = self.encoder( 2025-12-04T08:55:17.0364889Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:17.0364977Z layer_outputs = layer_module( 2025-12-04T08:55:17.0365223Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:17.0365310Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:17.0365621Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0365700Z return func(*args, **kwargs) 2025-12-04T08:55:17.0366033Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T08:55:17.0366127Z layer_output = apply_chunking_to_forward( 2025-12-04T08:55:17.0366435Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:55:17.0366553Z return forward_fn(*input_tensors) 2025-12-04T08:55:17.0366924Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-12-04T08:55:17.0367070Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T08:55:17.0367403Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 404, in forward 2025-12-04T08:55:17.0367493Z hidden_states = self.dense(hidden_states) 2025-12-04T08:55:17.0367497Z 2025-12-04T08:55:17.0367618Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:17.0367839Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:17.0367931Z res = mod(**inputs) 2025-12-04T08:55:17.0368275Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:17.0368351Z outputs = self.bert( 2025-12-04T08:55:17.0368685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:17.0368765Z encoder_outputs = self.encoder( 2025-12-04T08:55:17.0369094Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:17.0369180Z layer_outputs = layer_module( 2025-12-04T08:55:17.0369432Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:17.0369518Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:17.0369803Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0369882Z return func(*args, **kwargs) 2025-12-04T08:55:17.0370213Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T08:55:17.0370305Z layer_output = apply_chunking_to_forward( 2025-12-04T08:55:17.0370604Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:55:17.0370695Z return forward_fn(*input_tensors) 2025-12-04T08:55:17.0371052Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-12-04T08:55:17.0371207Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T08:55:17.0371536Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 406, in forward 2025-12-04T08:55:17.0371623Z return input_tensor + hidden_states 2025-12-04T08:55:17.0371627Z 2025-12-04T08:55:17.0371746Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:17.0371959Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:17.0372029Z res = mod(**inputs) 2025-12-04T08:55:17.0372365Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:17.0372438Z outputs = self.bert( 2025-12-04T08:55:17.0372779Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:17.0372860Z encoder_outputs = self.encoder( 2025-12-04T08:55:17.0373172Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:17.0373280Z layer_outputs = layer_module( 2025-12-04T08:55:17.0373539Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:17.0373631Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:17.0373895Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0373970Z return func(*args, **kwargs) 2025-12-04T08:55:17.0374297Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T08:55:17.0374388Z self_attention_outputs = self.attention( 2025-12-04T08:55:17.0374652Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0374752Z return func(*args, **kwargs) 2025-12-04T08:55:17.0375076Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T08:55:17.0375162Z self_outputs = self.self( 2025-12-04T08:55:17.0375426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0375499Z return func(*args, **kwargs) 2025-12-04T08:55:17.0375823Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 221, in forward 2025-12-04T08:55:17.0375912Z query_layer = self.query(hidden_states) 2025-12-04T08:55:17.0375917Z 2025-12-04T08:55:17.0376033Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:17.0376249Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:17.0376320Z res = mod(**inputs) 2025-12-04T08:55:17.0376651Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:17.0376725Z outputs = self.bert( 2025-12-04T08:55:17.0377040Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:17.0377126Z encoder_outputs = self.encoder( 2025-12-04T08:55:17.0377453Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:17.0377535Z layer_outputs = layer_module( 2025-12-04T08:55:17.0377783Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:17.0377865Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:17.0378140Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0378216Z return func(*args, **kwargs) 2025-12-04T08:55:17.0378551Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T08:55:17.0378640Z self_attention_outputs = self.attention( 2025-12-04T08:55:17.0378906Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0378985Z return func(*args, **kwargs) 2025-12-04T08:55:17.0379329Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T08:55:17.0379408Z self_outputs = self.self( 2025-12-04T08:55:17.0379679Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0379751Z return func(*args, **kwargs) 2025-12-04T08:55:17.0380089Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 245, in forward 2025-12-04T08:55:17.0380191Z key_layer = self.key(current_states) 2025-12-04T08:55:17.0380195Z 2025-12-04T08:55:17.0380544Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:17.0380777Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:17.0380845Z res = mod(**inputs) 2025-12-04T08:55:17.0381176Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:17.0381250Z outputs = self.bert( 2025-12-04T08:55:17.0381572Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:17.0381660Z encoder_outputs = self.encoder( 2025-12-04T08:55:17.0382022Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:17.0382103Z layer_outputs = layer_module( 2025-12-04T08:55:17.0382354Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:17.0382439Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:17.0382711Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0382785Z return func(*args, **kwargs) 2025-12-04T08:55:17.0383111Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T08:55:17.0383208Z self_attention_outputs = self.attention( 2025-12-04T08:55:17.0383468Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0383550Z return func(*args, **kwargs) 2025-12-04T08:55:17.0383864Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T08:55:17.0383940Z self_outputs = self.self( 2025-12-04T08:55:17.0384208Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0384281Z return func(*args, **kwargs) 2025-12-04T08:55:17.0384604Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 249, in forward 2025-12-04T08:55:17.0384697Z value_layer = self.value(current_states) 2025-12-04T08:55:17.0384701Z 2025-12-04T08:55:17.0384788Z cudagraph partition due to non gpu ops 2025-12-04T08:55:17.0384880Z cudagraph partition due to non gpu ops 2025-12-04T08:55:17.0384991Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:17.0385207Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:17.0385284Z res = mod(**inputs) 2025-12-04T08:55:17.0385609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:17.0385680Z outputs = self.bert( 2025-12-04T08:55:17.0386007Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:17.0386085Z encoder_outputs = self.encoder( 2025-12-04T08:55:17.0386427Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:17.0386505Z layer_outputs = layer_module( 2025-12-04T08:55:17.0386747Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:17.0386839Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:17.0387133Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0387239Z return func(*args, **kwargs) 2025-12-04T08:55:17.0387566Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T08:55:17.0387653Z self_attention_outputs = self.attention( 2025-12-04T08:55:17.0387933Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0388009Z return func(*args, **kwargs) 2025-12-04T08:55:17.0388349Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 375, in forward 2025-12-04T08:55:17.0388499Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T08:55:17.0388847Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 322, in forward 2025-12-04T08:55:17.0388948Z hidden_states = self.dense(hidden_states) 2025-12-04T08:55:17.0388952Z 2025-12-04T08:55:17.0389067Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:17.0389291Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:17.0389369Z res = mod(**inputs) 2025-12-04T08:55:17.0389685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:17.0389766Z outputs = self.bert( 2025-12-04T08:55:17.0390077Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:17.0390155Z encoder_outputs = self.encoder( 2025-12-04T08:55:17.0390477Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:17.0390554Z layer_outputs = layer_module( 2025-12-04T08:55:17.0390794Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:17.0390885Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:17.0391144Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0391225Z return func(*args, **kwargs) 2025-12-04T08:55:17.0391545Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T08:55:17.0391636Z layer_output = apply_chunking_to_forward( 2025-12-04T08:55:17.0391925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:55:17.0392011Z return forward_fn(*input_tensors) 2025-12-04T08:55:17.0392362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-12-04T08:55:17.0392477Z intermediate_output = self.intermediate(ln_output) 2025-12-04T08:55:17.0392787Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 391, in forward 2025-12-04T08:55:17.0392885Z hidden_states = self.dense(hidden_states) 2025-12-04T08:55:17.0392889Z 2025-12-04T08:55:17.0393024Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:17.0393243Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:17.0393320Z res = mod(**inputs) 2025-12-04T08:55:17.0393648Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:17.0393728Z outputs = self.bert( 2025-12-04T08:55:17.0394067Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:17.0394165Z encoder_outputs = self.encoder( 2025-12-04T08:55:17.0394494Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:17.0394573Z layer_outputs = layer_module( 2025-12-04T08:55:17.0394832Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:17.0394918Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:17.0395203Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0395286Z return func(*args, **kwargs) 2025-12-04T08:55:17.0395627Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T08:55:17.0395720Z layer_output = apply_chunking_to_forward( 2025-12-04T08:55:17.0396020Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:55:17.0396105Z return forward_fn(*input_tensors) 2025-12-04T08:55:17.0396467Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-12-04T08:55:17.0396583Z intermediate_output = self.intermediate(ln_output) 2025-12-04T08:55:17.0396906Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 392, in forward 2025-12-04T08:55:17.0397040Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:55:17.0397284Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T08:55:17.0397373Z return self.act(input) 2025-12-04T08:55:17.0397379Z 2025-12-04T08:55:17.0397494Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:17.0397712Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:17.0397792Z res = mod(**inputs) 2025-12-04T08:55:17.0398119Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:17.0398193Z outputs = self.bert( 2025-12-04T08:55:17.0398623Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:17.0398711Z encoder_outputs = self.encoder( 2025-12-04T08:55:17.0399047Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:17.0399129Z layer_outputs = layer_module( 2025-12-04T08:55:17.0399385Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:17.0399480Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:17.0399755Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0399840Z return func(*args, **kwargs) 2025-12-04T08:55:17.0400189Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T08:55:17.0400286Z layer_output = apply_chunking_to_forward( 2025-12-04T08:55:17.0400588Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:55:17.0400674Z return forward_fn(*input_tensors) 2025-12-04T08:55:17.0401053Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-12-04T08:55:17.0401230Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T08:55:17.0401558Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 404, in forward 2025-12-04T08:55:17.0401659Z hidden_states = self.dense(hidden_states) 2025-12-04T08:55:17.0401663Z 2025-12-04T08:55:17.0401777Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:17.0402002Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:17.0402081Z res = mod(**inputs) 2025-12-04T08:55:17.0402409Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:17.0402515Z outputs = self.bert( 2025-12-04T08:55:17.0402848Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:17.0402930Z encoder_outputs = self.encoder( 2025-12-04T08:55:17.0403268Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:17.0403346Z layer_outputs = layer_module( 2025-12-04T08:55:17.0403600Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:17.0403695Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:17.0403974Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0404057Z return func(*args, **kwargs) 2025-12-04T08:55:17.0404388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T08:55:17.0404483Z self_attention_outputs = self.attention( 2025-12-04T08:55:17.0404767Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0404843Z return func(*args, **kwargs) 2025-12-04T08:55:17.0405180Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T08:55:17.0405256Z self_outputs = self.self( 2025-12-04T08:55:17.0405532Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0405614Z return func(*args, **kwargs) 2025-12-04T08:55:17.0405947Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 221, in forward 2025-12-04T08:55:17.0406039Z query_layer = self.query(hidden_states) 2025-12-04T08:55:17.0406051Z 2025-12-04T08:55:17.0406167Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:17.0406393Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:17.0406471Z res = mod(**inputs) 2025-12-04T08:55:17.0406808Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:17.0406882Z outputs = self.bert( 2025-12-04T08:55:17.0407243Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:17.0407326Z encoder_outputs = self.encoder( 2025-12-04T08:55:17.0407654Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:17.0407734Z layer_outputs = layer_module( 2025-12-04T08:55:17.0408009Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:17.0408122Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:17.0408405Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0408482Z return func(*args, **kwargs) 2025-12-04T08:55:17.0408811Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T08:55:17.0408903Z self_attention_outputs = self.attention( 2025-12-04T08:55:17.0409180Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0409255Z return func(*args, **kwargs) 2025-12-04T08:55:17.0409594Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T08:55:17.0409711Z self_outputs = self.self( 2025-12-04T08:55:17.0409986Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0410069Z return func(*args, **kwargs) 2025-12-04T08:55:17.0410392Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 245, in forward 2025-12-04T08:55:17.0410478Z key_layer = self.key(current_states) 2025-12-04T08:55:17.0410482Z 2025-12-04T08:55:17.0410605Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:17.0410832Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:17.0410905Z res = mod(**inputs) 2025-12-04T08:55:17.0411240Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:17.0411316Z outputs = self.bert( 2025-12-04T08:55:17.0411650Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:17.0411733Z encoder_outputs = self.encoder( 2025-12-04T08:55:17.0412056Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:17.0412143Z layer_outputs = layer_module( 2025-12-04T08:55:17.0412394Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:17.0412483Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:17.0412762Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0412837Z return func(*args, **kwargs) 2025-12-04T08:55:17.0413181Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T08:55:17.0413272Z self_attention_outputs = self.attention( 2025-12-04T08:55:17.0413536Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0413616Z return func(*args, **kwargs) 2025-12-04T08:55:17.0413941Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T08:55:17.0414023Z self_outputs = self.self( 2025-12-04T08:55:17.0414311Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0414385Z return func(*args, **kwargs) 2025-12-04T08:55:17.0414707Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 249, in forward 2025-12-04T08:55:17.0414795Z value_layer = self.value(current_states) 2025-12-04T08:55:17.0414799Z 2025-12-04T08:55:17.0414928Z cudagraph partition due to non gpu ops 2025-12-04T08:55:17.0415025Z cudagraph partition due to non gpu ops 2025-12-04T08:55:17.0415138Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:17.0415367Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:17.0415439Z res = mod(**inputs) 2025-12-04T08:55:17.0415781Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:17.0415863Z outputs = self.bert( 2025-12-04T08:55:17.0416182Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:17.0416267Z encoder_outputs = self.encoder( 2025-12-04T08:55:17.0416608Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:17.0416687Z layer_outputs = layer_module( 2025-12-04T08:55:17.0416937Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:17.0417020Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:17.0417286Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0417369Z return func(*args, **kwargs) 2025-12-04T08:55:17.0417694Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T08:55:17.0417791Z self_attention_outputs = self.attention( 2025-12-04T08:55:17.0418056Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0418130Z return func(*args, **kwargs) 2025-12-04T08:55:17.0418457Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 375, in forward 2025-12-04T08:55:17.0418598Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T08:55:17.0418912Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 322, in forward 2025-12-04T08:55:17.0419009Z hidden_states = self.dense(hidden_states) 2025-12-04T08:55:17.0419013Z 2025-12-04T08:55:17.0419122Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:17.0419346Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:17.0419416Z res = mod(**inputs) 2025-12-04T08:55:17.0419734Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:17.0419814Z outputs = self.bert( 2025-12-04T08:55:17.0420130Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:17.0420217Z encoder_outputs = self.encoder( 2025-12-04T08:55:17.0420527Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:17.0420603Z layer_outputs = layer_module( 2025-12-04T08:55:17.0421029Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:17.0421185Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:17.0421458Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0421542Z return func(*args, **kwargs) 2025-12-04T08:55:17.0421895Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T08:55:17.0422025Z layer_output = apply_chunking_to_forward( 2025-12-04T08:55:17.0422318Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:55:17.0422404Z return forward_fn(*input_tensors) 2025-12-04T08:55:17.0422767Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-12-04T08:55:17.0422882Z intermediate_output = self.intermediate(ln_output) 2025-12-04T08:55:17.0423213Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 391, in forward 2025-12-04T08:55:17.0423306Z hidden_states = self.dense(hidden_states) 2025-12-04T08:55:17.0423310Z 2025-12-04T08:55:17.0423454Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:17.0423686Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:17.0423761Z res = mod(**inputs) 2025-12-04T08:55:17.0424093Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:17.0424167Z outputs = self.bert( 2025-12-04T08:55:17.0424489Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:17.0424576Z encoder_outputs = self.encoder( 2025-12-04T08:55:17.0424916Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:17.0424996Z layer_outputs = layer_module( 2025-12-04T08:55:17.0425254Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:17.0425343Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:17.0425627Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0425705Z return func(*args, **kwargs) 2025-12-04T08:55:17.0426036Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T08:55:17.0426136Z layer_output = apply_chunking_to_forward( 2025-12-04T08:55:17.0426435Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:55:17.0426520Z return forward_fn(*input_tensors) 2025-12-04T08:55:17.0426886Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-12-04T08:55:17.0427003Z intermediate_output = self.intermediate(ln_output) 2025-12-04T08:55:17.0427339Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 392, in forward 2025-12-04T08:55:17.0427467Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:55:17.0427707Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T08:55:17.0427793Z return self.act(input) 2025-12-04T08:55:17.0427797Z 2025-12-04T08:55:17.0427909Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:17.0428165Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:17.0428239Z res = mod(**inputs) 2025-12-04T08:55:17.0428568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:17.0428652Z outputs = self.bert( 2025-12-04T08:55:17.0428991Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:17.0429096Z encoder_outputs = self.encoder( 2025-12-04T08:55:17.0429418Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:17.0429496Z layer_outputs = layer_module( 2025-12-04T08:55:17.0429751Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:17.0429837Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:17.0430109Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0430193Z return func(*args, **kwargs) 2025-12-04T08:55:17.0430514Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T08:55:17.0430637Z layer_output = apply_chunking_to_forward( 2025-12-04T08:55:17.0430929Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:55:17.0431011Z return forward_fn(*input_tensors) 2025-12-04T08:55:17.0431377Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-12-04T08:55:17.0431525Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T08:55:17.0431856Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 404, in forward 2025-12-04T08:55:17.0431946Z hidden_states = self.dense(hidden_states) 2025-12-04T08:55:17.0431950Z 2025-12-04T08:55:17.0432063Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:17.0432293Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:17.0432365Z res = mod(**inputs) 2025-12-04T08:55:17.0432690Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:17.0432770Z outputs = self.bert( 2025-12-04T08:55:17.0433095Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:17.0433182Z encoder_outputs = self.encoder( 2025-12-04T08:55:17.0433505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:17.0433583Z layer_outputs = layer_module( 2025-12-04T08:55:17.0433839Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:17.0433928Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:17.0434206Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0434285Z return func(*args, **kwargs) 2025-12-04T08:55:17.0434607Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T08:55:17.0434705Z layer_output = apply_chunking_to_forward( 2025-12-04T08:55:17.0434995Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:55:17.0435101Z return forward_fn(*input_tensors) 2025-12-04T08:55:17.0435468Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-12-04T08:55:17.0435617Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T08:55:17.0435967Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 406, in forward 2025-12-04T08:55:17.0436071Z return input_tensor + hidden_states 2025-12-04T08:55:17.0436075Z 2025-12-04T08:55:17.0436189Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:17.0436417Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:17.0436488Z res = mod(**inputs) 2025-12-04T08:55:17.0436820Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:17.0436894Z outputs = self.bert( 2025-12-04T08:55:17.0437217Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:17.0437306Z encoder_outputs = self.encoder( 2025-12-04T08:55:17.0437652Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:17.0437733Z layer_outputs = layer_module( 2025-12-04T08:55:17.0437990Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:17.0438075Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:17.0438414Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0438498Z return func(*args, **kwargs) 2025-12-04T08:55:17.0438823Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T08:55:17.0438922Z self_attention_outputs = self.attention( 2025-12-04T08:55:17.0439190Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0439275Z return func(*args, **kwargs) 2025-12-04T08:55:17.0439602Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T08:55:17.0439682Z self_outputs = self.self( 2025-12-04T08:55:17.0439961Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0440037Z return func(*args, **kwargs) 2025-12-04T08:55:17.0440362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 221, in forward 2025-12-04T08:55:17.0440463Z query_layer = self.query(hidden_states) 2025-12-04T08:55:17.0440468Z 2025-12-04T08:55:17.0440581Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:17.0440808Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:17.0440881Z res = mod(**inputs) 2025-12-04T08:55:17.0441209Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:17.0441292Z outputs = self.bert( 2025-12-04T08:55:17.0441616Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:17.0441697Z encoder_outputs = self.encoder( 2025-12-04T08:55:17.0442027Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:17.0442143Z layer_outputs = layer_module( 2025-12-04T08:55:17.0442402Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:17.0442488Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:17.0442760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0442864Z return func(*args, **kwargs) 2025-12-04T08:55:17.0443203Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T08:55:17.0443302Z self_attention_outputs = self.attention( 2025-12-04T08:55:17.0443569Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0443644Z return func(*args, **kwargs) 2025-12-04T08:55:17.0443976Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T08:55:17.0444054Z self_outputs = self.self( 2025-12-04T08:55:17.0444324Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0444435Z return func(*args, **kwargs) 2025-12-04T08:55:17.0444760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 245, in forward 2025-12-04T08:55:17.0444857Z key_layer = self.key(current_states) 2025-12-04T08:55:17.0444861Z 2025-12-04T08:55:17.0444976Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:17.0445211Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:17.0445291Z res = mod(**inputs) 2025-12-04T08:55:17.0445621Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:17.0445703Z outputs = self.bert( 2025-12-04T08:55:17.0446027Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:17.0446110Z encoder_outputs = self.encoder( 2025-12-04T08:55:17.0446443Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:17.0446523Z layer_outputs = layer_module( 2025-12-04T08:55:17.0446777Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:17.0446869Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:17.0447135Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0447216Z return func(*args, **kwargs) 2025-12-04T08:55:17.0447531Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T08:55:17.0447619Z self_attention_outputs = self.attention( 2025-12-04T08:55:17.0447892Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0447970Z return func(*args, **kwargs) 2025-12-04T08:55:17.0448296Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T08:55:17.0448371Z self_outputs = self.self( 2025-12-04T08:55:17.0448636Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0448714Z return func(*args, **kwargs) 2025-12-04T08:55:17.0449053Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 249, in forward 2025-12-04T08:55:17.0449140Z value_layer = self.value(current_states) 2025-12-04T08:55:17.0449144Z 2025-12-04T08:55:17.0449239Z cudagraph partition due to non gpu ops 2025-12-04T08:55:17.0449324Z cudagraph partition due to non gpu ops 2025-12-04T08:55:17.0449445Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:17.0449678Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:17.0449769Z res = mod(**inputs) 2025-12-04T08:55:17.0450095Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:17.0450167Z outputs = self.bert( 2025-12-04T08:55:17.0450485Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:17.0450571Z encoder_outputs = self.encoder( 2025-12-04T08:55:17.0450885Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:17.0450970Z layer_outputs = layer_module( 2025-12-04T08:55:17.0451213Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:17.0451316Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:17.0451593Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0451667Z return func(*args, **kwargs) 2025-12-04T08:55:17.0451989Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T08:55:17.0452078Z self_attention_outputs = self.attention( 2025-12-04T08:55:17.0452343Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0452426Z return func(*args, **kwargs) 2025-12-04T08:55:17.0452743Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 375, in forward 2025-12-04T08:55:17.0452883Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T08:55:17.0453209Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 322, in forward 2025-12-04T08:55:17.0453298Z hidden_states = self.dense(hidden_states) 2025-12-04T08:55:17.0453302Z 2025-12-04T08:55:17.0453419Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:17.0453634Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:17.0453704Z res = mod(**inputs) 2025-12-04T08:55:17.0454032Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:17.0454103Z outputs = self.bert( 2025-12-04T08:55:17.0454425Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:17.0454505Z encoder_outputs = self.encoder( 2025-12-04T08:55:17.0454822Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:17.0454910Z layer_outputs = layer_module( 2025-12-04T08:55:17.0455155Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:17.0455239Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:17.0455512Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0455586Z return func(*args, **kwargs) 2025-12-04T08:55:17.0455930Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T08:55:17.0456022Z layer_output = apply_chunking_to_forward( 2025-12-04T08:55:17.0456305Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:55:17.0456415Z return forward_fn(*input_tensors) 2025-12-04T08:55:17.0456777Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-12-04T08:55:17.0456895Z intermediate_output = self.intermediate(ln_output) 2025-12-04T08:55:17.0457207Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 391, in forward 2025-12-04T08:55:17.0457294Z hidden_states = self.dense(hidden_states) 2025-12-04T08:55:17.0457297Z 2025-12-04T08:55:17.0457416Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:17.0457629Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:17.0457698Z res = mod(**inputs) 2025-12-04T08:55:17.0458037Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:17.0458133Z outputs = self.bert( 2025-12-04T08:55:17.0458460Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:17.0458551Z encoder_outputs = self.encoder( 2025-12-04T08:55:17.0458861Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:17.0458945Z layer_outputs = layer_module( 2025-12-04T08:55:17.0459187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:17.0459278Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:17.0459540Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0459617Z return func(*args, **kwargs) 2025-12-04T08:55:17.0459959Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T08:55:17.0460052Z layer_output = apply_chunking_to_forward( 2025-12-04T08:55:17.0460344Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:55:17.0460445Z return forward_fn(*input_tensors) 2025-12-04T08:55:17.0460790Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-12-04T08:55:17.0460910Z intermediate_output = self.intermediate(ln_output) 2025-12-04T08:55:17.0461223Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 392, in forward 2025-12-04T08:55:17.0461345Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:55:17.0461594Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T08:55:17.0461672Z return self.act(input) 2025-12-04T08:55:17.0461675Z 2025-12-04T08:55:17.0461794Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:17.0462007Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:17.0462077Z res = mod(**inputs) 2025-12-04T08:55:17.0462400Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:17.0462491Z outputs = self.bert( 2025-12-04T08:55:17.0462806Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:17.0462895Z encoder_outputs = self.encoder( 2025-12-04T08:55:17.0463213Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:17.0463335Z layer_outputs = layer_module( 2025-12-04T08:55:17.0463579Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:17.0463665Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:17.0463938Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0464011Z return func(*args, **kwargs) 2025-12-04T08:55:17.0464334Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T08:55:17.0464423Z layer_output = apply_chunking_to_forward( 2025-12-04T08:55:17.0464710Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:55:17.0464830Z return forward_fn(*input_tensors) 2025-12-04T08:55:17.0465184Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-12-04T08:55:17.0465330Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T08:55:17.0465657Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 404, in forward 2025-12-04T08:55:17.0465745Z hidden_states = self.dense(hidden_states) 2025-12-04T08:55:17.0465749Z 2025-12-04T08:55:17.0465884Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:17.0466100Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:17.0466169Z res = mod(**inputs) 2025-12-04T08:55:17.0466499Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:17.0466573Z outputs = self.bert( 2025-12-04T08:55:17.0466900Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:17.0466979Z encoder_outputs = self.encoder( 2025-12-04T08:55:17.0467299Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:17.0467385Z layer_outputs = layer_module( 2025-12-04T08:55:17.0467634Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:17.0467719Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:17.0467995Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0468069Z return func(*args, **kwargs) 2025-12-04T08:55:17.0468400Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T08:55:17.0468493Z self_attention_outputs = self.attention( 2025-12-04T08:55:17.0468768Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0468852Z return func(*args, **kwargs) 2025-12-04T08:55:17.0469181Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T08:55:17.0469267Z self_outputs = self.self( 2025-12-04T08:55:17.0469564Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0469643Z return func(*args, **kwargs) 2025-12-04T08:55:17.0469990Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 221, in forward 2025-12-04T08:55:17.0470083Z query_layer = self.query(hidden_states) 2025-12-04T08:55:17.0470105Z 2025-12-04T08:55:17.0470278Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:17.0470511Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:17.0470582Z res = mod(**inputs) 2025-12-04T08:55:17.0470917Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:17.0470990Z outputs = self.bert( 2025-12-04T08:55:17.0471315Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:17.0471405Z encoder_outputs = self.encoder( 2025-12-04T08:55:17.0471727Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:17.0471834Z layer_outputs = layer_module( 2025-12-04T08:55:17.0472083Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:17.0472170Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:17.0472448Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0472523Z return func(*args, **kwargs) 2025-12-04T08:55:17.0472855Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T08:55:17.0472956Z self_attention_outputs = self.attention( 2025-12-04T08:55:17.0473228Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0473313Z return func(*args, **kwargs) 2025-12-04T08:55:17.0473638Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T08:55:17.0473720Z self_outputs = self.self( 2025-12-04T08:55:17.0473997Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0474070Z return func(*args, **kwargs) 2025-12-04T08:55:17.0474404Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 245, in forward 2025-12-04T08:55:17.0474498Z key_layer = self.key(current_states) 2025-12-04T08:55:17.0474503Z 2025-12-04T08:55:17.0474617Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:17.0474845Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:17.0474916Z res = mod(**inputs) 2025-12-04T08:55:17.0475241Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:17.0475327Z outputs = self.bert( 2025-12-04T08:55:17.0475652Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:17.0475738Z encoder_outputs = self.encoder( 2025-12-04T08:55:17.0476069Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:17.0476147Z layer_outputs = layer_module( 2025-12-04T08:55:17.0476420Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:17.0476507Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:17.0476777Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0476862Z return func(*args, **kwargs) 2025-12-04T08:55:17.0477208Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T08:55:17.0477356Z self_attention_outputs = self.attention( 2025-12-04T08:55:17.0477629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0477704Z return func(*args, **kwargs) 2025-12-04T08:55:17.0478063Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T08:55:17.0478144Z self_outputs = self.self( 2025-12-04T08:55:17.0478494Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0478575Z return func(*args, **kwargs) 2025-12-04T08:55:17.0478909Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 249, in forward 2025-12-04T08:55:17.0479038Z value_layer = self.value(current_states) 2025-12-04T08:55:17.0479045Z 2025-12-04T08:55:17.0479135Z cudagraph partition due to non gpu ops 2025-12-04T08:55:17.0479225Z cudagraph partition due to non gpu ops 2025-12-04T08:55:17.0479349Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:17.0479575Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:17.0479654Z res = mod(**inputs) 2025-12-04T08:55:17.0480000Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:17.0480075Z outputs = self.bert( 2025-12-04T08:55:17.0480414Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:17.0480497Z encoder_outputs = self.encoder( 2025-12-04T08:55:17.0480832Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:17.0480922Z layer_outputs = layer_module( 2025-12-04T08:55:17.0481179Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:17.0481274Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:17.0481553Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0481630Z return func(*args, **kwargs) 2025-12-04T08:55:17.0481973Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T08:55:17.0482067Z self_attention_outputs = self.attention( 2025-12-04T08:55:17.0482349Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0482428Z return func(*args, **kwargs) 2025-12-04T08:55:17.0482764Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 375, in forward 2025-12-04T08:55:17.0482914Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T08:55:17.0483248Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 322, in forward 2025-12-04T08:55:17.0483341Z hidden_states = self.dense(hidden_states) 2025-12-04T08:55:17.0483353Z 2025-12-04T08:55:17.0483493Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:17.0483723Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:17.0483803Z res = mod(**inputs) 2025-12-04T08:55:17.0484133Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:17.0484223Z outputs = self.bert( 2025-12-04T08:55:17.0484572Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:17.0484652Z encoder_outputs = self.encoder( 2025-12-04T08:55:17.0484991Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:17.0485068Z layer_outputs = layer_module( 2025-12-04T08:55:17.0485321Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:17.0485413Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:17.0485683Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0485780Z return func(*args, **kwargs) 2025-12-04T08:55:17.0486112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T08:55:17.0486205Z layer_output = apply_chunking_to_forward( 2025-12-04T08:55:17.0486504Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:55:17.0486588Z return forward_fn(*input_tensors) 2025-12-04T08:55:17.0486944Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-12-04T08:55:17.0487069Z intermediate_output = self.intermediate(ln_output) 2025-12-04T08:55:17.0487392Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 391, in forward 2025-12-04T08:55:17.0487490Z hidden_states = self.dense(hidden_states) 2025-12-04T08:55:17.0487496Z 2025-12-04T08:55:17.0487607Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:17.0487830Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:17.0487914Z res = mod(**inputs) 2025-12-04T08:55:17.0488240Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:17.0488320Z outputs = self.bert( 2025-12-04T08:55:17.0488642Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:17.0488722Z encoder_outputs = self.encoder( 2025-12-04T08:55:17.0489049Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:17.0489127Z layer_outputs = layer_module( 2025-12-04T08:55:17.0489378Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:17.0489473Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:17.0489745Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0489828Z return func(*args, **kwargs) 2025-12-04T08:55:17.0490150Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T08:55:17.0490240Z layer_output = apply_chunking_to_forward( 2025-12-04T08:55:17.0490560Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:55:17.0490645Z return forward_fn(*input_tensors) 2025-12-04T08:55:17.0490999Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-12-04T08:55:17.0491124Z intermediate_output = self.intermediate(ln_output) 2025-12-04T08:55:17.0491464Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 392, in forward 2025-12-04T08:55:17.0491624Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:55:17.0491861Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T08:55:17.0491941Z return self.act(input) 2025-12-04T08:55:17.0491945Z 2025-12-04T08:55:17.0492070Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:17.0492302Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:17.0492383Z res = mod(**inputs) 2025-12-04T08:55:17.0492712Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:17.0492807Z outputs = self.bert( 2025-12-04T08:55:17.0493138Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:17.0493220Z encoder_outputs = self.encoder( 2025-12-04T08:55:17.0493544Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:17.0493627Z layer_outputs = layer_module( 2025-12-04T08:55:17.0493875Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:17.0493969Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:17.0494243Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0494319Z return func(*args, **kwargs) 2025-12-04T08:55:17.0494655Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T08:55:17.0494746Z layer_output = apply_chunking_to_forward( 2025-12-04T08:55:17.0495034Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:55:17.0495115Z return forward_fn(*input_tensors) 2025-12-04T08:55:17.0495457Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-12-04T08:55:17.0495608Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T08:55:17.0495936Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 404, in forward 2025-12-04T08:55:17.0496030Z hidden_states = self.dense(hidden_states) 2025-12-04T08:55:17.0496034Z 2025-12-04T08:55:17.0496145Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:17.0496361Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:17.0496438Z res = mod(**inputs) 2025-12-04T08:55:17.0496755Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:17.0496827Z outputs = self.bert( 2025-12-04T08:55:17.0497152Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:17.0497233Z encoder_outputs = self.encoder( 2025-12-04T08:55:17.0497588Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:17.0497669Z layer_outputs = layer_module( 2025-12-04T08:55:17.0497920Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:17.0498019Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:17.0498343Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0498420Z return func(*args, **kwargs) 2025-12-04T08:55:17.0498743Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T08:55:17.0498830Z layer_output = apply_chunking_to_forward( 2025-12-04T08:55:17.0499131Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:55:17.0499214Z return forward_fn(*input_tensors) 2025-12-04T08:55:17.0499576Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-12-04T08:55:17.0499751Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T08:55:17.0500075Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 406, in forward 2025-12-04T08:55:17.0500168Z return input_tensor + hidden_states 2025-12-04T08:55:17.0500172Z 2025-12-04T08:55:17.0500286Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:17.0500506Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:17.0500586Z res = mod(**inputs) 2025-12-04T08:55:17.0500915Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:17.0500996Z outputs = self.bert( 2025-12-04T08:55:17.0501317Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:17.0501399Z encoder_outputs = self.encoder( 2025-12-04T08:55:17.0501729Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:17.0501809Z layer_outputs = layer_module( 2025-12-04T08:55:17.0502054Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:17.0502149Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:17.0502422Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0502519Z return func(*args, **kwargs) 2025-12-04T08:55:17.0502833Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T08:55:17.0502921Z self_attention_outputs = self.attention( 2025-12-04T08:55:17.0503223Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0503299Z return func(*args, **kwargs) 2025-12-04T08:55:17.0503621Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T08:55:17.0503698Z self_outputs = self.self( 2025-12-04T08:55:17.0503959Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0504040Z return func(*args, **kwargs) 2025-12-04T08:55:17.0504372Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 221, in forward 2025-12-04T08:55:17.0504461Z query_layer = self.query(hidden_states) 2025-12-04T08:55:17.0504465Z 2025-12-04T08:55:17.0504583Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:17.0504799Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:17.0504879Z res = mod(**inputs) 2025-12-04T08:55:17.0505229Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:17.0505321Z outputs = self.bert( 2025-12-04T08:55:17.0505659Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:17.0505739Z encoder_outputs = self.encoder( 2025-12-04T08:55:17.0506075Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:17.0506162Z layer_outputs = layer_module( 2025-12-04T08:55:17.0506419Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:17.0506512Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:17.0506822Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0506897Z return func(*args, **kwargs) 2025-12-04T08:55:17.0507223Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T08:55:17.0507312Z self_attention_outputs = self.attention( 2025-12-04T08:55:17.0507585Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0507659Z return func(*args, **kwargs) 2025-12-04T08:55:17.0507976Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T08:55:17.0508060Z self_outputs = self.self( 2025-12-04T08:55:17.0508323Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0508398Z return func(*args, **kwargs) 2025-12-04T08:55:17.0508725Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 245, in forward 2025-12-04T08:55:17.0508810Z key_layer = self.key(current_states) 2025-12-04T08:55:17.0508814Z 2025-12-04T08:55:17.0508929Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:17.0509149Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:17.0509217Z res = mod(**inputs) 2025-12-04T08:55:17.0509551Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:17.0509622Z outputs = self.bert( 2025-12-04T08:55:17.0509947Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:17.0510030Z encoder_outputs = self.encoder( 2025-12-04T08:55:17.0510357Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:17.0510446Z layer_outputs = layer_module( 2025-12-04T08:55:17.0510703Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:17.0510786Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:17.0511064Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0511160Z return func(*args, **kwargs) 2025-12-04T08:55:17.0511481Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T08:55:17.0511570Z self_attention_outputs = self.attention( 2025-12-04T08:55:17.0511852Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0511951Z return func(*args, **kwargs) 2025-12-04T08:55:17.0512272Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T08:55:17.0512354Z self_outputs = self.self( 2025-12-04T08:55:17.0512628Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0512703Z return func(*args, **kwargs) 2025-12-04T08:55:17.0513040Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 249, in forward 2025-12-04T08:55:17.0513129Z value_layer = self.value(current_states) 2025-12-04T08:55:17.0513133Z 2025-12-04T08:55:17.0513223Z cudagraph partition due to non gpu ops 2025-12-04T08:55:17.0513341Z cudagraph partition due to non gpu ops 2025-12-04T08:55:17.0513454Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:17.0513683Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:17.0513758Z res = mod(**inputs) 2025-12-04T08:55:17.0514086Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:17.0514167Z outputs = self.bert( 2025-12-04T08:55:17.0514492Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:17.0514576Z encoder_outputs = self.encoder( 2025-12-04T08:55:17.0514907Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:17.0514984Z layer_outputs = layer_module( 2025-12-04T08:55:17.0515245Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:17.0515332Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:17.0515608Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0515694Z return func(*args, **kwargs) 2025-12-04T08:55:17.0516023Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T08:55:17.0516121Z self_attention_outputs = self.attention( 2025-12-04T08:55:17.0516393Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0516468Z return func(*args, **kwargs) 2025-12-04T08:55:17.0516801Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 375, in forward 2025-12-04T08:55:17.0516944Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T08:55:17.0517272Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 322, in forward 2025-12-04T08:55:17.0517375Z hidden_states = self.dense(hidden_states) 2025-12-04T08:55:17.0517379Z 2025-12-04T08:55:17.0517490Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:17.0517719Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:17.0517789Z res = mod(**inputs) 2025-12-04T08:55:17.0518156Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:17.0518303Z outputs = self.bert( 2025-12-04T08:55:17.0518636Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:17.0518729Z encoder_outputs = self.encoder( 2025-12-04T08:55:17.0519083Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:17.0519182Z layer_outputs = layer_module( 2025-12-04T08:55:17.0519446Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:17.0519534Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:17.0519806Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0519895Z return func(*args, **kwargs) 2025-12-04T08:55:17.0520221Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T08:55:17.0520324Z layer_output = apply_chunking_to_forward( 2025-12-04T08:55:17.0520646Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:55:17.0520904Z return forward_fn(*input_tensors) 2025-12-04T08:55:17.0521276Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-12-04T08:55:17.0521393Z intermediate_output = self.intermediate(ln_output) 2025-12-04T08:55:17.0521725Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 391, in forward 2025-12-04T08:55:17.0521820Z hidden_states = self.dense(hidden_states) 2025-12-04T08:55:17.0521824Z 2025-12-04T08:55:17.0521938Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:17.0522169Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:17.0522242Z res = mod(**inputs) 2025-12-04T08:55:17.0522574Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:17.0522659Z outputs = self.bert( 2025-12-04T08:55:17.0522981Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:17.0523072Z encoder_outputs = self.encoder( 2025-12-04T08:55:17.0523397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:17.0523476Z layer_outputs = layer_module( 2025-12-04T08:55:17.0523736Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:17.0523825Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:17.0524106Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0524184Z return func(*args, **kwargs) 2025-12-04T08:55:17.0524512Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T08:55:17.0524615Z layer_output = apply_chunking_to_forward( 2025-12-04T08:55:17.0524910Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:55:17.0524993Z return forward_fn(*input_tensors) 2025-12-04T08:55:17.0525411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-12-04T08:55:17.0525529Z intermediate_output = self.intermediate(ln_output) 2025-12-04T08:55:17.0525860Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 392, in forward 2025-12-04T08:55:17.0525988Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:55:17.0526262Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T08:55:17.0526374Z return self.act(input) 2025-12-04T08:55:17.0526378Z 2025-12-04T08:55:17.0526492Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:17.0526723Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:17.0526795Z res = mod(**inputs) 2025-12-04T08:55:17.0527124Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:17.0527208Z outputs = self.bert( 2025-12-04T08:55:17.0527534Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:17.0527643Z encoder_outputs = self.encoder( 2025-12-04T08:55:17.0527976Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:17.0528055Z layer_outputs = layer_module( 2025-12-04T08:55:17.0528313Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:17.0528400Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:17.0528669Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0528752Z return func(*args, **kwargs) 2025-12-04T08:55:17.0529076Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T08:55:17.0529174Z layer_output = apply_chunking_to_forward( 2025-12-04T08:55:17.0529465Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:55:17.0529553Z return forward_fn(*input_tensors) 2025-12-04T08:55:17.0529919Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-12-04T08:55:17.0530065Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T08:55:17.0530389Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 404, in forward 2025-12-04T08:55:17.0530488Z hidden_states = self.dense(hidden_states) 2025-12-04T08:55:17.0530492Z 2025-12-04T08:55:17.0530606Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:17.0530837Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:17.0530908Z res = mod(**inputs) 2025-12-04T08:55:17.0531232Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:17.0531318Z outputs = self.bert( 2025-12-04T08:55:17.0531639Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:17.0531727Z encoder_outputs = self.encoder( 2025-12-04T08:55:17.0532047Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:17.0532126Z layer_outputs = layer_module( 2025-12-04T08:55:17.0532408Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:17.0532496Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:17.0532768Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0532854Z return func(*args, **kwargs) 2025-12-04T08:55:17.0533198Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T08:55:17.0533318Z self_attention_outputs = self.attention( 2025-12-04T08:55:17.0533587Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0533664Z return func(*args, **kwargs) 2025-12-04T08:55:17.0534002Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T08:55:17.0534082Z self_outputs = self.self( 2025-12-04T08:55:17.0534367Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0534443Z return func(*args, **kwargs) 2025-12-04T08:55:17.0534778Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 221, in forward 2025-12-04T08:55:17.0534900Z query_layer = self.query(hidden_states) 2025-12-04T08:55:17.0534906Z 2025-12-04T08:55:17.0535020Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:17.0535240Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:17.0535319Z res = mod(**inputs) 2025-12-04T08:55:17.0535654Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:17.0535734Z outputs = self.bert( 2025-12-04T08:55:17.0536059Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:17.0536141Z encoder_outputs = self.encoder( 2025-12-04T08:55:17.0536471Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:17.0536552Z layer_outputs = layer_module( 2025-12-04T08:55:17.0536807Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:17.0536891Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:17.0537181Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0537264Z return func(*args, **kwargs) 2025-12-04T08:55:17.0537588Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T08:55:17.0537680Z self_attention_outputs = self.attention( 2025-12-04T08:55:17.0537958Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0538035Z return func(*args, **kwargs) 2025-12-04T08:55:17.0538366Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T08:55:17.0538444Z self_outputs = self.self( 2025-12-04T08:55:17.0538715Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0538798Z return func(*args, **kwargs) 2025-12-04T08:55:17.0539121Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 245, in forward 2025-12-04T08:55:17.0539207Z key_layer = self.key(current_states) 2025-12-04T08:55:17.0539242Z 2025-12-04T08:55:17.0539358Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:17.0539579Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:17.0539657Z res = mod(**inputs) 2025-12-04T08:55:17.0540002Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:17.0540094Z outputs = self.bert( 2025-12-04T08:55:17.0540426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:17.0540506Z encoder_outputs = self.encoder( 2025-12-04T08:55:17.0540837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:17.0540916Z layer_outputs = layer_module( 2025-12-04T08:55:17.0541171Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:17.0541265Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:17.0541534Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0541631Z return func(*args, **kwargs) 2025-12-04T08:55:17.0541966Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T08:55:17.0542057Z self_attention_outputs = self.attention( 2025-12-04T08:55:17.0542338Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0542413Z return func(*args, **kwargs) 2025-12-04T08:55:17.0542738Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 366, in forward 2025-12-04T08:55:17.0542823Z self_outputs = self.self( 2025-12-04T08:55:17.0543095Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0543176Z return func(*args, **kwargs) 2025-12-04T08:55:17.0543504Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 249, in forward 2025-12-04T08:55:17.0543595Z value_layer = self.value(current_states) 2025-12-04T08:55:17.0543599Z 2025-12-04T08:55:17.0543695Z cudagraph partition due to non gpu ops 2025-12-04T08:55:17.0543784Z cudagraph partition due to non gpu ops 2025-12-04T08:55:17.0543897Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:17.0544127Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:17.0544199Z res = mod(**inputs) 2025-12-04T08:55:17.0544536Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:17.0544610Z outputs = self.bert( 2025-12-04T08:55:17.0544938Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:17.0545027Z encoder_outputs = self.encoder( 2025-12-04T08:55:17.0545355Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:17.0545434Z layer_outputs = layer_module( 2025-12-04T08:55:17.0545690Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:17.0545777Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:17.0546056Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0546169Z return func(*args, **kwargs) 2025-12-04T08:55:17.0546492Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 439, in forward 2025-12-04T08:55:17.0546590Z self_attention_outputs = self.attention( 2025-12-04T08:55:17.0546872Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0546994Z return func(*args, **kwargs) 2025-12-04T08:55:17.0547317Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 375, in forward 2025-12-04T08:55:17.0547458Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T08:55:17.0547802Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 322, in forward 2025-12-04T08:55:17.0547894Z hidden_states = self.dense(hidden_states) 2025-12-04T08:55:17.0547899Z 2025-12-04T08:55:17.0548018Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:17.0548240Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:17.0548312Z res = mod(**inputs) 2025-12-04T08:55:17.0548666Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:17.0548741Z outputs = self.bert( 2025-12-04T08:55:17.0549081Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:17.0549168Z encoder_outputs = self.encoder( 2025-12-04T08:55:17.0549481Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:17.0549567Z layer_outputs = layer_module( 2025-12-04T08:55:17.0549812Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:17.0549896Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:17.0550167Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0550243Z return func(*args, **kwargs) 2025-12-04T08:55:17.0550557Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T08:55:17.0550658Z layer_output = apply_chunking_to_forward( 2025-12-04T08:55:17.0550941Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:55:17.0551032Z return forward_fn(*input_tensors) 2025-12-04T08:55:17.0551380Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-12-04T08:55:17.0551494Z intermediate_output = self.intermediate(ln_output) 2025-12-04T08:55:17.0551817Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 391, in forward 2025-12-04T08:55:17.0551908Z hidden_states = self.dense(hidden_states) 2025-12-04T08:55:17.0551912Z 2025-12-04T08:55:17.0552031Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:17.0552248Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:17.0552316Z res = mod(**inputs) 2025-12-04T08:55:17.0552641Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:17.0552711Z outputs = self.bert( 2025-12-04T08:55:17.0553045Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:17.0553135Z encoder_outputs = self.encoder( 2025-12-04T08:55:17.0553448Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:17.0553534Z layer_outputs = layer_module( 2025-12-04T08:55:17.0553799Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:17.0553900Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:17.0554171Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0554246Z return func(*args, **kwargs) 2025-12-04T08:55:17.0554566Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T08:55:17.0554655Z layer_output = apply_chunking_to_forward( 2025-12-04T08:55:17.0554943Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:55:17.0555033Z return forward_fn(*input_tensors) 2025-12-04T08:55:17.0555377Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 476, in feed_forward_chunk 2025-12-04T08:55:17.0555510Z intermediate_output = self.intermediate(ln_output) 2025-12-04T08:55:17.0555835Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 392, in forward 2025-12-04T08:55:17.0555957Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:55:17.0556198Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T08:55:17.0556275Z return self.act(input) 2025-12-04T08:55:17.0556279Z 2025-12-04T08:55:17.0556391Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:17.0556612Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:17.0556682Z res = mod(**inputs) 2025-12-04T08:55:17.0557006Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:17.0557081Z outputs = self.bert( 2025-12-04T08:55:17.0557394Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:17.0557480Z encoder_outputs = self.encoder( 2025-12-04T08:55:17.0557792Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:17.0557868Z layer_outputs = layer_module( 2025-12-04T08:55:17.0558125Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:17.0558285Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:17.0558583Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0558664Z return func(*args, **kwargs) 2025-12-04T08:55:17.0558989Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T08:55:17.0559093Z layer_output = apply_chunking_to_forward( 2025-12-04T08:55:17.0559390Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:55:17.0559493Z return forward_fn(*input_tensors) 2025-12-04T08:55:17.0559843Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-12-04T08:55:17.0560010Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T08:55:17.0560329Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 404, in forward 2025-12-04T08:55:17.0560417Z hidden_states = self.dense(hidden_states) 2025-12-04T08:55:17.0560423Z 2025-12-04T08:55:17.0560544Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:17.0560776Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:17.0560866Z res = mod(**inputs) 2025-12-04T08:55:17.0561197Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1056, in forward 2025-12-04T08:55:17.0561270Z outputs = self.bert( 2025-12-04T08:55:17.0561586Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 848, in forward 2025-12-04T08:55:17.0561674Z encoder_outputs = self.encoder( 2025-12-04T08:55:17.0561989Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 532, in forward 2025-12-04T08:55:17.0562073Z layer_outputs = layer_module( 2025-12-04T08:55:17.0562338Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:55:17.0562422Z return super().__call__(*args, **kwargs) 2025-12-04T08:55:17.0562696Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:55:17.0562771Z return func(*args, **kwargs) 2025-12-04T08:55:17.0563085Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 469, in forward 2025-12-04T08:55:17.0563179Z layer_output = apply_chunking_to_forward( 2025-12-04T08:55:17.0563464Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:55:17.0563550Z return forward_fn(*input_tensors) 2025-12-04T08:55:17.0563893Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 477, in feed_forward_chunk 2025-12-04T08:55:17.0564041Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T08:55:17.0564366Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 406, in forward 2025-12-04T08:55:17.0564449Z return input_tensor + hidden_states 2025-12-04T08:55:17.0564453Z 2025-12-04T08:55:17.0564571Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:17.0564788Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:17.0564857Z res = mod(**inputs) 2025-12-04T08:55:17.0565185Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1074, in forward 2025-12-04T08:55:17.0565292Z prediction_scores = self.cls(sequence_output) 2025-12-04T08:55:17.0565610Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 644, in forward 2025-12-04T08:55:17.0565739Z prediction_scores = self.predictions(sequence_output) 2025-12-04T08:55:17.0566053Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 632, in forward 2025-12-04T08:55:17.0566163Z hidden_states = self.transform(hidden_states) 2025-12-04T08:55:17.0566479Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 607, in forward 2025-12-04T08:55:17.0566567Z hidden_states = self.dense(hidden_states) 2025-12-04T08:55:17.0566578Z 2025-12-04T08:55:17.0566709Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:17.0566928Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:17.0567003Z res = mod(**inputs) 2025-12-04T08:55:17.0567328Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1074, in forward 2025-12-04T08:55:17.0567445Z prediction_scores = self.cls(sequence_output) 2025-12-04T08:55:17.0567787Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 644, in forward 2025-12-04T08:55:17.0567908Z prediction_scores = self.predictions(sequence_output) 2025-12-04T08:55:17.0568232Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 633, in forward 2025-12-04T08:55:17.0568329Z hidden_states = self.decoder(hidden_states) 2025-12-04T08:55:17.0568335Z 2025-12-04T08:55:17.0568447Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:55:17.0568670Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:55:17.0568739Z res = mod(**inputs) 2025-12-04T08:55:17.0569098Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1078, in forward 2025-12-04T08:55:17.0569180Z lm_loss = self.loss_function( 2025-12-04T08:55:17.0569448Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/loss/loss_utils.py", line 67, in ForCausalLMLoss 2025-12-04T08:55:17.0569651Z loss = fixed_cross_entropy(logits, shift_labels, num_items_in_batch, ignore_index, **kwargs) 2025-12-04T08:55:17.0569927Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/loss/loss_utils.py", line 36, in fixed_cross_entropy 2025-12-04T08:55:17.0570143Z loss = nn.functional.cross_entropy(source, target, ignore_index=ignore_index, reduction=reduction) 2025-12-04T08:55:17.0570155Z 2025-12-04T08:55:31.3712709Z Compilation time (from dynamo_timed): 27.167160765 2025-12-04T08:55:31.3745739Z pass 2025-12-04T08:55:31.3746355Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T08:55:31.3747427Z TIMING: _recursive_pre_grad_passes:0.01112 _recursive_joint_graph_passes:0.86397 _recursive_post_grad_passes:0.11443 async_compile.wait:1.44509 code_gen:12.58985 inductor_compile:15.07678 backend_compile:21.34502 gc:0.00091 entire_frame_compile:27.16716 total_wall_time:27.16716 2025-12-04T08:55:31.3748460Z STATS: call_* op count: 723 | FakeTensorMode.__torch_dispatch__:16128 | FakeTensor.__torch_dispatch__:8250 | ProxyTorchDispatchMode.__torch_dispatch__:4916 2025-12-04T08:55:31.3749015Z Dynamo produced 1 graphs covering 723 ops with 0 graph breaks (0 unique) 2025-12-04T08:55:34.3813004Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/__init__.py:65: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-12-04T08:55:34.3815428Z import pynvml # type: ignore[import] 2025-12-04T08:55:37.9911987Z 2025-12-04T08:55:38.5318977Z loading model: 0it [00:00, ?it/s] 2025-12-04T08:55:38.5319311Z loading model: 0it [00:00, ?it/s] 2025-12-04T08:55:38.5394375Z cpu eval MobileBertForMaskedLM 2025-12-04T08:55:38.8982373Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T08:55:39.1105555Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T08:55:39.5181357Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T08:56:06.9122295Z cudagraph partition due to non gpu ops 2025-12-04T08:56:06.9122969Z cudagraph partition due to non gpu ops 2025-12-04T08:56:06.9123264Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:06.9123698Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:06.9124065Z res = mod(**inputs) 2025-12-04T08:56:06.9124556Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:06.9125171Z outputs = self.mobilebert( 2025-12-04T08:56:06.9125626Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 791, in forward 2025-12-04T08:56:06.9126099Z embedding_output = self.embeddings( 2025-12-04T08:56:06.9126570Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 199, in forward 2025-12-04T08:56:06.9127033Z inputs_embeds = torch.cat( 2025-12-04T08:56:06.9127163Z 2025-12-04T08:56:06.9127285Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:06.9127696Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:06.9128059Z res = mod(**inputs) 2025-12-04T08:56:06.9128492Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 989, in forward 2025-12-04T08:56:06.9129019Z prediction_scores = self.cls(sequence_output) 2025-12-04T08:56:06.9129522Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 643, in forward 2025-12-04T08:56:06.9130046Z prediction_scores = self.predictions(sequence_output) 2025-12-04T08:56:06.9130533Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 632, in forward 2025-12-04T08:56:06.9131139Z hidden_states = hidden_states.matmul(torch.cat([self.decoder.weight.t(), self.dense.weight], dim=0)) 2025-12-04T08:56:06.9131431Z 2025-12-04T08:56:06.9131548Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:06.9131962Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:06.9132315Z res = mod(**inputs) 2025-12-04T08:56:06.9132753Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:06.9133216Z outputs = self.mobilebert( 2025-12-04T08:56:06.9133667Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 791, in forward 2025-12-04T08:56:06.9134132Z embedding_output = self.embeddings( 2025-12-04T08:56:06.9134601Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 208, in forward 2025-12-04T08:56:06.9135146Z inputs_embeds = self.embedding_transformation(inputs_embeds) 2025-12-04T08:56:06.9135343Z 2025-12-04T08:56:06.9135465Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:06.9135859Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:06.9136343Z res = mod(**inputs) 2025-12-04T08:56:06.9136780Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:06.9137243Z outputs = self.mobilebert( 2025-12-04T08:56:06.9137698Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 791, in forward 2025-12-04T08:56:06.9138177Z embedding_output = self.embeddings( 2025-12-04T08:56:06.9138643Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 215, in forward 2025-12-04T08:56:06.9139136Z embeddings = self.LayerNorm(embeddings) 2025-12-04T08:56:06.9139643Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:06.9140140Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:06.9140310Z 2025-12-04T08:56:06.9140437Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:06.9140859Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:06.9141246Z res = mod(**inputs) 2025-12-04T08:56:06.9178112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:06.9178818Z outputs = self.mobilebert( 2025-12-04T08:56:06.9179421Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:06.9179947Z encoder_outputs = self.encoder( 2025-12-04T08:56:06.9180456Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:06.9180924Z layer_outputs = layer_module( 2025-12-04T08:56:06.9181376Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T08:56:06.9182064Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T08:56:06.9182643Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-12-04T08:56:06.9183146Z shared_attention_input = self.attention(hidden_states) 2025-12-04T08:56:06.9183653Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-12-04T08:56:06.9184136Z layer_input = self.dense(hidden_states) 2025-12-04T08:56:06.9184290Z 2025-12-04T08:56:06.9184426Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:06.9184824Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:06.9185207Z res = mod(**inputs) 2025-12-04T08:56:06.9185682Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:06.9186148Z outputs = self.mobilebert( 2025-12-04T08:56:06.9186604Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:06.9187064Z encoder_outputs = self.encoder( 2025-12-04T08:56:06.9187539Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:06.9188001Z layer_outputs = layer_module( 2025-12-04T08:56:06.9188495Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T08:56:06.9188979Z self_attention_outputs = self.attention( 2025-12-04T08:56:06.9189466Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T08:56:06.9189926Z self_outputs = self.self( 2025-12-04T08:56:06.9190375Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-12-04T08:56:06.9190831Z self.value(value_tensor) 2025-12-04T08:56:06.9190963Z 2025-12-04T08:56:06.9191086Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:06.9191505Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:06.9191865Z res = mod(**inputs) 2025-12-04T08:56:06.9192359Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:06.9192818Z outputs = self.mobilebert( 2025-12-04T08:56:06.9193274Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:06.9193750Z encoder_outputs = self.encoder( 2025-12-04T08:56:06.9194262Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:06.9194795Z layer_outputs = layer_module( 2025-12-04T08:56:06.9195258Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T08:56:06.9195824Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T08:56:06.9196390Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-12-04T08:56:06.9196914Z bottlenecked_hidden_states = self.input(hidden_states) 2025-12-04T08:56:06.9197437Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-12-04T08:56:06.9197912Z layer_input = self.dense(hidden_states) 2025-12-04T08:56:06.9198186Z 2025-12-04T08:56:06.9198315Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:06.9198743Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:06.9199117Z res = mod(**inputs) 2025-12-04T08:56:06.9199555Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:06.9200026Z outputs = self.mobilebert( 2025-12-04T08:56:06.9200486Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:06.9200955Z encoder_outputs = self.encoder( 2025-12-04T08:56:06.9201416Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:06.9201888Z layer_outputs = layer_module( 2025-12-04T08:56:06.9202341Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T08:56:06.9202908Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T08:56:06.9203470Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-12-04T08:56:06.9203992Z shared_attention_input = self.attention(hidden_states) 2025-12-04T08:56:06.9204509Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-12-04T08:56:06.9204996Z layer_input = self.LayerNorm(layer_input) 2025-12-04T08:56:06.9205470Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:06.9205960Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:06.9206129Z 2025-12-04T08:56:06.9206260Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:06.9206660Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:06.9207030Z res = mod(**inputs) 2025-12-04T08:56:06.9207467Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:06.9207933Z outputs = self.mobilebert( 2025-12-04T08:56:06.9208385Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:06.9208857Z encoder_outputs = self.encoder( 2025-12-04T08:56:06.9209352Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:06.9209823Z layer_outputs = layer_module( 2025-12-04T08:56:06.9210286Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T08:56:06.9210800Z self_attention_outputs = self.attention( 2025-12-04T08:56:06.9211311Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T08:56:06.9211772Z self_outputs = self.self( 2025-12-04T08:56:06.9212213Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-12-04T08:56:06.9212668Z self.query(query_tensor) 2025-12-04T08:56:06.9212799Z 2025-12-04T08:56:06.9212916Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:06.9213314Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:06.9213672Z res = mod(**inputs) 2025-12-04T08:56:06.9214093Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:06.9214554Z outputs = self.mobilebert( 2025-12-04T08:56:06.9214992Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:06.9215446Z encoder_outputs = self.encoder( 2025-12-04T08:56:06.9215884Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:06.9216324Z layer_outputs = layer_module( 2025-12-04T08:56:06.9218038Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T08:56:06.9219066Z self_attention_outputs = self.attention( 2025-12-04T08:56:06.9219839Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T08:56:06.9220581Z self_outputs = self.self( 2025-12-04T08:56:06.9221513Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-12-04T08:56:06.9222197Z self.key(key_tensor) 2025-12-04T08:56:06.9222605Z 2025-12-04T08:56:06.9222782Z cudagraph partition due to non gpu ops 2025-12-04T08:56:06.9224266Z cudagraph partition due to non gpu ops 2025-12-04T08:56:06.9224762Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:06.9225274Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:06.9225665Z res = mod(**inputs) 2025-12-04T08:56:06.9226138Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:06.9226614Z outputs = self.mobilebert( 2025-12-04T08:56:06.9227153Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:06.9227638Z encoder_outputs = self.encoder( 2025-12-04T08:56:06.9228113Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:06.9229802Z layer_outputs = layer_module( 2025-12-04T08:56:06.9234031Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T08:56:06.9234835Z self_attention_outputs = self.attention( 2025-12-04T08:56:06.9238918Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-12-04T08:56:06.9243507Z attention_output = self.output(self_outputs[0], layer_input) 2025-12-04T08:56:06.9246089Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-12-04T08:56:06.9246930Z layer_outputs = self.dense(hidden_states) 2025-12-04T08:56:06.9278754Z 2025-12-04T08:56:06.9279266Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:06.9280017Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:06.9280615Z res = mod(**inputs) 2025-12-04T08:56:06.9281352Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:06.9282095Z outputs = self.mobilebert( 2025-12-04T08:56:06.9282773Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:06.9283533Z encoder_outputs = self.encoder( 2025-12-04T08:56:06.9284249Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:06.9284968Z layer_outputs = layer_module( 2025-12-04T08:56:06.9285812Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T08:56:06.9286582Z self_attention_outputs = self.attention( 2025-12-04T08:56:06.9287259Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-12-04T08:56:06.9287784Z attention_output = self.output(self_outputs[0], layer_input) 2025-12-04T08:56:06.9288458Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-12-04T08:56:06.9289105Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T08:56:06.9289874Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:06.9290591Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:06.9290843Z 2025-12-04T08:56:06.9290988Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:06.9291591Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:06.9294588Z res = mod(**inputs) 2025-12-04T08:56:06.9295350Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:06.9296105Z outputs = self.mobilebert( 2025-12-04T08:56:06.9318440Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:06.9319283Z encoder_outputs = self.encoder( 2025-12-04T08:56:06.9320004Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:06.9320881Z layer_outputs = layer_module( 2025-12-04T08:56:06.9321615Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:06.9322400Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:06.9323170Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T08:56:06.9323870Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T08:56:06.9324530Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T08:56:06.9325261Z hidden_states = self.dense(hidden_states) 2025-12-04T08:56:06.9325507Z 2025-12-04T08:56:06.9325834Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:06.9326275Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:06.9326853Z res = mod(**inputs) 2025-12-04T08:56:06.9327522Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:06.9328310Z outputs = self.mobilebert( 2025-12-04T08:56:06.9329098Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:06.9329836Z encoder_outputs = self.encoder( 2025-12-04T08:56:06.9330530Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:06.9331219Z layer_outputs = layer_module( 2025-12-04T08:56:06.9331920Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:06.9332663Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:06.9333411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T08:56:06.9334257Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T08:56:06.9335042Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T08:56:06.9336416Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:56:06.9336930Z 2025-12-04T08:56:06.9337162Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:06.9337852Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:06.9338422Z res = mod(**inputs) 2025-12-04T08:56:06.9339135Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:06.9339669Z outputs = self.mobilebert( 2025-12-04T08:56:06.9340353Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:06.9341086Z encoder_outputs = self.encoder( 2025-12-04T08:56:06.9341796Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:06.9342520Z layer_outputs = layer_module( 2025-12-04T08:56:06.9343230Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:06.9343944Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:06.9344669Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T08:56:06.9345500Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T08:56:06.9346407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T08:56:06.9347548Z layer_outputs = self.dense(hidden_states) 2025-12-04T08:56:06.9347776Z 2025-12-04T08:56:06.9347971Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:06.9348624Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:06.9349173Z res = mod(**inputs) 2025-12-04T08:56:06.9349863Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:06.9350509Z outputs = self.mobilebert( 2025-12-04T08:56:06.9351322Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:06.9351999Z encoder_outputs = self.encoder( 2025-12-04T08:56:06.9352721Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:06.9353449Z layer_outputs = layer_module( 2025-12-04T08:56:06.9354110Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:06.9354898Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:06.9355660Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T08:56:06.9356486Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T08:56:06.9357625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T08:56:06.9358526Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T08:56:06.9359362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:06.9360161Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:06.9360479Z 2025-12-04T08:56:06.9360665Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:06.9361305Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:06.9363621Z res = mod(**inputs) 2025-12-04T08:56:06.9364253Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:06.9364883Z outputs = self.mobilebert( 2025-12-04T08:56:06.9365537Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:06.9366251Z encoder_outputs = self.encoder( 2025-12-04T08:56:06.9367402Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:06.9368067Z layer_outputs = layer_module( 2025-12-04T08:56:06.9368742Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:06.9369504Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:06.9370142Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T08:56:06.9370914Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T08:56:06.9371645Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T08:56:06.9372373Z hidden_states = self.dense(hidden_states) 2025-12-04T08:56:06.9372607Z 2025-12-04T08:56:06.9372739Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:06.9373351Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:06.9373849Z res = mod(**inputs) 2025-12-04T08:56:06.9374524Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:06.9375275Z outputs = self.mobilebert( 2025-12-04T08:56:06.9375976Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:06.9376774Z encoder_outputs = self.encoder( 2025-12-04T08:56:06.9377506Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:06.9378259Z layer_outputs = layer_module( 2025-12-04T08:56:06.9379082Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:06.9379866Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:06.9380628Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T08:56:06.9381508Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T08:56:06.9382362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T08:56:06.9383185Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:56:06.9383485Z 2025-12-04T08:56:06.9383662Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:06.9384304Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:06.9384884Z res = mod(**inputs) 2025-12-04T08:56:06.9385574Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:06.9386331Z outputs = self.mobilebert( 2025-12-04T08:56:06.9389860Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:06.9397923Z encoder_outputs = self.encoder( 2025-12-04T08:56:06.9398849Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:06.9407243Z layer_outputs = layer_module( 2025-12-04T08:56:06.9408137Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:06.9408952Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:06.9409687Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T08:56:06.9410525Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T08:56:06.9411595Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T08:56:06.9413891Z layer_outputs = self.dense(hidden_states) 2025-12-04T08:56:06.9414156Z 2025-12-04T08:56:06.9414347Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:06.9414956Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:06.9415488Z res = mod(**inputs) 2025-12-04T08:56:06.9417292Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:06.9418814Z outputs = self.mobilebert( 2025-12-04T08:56:06.9419309Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:06.9419790Z encoder_outputs = self.encoder( 2025-12-04T08:56:06.9420259Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:06.9426821Z layer_outputs = layer_module( 2025-12-04T08:56:06.9427532Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:06.9428041Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:06.9428539Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T08:56:06.9429065Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T08:56:06.9429784Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T08:56:06.9430295Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T08:56:06.9430819Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:06.9431318Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:06.9431494Z 2025-12-04T08:56:06.9431683Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:06.9432139Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:06.9432529Z res = mod(**inputs) 2025-12-04T08:56:06.9432969Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:06.9433438Z outputs = self.mobilebert( 2025-12-04T08:56:06.9433880Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:06.9434339Z encoder_outputs = self.encoder( 2025-12-04T08:56:06.9434795Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:06.9435244Z layer_outputs = layer_module( 2025-12-04T08:56:06.9435741Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:06.9436220Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:06.9436696Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T08:56:06.9437191Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T08:56:06.9437685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T08:56:06.9438244Z hidden_states = self.dense(hidden_states) 2025-12-04T08:56:06.9438408Z 2025-12-04T08:56:06.9438537Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:06.9438945Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:06.9439314Z res = mod(**inputs) 2025-12-04T08:56:06.9439755Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:06.9440247Z outputs = self.mobilebert( 2025-12-04T08:56:06.9440696Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:06.9441157Z encoder_outputs = self.encoder( 2025-12-04T08:56:06.9441609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:06.9442075Z layer_outputs = layer_module( 2025-12-04T08:56:06.9442526Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:06.9443016Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:06.9443506Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T08:56:06.9444012Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T08:56:06.9444519Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T08:56:06.9445026Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:56:06.9445212Z 2025-12-04T08:56:06.9445338Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:06.9445732Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:06.9446126Z res = mod(**inputs) 2025-12-04T08:56:06.9446568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:06.9447042Z outputs = self.mobilebert( 2025-12-04T08:56:06.9447502Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:06.9448000Z encoder_outputs = self.encoder( 2025-12-04T08:56:06.9448503Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:06.9448967Z layer_outputs = layer_module( 2025-12-04T08:56:06.9449428Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:06.9449926Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:06.9450419Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T08:56:06.9450936Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T08:56:06.9451471Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T08:56:06.9452001Z layer_outputs = self.dense(hidden_states) 2025-12-04T08:56:06.9452163Z 2025-12-04T08:56:06.9452291Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:06.9452691Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:06.9453052Z res = mod(**inputs) 2025-12-04T08:56:06.9453497Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:06.9453990Z outputs = self.mobilebert( 2025-12-04T08:56:06.9454453Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:06.9454906Z encoder_outputs = self.encoder( 2025-12-04T08:56:06.9455349Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:06.9455807Z layer_outputs = layer_module( 2025-12-04T08:56:06.9456253Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:06.9456745Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:06.9457217Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T08:56:06.9457714Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T08:56:06.9458257Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T08:56:06.9458773Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T08:56:06.9459277Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:06.9459762Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:06.9459933Z 2025-12-04T08:56:06.9460051Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:06.9460449Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:06.9460799Z res = mod(**inputs) 2025-12-04T08:56:06.9461223Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:06.9461677Z outputs = self.mobilebert( 2025-12-04T08:56:06.9479088Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:06.9479663Z encoder_outputs = self.encoder( 2025-12-04T08:56:06.9480169Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:06.9480676Z layer_outputs = layer_module( 2025-12-04T08:56:06.9481184Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-12-04T08:56:06.9481741Z intermediate_output = self.intermediate(attention_output) 2025-12-04T08:56:06.9482268Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T08:56:06.9482751Z hidden_states = self.dense(hidden_states) 2025-12-04T08:56:06.9482923Z 2025-12-04T08:56:06.9483055Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:06.9483485Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:06.9483853Z res = mod(**inputs) 2025-12-04T08:56:06.9484287Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:06.9484784Z outputs = self.mobilebert( 2025-12-04T08:56:06.9485240Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:06.9485699Z encoder_outputs = self.encoder( 2025-12-04T08:56:06.9486158Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:06.9486610Z layer_outputs = layer_module( 2025-12-04T08:56:06.9487127Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-12-04T08:56:06.9487648Z intermediate_output = self.intermediate(attention_output) 2025-12-04T08:56:06.9488152Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T08:56:06.9488665Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:56:06.9488862Z 2025-12-04T08:56:06.9488983Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:06.9489394Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:06.9489760Z res = mod(**inputs) 2025-12-04T08:56:06.9490196Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:06.9490666Z outputs = self.mobilebert( 2025-12-04T08:56:06.9491115Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:06.9491582Z encoder_outputs = self.encoder( 2025-12-04T08:56:06.9492043Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:06.9492505Z layer_outputs = layer_module( 2025-12-04T08:56:06.9492948Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T08:56:06.9493516Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T08:56:06.9494085Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-12-04T08:56:06.9494574Z layer_output = self.dense(intermediate_states) 2025-12-04T08:56:06.9494746Z 2025-12-04T08:56:06.9494868Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:06.9495286Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:06.9495678Z res = mod(**inputs) 2025-12-04T08:56:06.9496110Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:06.9496576Z outputs = self.mobilebert( 2025-12-04T08:56:06.9497030Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:06.9497536Z encoder_outputs = self.encoder( 2025-12-04T08:56:06.9497988Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:06.9498452Z layer_outputs = layer_module( 2025-12-04T08:56:06.9498903Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T08:56:06.9499458Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T08:56:06.9500012Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-12-04T08:56:06.9500532Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-12-04T08:56:06.9501054Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:06.9501569Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:06.9501738Z 2025-12-04T08:56:06.9501858Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:06.9502278Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:06.9502643Z res = mod(**inputs) 2025-12-04T08:56:06.9503072Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:06.9503535Z outputs = self.mobilebert( 2025-12-04T08:56:06.9503987Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:06.9504456Z encoder_outputs = self.encoder( 2025-12-04T08:56:06.9504906Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:06.9505371Z layer_outputs = layer_module( 2025-12-04T08:56:06.9505828Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T08:56:06.9506393Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T08:56:06.9506947Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-12-04T08:56:06.9507481Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-12-04T08:56:06.9508005Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-12-04T08:56:06.9508485Z layer_outputs = self.dense(hidden_states) 2025-12-04T08:56:06.9508650Z 2025-12-04T08:56:06.9508769Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:06.9509173Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:06.9509534Z res = mod(**inputs) 2025-12-04T08:56:06.9509955Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:06.9510419Z outputs = self.mobilebert( 2025-12-04T08:56:06.9510870Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:06.9511333Z encoder_outputs = self.encoder( 2025-12-04T08:56:06.9511801Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:06.9512267Z layer_outputs = layer_module( 2025-12-04T08:56:06.9512723Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T08:56:06.9513290Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T08:56:06.9513879Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-12-04T08:56:06.9514441Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-12-04T08:56:06.9514970Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-12-04T08:56:06.9515532Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T08:56:06.9516061Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:06.9516592Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:06.9516757Z 2025-12-04T08:56:06.9516885Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:06.9517313Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:06.9517682Z res = mod(**inputs) 2025-12-04T08:56:06.9518317Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:06.9518817Z outputs = self.mobilebert( 2025-12-04T08:56:06.9519278Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:06.9519797Z encoder_outputs = self.encoder( 2025-12-04T08:56:06.9520279Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:06.9520966Z layer_outputs = layer_module( 2025-12-04T08:56:06.9532857Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T08:56:06.9533495Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T08:56:06.9534088Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-12-04T08:56:06.9534614Z shared_attention_input = self.attention(hidden_states) 2025-12-04T08:56:06.9535133Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-12-04T08:56:06.9535626Z layer_input = self.dense(hidden_states) 2025-12-04T08:56:06.9535792Z 2025-12-04T08:56:06.9535930Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:06.9536346Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:06.9536785Z res = mod(**inputs) 2025-12-04T08:56:06.9537236Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:06.9537719Z outputs = self.mobilebert( 2025-12-04T08:56:06.9538185Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:06.9538658Z encoder_outputs = self.encoder( 2025-12-04T08:56:06.9539135Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:06.9539602Z layer_outputs = layer_module( 2025-12-04T08:56:06.9540265Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T08:56:06.9540760Z self_attention_outputs = self.attention( 2025-12-04T08:56:06.9541243Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T08:56:06.9541708Z self_outputs = self.self( 2025-12-04T08:56:06.9542241Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-12-04T08:56:06.9542748Z self.value(value_tensor) 2025-12-04T08:56:06.9542881Z 2025-12-04T08:56:06.9543000Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:06.9543417Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:06.9543786Z res = mod(**inputs) 2025-12-04T08:56:06.9544229Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:06.9544697Z outputs = self.mobilebert( 2025-12-04T08:56:06.9545154Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:06.9545626Z encoder_outputs = self.encoder( 2025-12-04T08:56:06.9546124Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:06.9546590Z layer_outputs = layer_module( 2025-12-04T08:56:06.9547046Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T08:56:06.9547607Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T08:56:06.9548162Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-12-04T08:56:06.9548675Z bottlenecked_hidden_states = self.input(hidden_states) 2025-12-04T08:56:06.9549181Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-12-04T08:56:06.9549651Z layer_input = self.dense(hidden_states) 2025-12-04T08:56:06.9549809Z 2025-12-04T08:56:06.9549927Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:06.9550334Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:06.9550708Z res = mod(**inputs) 2025-12-04T08:56:06.9551135Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:06.9551600Z outputs = self.mobilebert( 2025-12-04T08:56:06.9552048Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:06.9552511Z encoder_outputs = self.encoder( 2025-12-04T08:56:06.9552963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:06.9553425Z layer_outputs = layer_module( 2025-12-04T08:56:06.9553881Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T08:56:06.9554441Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T08:56:06.9554990Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-12-04T08:56:06.9555503Z shared_attention_input = self.attention(hidden_states) 2025-12-04T08:56:06.9556029Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-12-04T08:56:06.9556507Z layer_input = self.LayerNorm(layer_input) 2025-12-04T08:56:06.9557017Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:06.9557509Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:06.9557677Z 2025-12-04T08:56:06.9557806Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:06.9558333Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:06.9558788Z res = mod(**inputs) 2025-12-04T08:56:06.9559228Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:06.9559698Z outputs = self.mobilebert( 2025-12-04T08:56:06.9560142Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:06.9560613Z encoder_outputs = self.encoder( 2025-12-04T08:56:06.9561079Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:06.9561542Z layer_outputs = layer_module( 2025-12-04T08:56:06.9561989Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T08:56:06.9562498Z self_attention_outputs = self.attention( 2025-12-04T08:56:06.9562970Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T08:56:06.9563425Z self_outputs = self.self( 2025-12-04T08:56:06.9563883Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-12-04T08:56:06.9564342Z self.query(query_tensor) 2025-12-04T08:56:06.9564471Z 2025-12-04T08:56:06.9564601Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:06.9564984Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:06.9565334Z res = mod(**inputs) 2025-12-04T08:56:06.9565748Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:06.9566189Z outputs = self.mobilebert( 2025-12-04T08:56:06.9566623Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:06.9567071Z encoder_outputs = self.encoder( 2025-12-04T08:56:06.9567513Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:06.9567981Z layer_outputs = layer_module( 2025-12-04T08:56:06.9568423Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T08:56:06.9568891Z self_attention_outputs = self.attention( 2025-12-04T08:56:06.9569363Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T08:56:06.9569809Z self_outputs = self.self( 2025-12-04T08:56:06.9570250Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-12-04T08:56:06.9570701Z self.key(key_tensor) 2025-12-04T08:56:06.9570820Z 2025-12-04T08:56:06.9570914Z cudagraph partition due to non gpu ops 2025-12-04T08:56:06.9571155Z cudagraph partition due to non gpu ops 2025-12-04T08:56:06.9571419Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:06.9571814Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:06.9572163Z res = mod(**inputs) 2025-12-04T08:56:06.9572607Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:06.9573066Z outputs = self.mobilebert( 2025-12-04T08:56:06.9573497Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:06.9573953Z encoder_outputs = self.encoder( 2025-12-04T08:56:06.9574420Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:06.9574891Z layer_outputs = layer_module( 2025-12-04T08:56:06.9575324Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T08:56:06.9575789Z self_attention_outputs = self.attention( 2025-12-04T08:56:06.9576255Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-12-04T08:56:06.9576758Z attention_output = self.output(self_outputs[0], layer_input) 2025-12-04T08:56:06.9577249Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-12-04T08:56:06.9577712Z layer_outputs = self.dense(hidden_states) 2025-12-04T08:56:06.9577887Z 2025-12-04T08:56:06.9578011Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:06.9578397Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:06.9578752Z res = mod(**inputs) 2025-12-04T08:56:06.9579175Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:06.9579622Z outputs = self.mobilebert( 2025-12-04T08:56:06.9580050Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:06.9580507Z encoder_outputs = self.encoder( 2025-12-04T08:56:06.9580954Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:06.9581403Z layer_outputs = layer_module( 2025-12-04T08:56:06.9581839Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T08:56:06.9582306Z self_attention_outputs = self.attention( 2025-12-04T08:56:06.9582769Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-12-04T08:56:06.9583264Z attention_output = self.output(self_outputs[0], layer_input) 2025-12-04T08:56:06.9583784Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-12-04T08:56:06.9584327Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T08:56:06.9584857Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:06.9585322Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:06.9585496Z 2025-12-04T08:56:06.9585608Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:06.9585996Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:06.9586350Z res = mod(**inputs) 2025-12-04T08:56:06.9586770Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:06.9587220Z outputs = self.mobilebert( 2025-12-04T08:56:06.9587651Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:06.9588093Z encoder_outputs = self.encoder( 2025-12-04T08:56:06.9588563Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:06.9589013Z layer_outputs = layer_module( 2025-12-04T08:56:06.9589454Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:06.9589949Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:06.9590449Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T08:56:06.9590955Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T08:56:06.9591453Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T08:56:06.9591919Z hidden_states = self.dense(hidden_states) 2025-12-04T08:56:06.9592080Z 2025-12-04T08:56:06.9592197Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:06.9592596Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:06.9592948Z res = mod(**inputs) 2025-12-04T08:56:06.9593377Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:06.9593860Z outputs = self.mobilebert( 2025-12-04T08:56:06.9594320Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:06.9594786Z encoder_outputs = self.encoder( 2025-12-04T08:56:06.9595250Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:06.9595718Z layer_outputs = layer_module( 2025-12-04T08:56:06.9596178Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:06.9596666Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:06.9597161Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T08:56:06.9597677Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T08:56:06.9598279Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T08:56:06.9598807Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:56:06.9599003Z 2025-12-04T08:56:06.9599121Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:06.9599526Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:06.9599885Z res = mod(**inputs) 2025-12-04T08:56:06.9600324Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:06.9600791Z outputs = self.mobilebert( 2025-12-04T08:56:06.9601240Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:06.9601702Z encoder_outputs = self.encoder( 2025-12-04T08:56:06.9602163Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:06.9602637Z layer_outputs = layer_module( 2025-12-04T08:56:06.9603088Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:06.9603576Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:06.9604114Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T08:56:06.9604647Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T08:56:06.9605174Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T08:56:06.9605665Z layer_outputs = self.dense(hidden_states) 2025-12-04T08:56:06.9605830Z 2025-12-04T08:56:06.9607648Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:06.9608076Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:06.9608441Z res = mod(**inputs) 2025-12-04T08:56:06.9608874Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:06.9609330Z outputs = self.mobilebert( 2025-12-04T08:56:06.9609776Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:06.9610226Z encoder_outputs = self.encoder( 2025-12-04T08:56:06.9610675Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:06.9611127Z layer_outputs = layer_module( 2025-12-04T08:56:06.9611592Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:06.9612069Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:06.9612547Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T08:56:06.9613070Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T08:56:06.9613568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T08:56:06.9614086Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T08:56:06.9614593Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:06.9615070Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:06.9615237Z 2025-12-04T08:56:06.9615352Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:06.9615747Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:06.9616106Z res = mod(**inputs) 2025-12-04T08:56:06.9616525Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:06.9616980Z outputs = self.mobilebert( 2025-12-04T08:56:06.9617419Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:06.9617875Z encoder_outputs = self.encoder( 2025-12-04T08:56:06.9618318Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:06.9618770Z layer_outputs = layer_module( 2025-12-04T08:56:06.9619218Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:06.9619695Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:06.9620163Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T08:56:06.9620658Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T08:56:06.9621620Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T08:56:06.9622088Z hidden_states = self.dense(hidden_states) 2025-12-04T08:56:06.9622323Z 2025-12-04T08:56:06.9622441Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:06.9622833Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:06.9623186Z res = mod(**inputs) 2025-12-04T08:56:06.9623638Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:06.9624150Z outputs = self.mobilebert( 2025-12-04T08:56:06.9624591Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:06.9625048Z encoder_outputs = self.encoder( 2025-12-04T08:56:06.9625487Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:06.9625938Z layer_outputs = layer_module( 2025-12-04T08:56:06.9626383Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:06.9626850Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:06.9627323Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T08:56:06.9627848Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T08:56:06.9628337Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T08:56:06.9628830Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:56:06.9629018Z 2025-12-04T08:56:06.9629133Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:06.9629534Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:06.9629897Z res = mod(**inputs) 2025-12-04T08:56:06.9630326Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:06.9630791Z outputs = self.mobilebert( 2025-12-04T08:56:06.9631240Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:06.9631700Z encoder_outputs = self.encoder( 2025-12-04T08:56:06.9632162Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:06.9632630Z layer_outputs = layer_module( 2025-12-04T08:56:06.9633084Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:06.9633572Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:06.9634071Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T08:56:06.9634594Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T08:56:06.9635116Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T08:56:06.9635590Z layer_outputs = self.dense(hidden_states) 2025-12-04T08:56:06.9635754Z 2025-12-04T08:56:06.9635874Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:06.9636282Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:06.9636640Z res = mod(**inputs) 2025-12-04T08:56:06.9637077Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:06.9637546Z outputs = self.mobilebert( 2025-12-04T08:56:06.9638023Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:06.9638565Z encoder_outputs = self.encoder( 2025-12-04T08:56:06.9639029Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:06.9639515Z layer_outputs = layer_module( 2025-12-04T08:56:06.9640003Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:06.9640516Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:06.9641010Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T08:56:06.9641543Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T08:56:06.9642060Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T08:56:06.9642589Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T08:56:06.9643106Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:06.9643629Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:06.9643796Z 2025-12-04T08:56:06.9643915Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:06.9644323Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:06.9644687Z res = mod(**inputs) 2025-12-04T08:56:06.9645123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:06.9645581Z outputs = self.mobilebert( 2025-12-04T08:56:06.9646028Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:06.9646508Z encoder_outputs = self.encoder( 2025-12-04T08:56:06.9646968Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:06.9647429Z layer_outputs = layer_module( 2025-12-04T08:56:06.9647876Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:06.9648363Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:06.9648829Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T08:56:06.9649338Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T08:56:06.9649825Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T08:56:06.9650291Z hidden_states = self.dense(hidden_states) 2025-12-04T08:56:06.9650442Z 2025-12-04T08:56:06.9650555Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:06.9650945Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:06.9651299Z res = mod(**inputs) 2025-12-04T08:56:06.9651721Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:06.9652169Z outputs = self.mobilebert( 2025-12-04T08:56:06.9652603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:06.9653072Z encoder_outputs = self.encoder( 2025-12-04T08:56:06.9653507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:06.9653953Z layer_outputs = layer_module( 2025-12-04T08:56:06.9654423Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:06.9654903Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:06.9655370Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T08:56:06.9655892Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T08:56:06.9656404Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T08:56:06.9656891Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:56:06.9657078Z 2025-12-04T08:56:06.9657191Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:06.9657580Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:06.9657937Z res = mod(**inputs) 2025-12-04T08:56:06.9658349Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:06.9658799Z outputs = self.mobilebert( 2025-12-04T08:56:06.9659234Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:06.9659732Z encoder_outputs = self.encoder( 2025-12-04T08:56:06.9660176Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:06.9660635Z layer_outputs = layer_module( 2025-12-04T08:56:06.9661083Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:06.9661553Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:06.9662037Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T08:56:06.9662553Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T08:56:06.9663061Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T08:56:06.9663523Z layer_outputs = self.dense(hidden_states) 2025-12-04T08:56:06.9663684Z 2025-12-04T08:56:06.9663799Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:06.9664193Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:06.9664550Z res = mod(**inputs) 2025-12-04T08:56:06.9664965Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:06.9665418Z outputs = self.mobilebert( 2025-12-04T08:56:06.9665874Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:06.9666335Z encoder_outputs = self.encoder( 2025-12-04T08:56:06.9666796Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:06.9667262Z layer_outputs = layer_module( 2025-12-04T08:56:06.9667725Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:06.9668205Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:06.9668694Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T08:56:06.9669214Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T08:56:06.9669765Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T08:56:06.9670278Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T08:56:06.9670795Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:06.9671284Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:06.9671450Z 2025-12-04T08:56:06.9671598Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:06.9672017Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:06.9672383Z res = mod(**inputs) 2025-12-04T08:56:06.9672818Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:06.9673277Z outputs = self.mobilebert( 2025-12-04T08:56:06.9673736Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:06.9674198Z encoder_outputs = self.encoder( 2025-12-04T08:56:06.9674653Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:06.9675133Z layer_outputs = layer_module( 2025-12-04T08:56:06.9675594Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-12-04T08:56:06.9676114Z intermediate_output = self.intermediate(attention_output) 2025-12-04T08:56:06.9676629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T08:56:06.9677103Z hidden_states = self.dense(hidden_states) 2025-12-04T08:56:06.9677267Z 2025-12-04T08:56:06.9677387Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:06.9677795Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:06.9678240Z res = mod(**inputs) 2025-12-04T08:56:06.9678686Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:06.9679159Z outputs = self.mobilebert( 2025-12-04T08:56:06.9679615Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:06.9680086Z encoder_outputs = self.encoder( 2025-12-04T08:56:06.9680544Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:06.9681011Z layer_outputs = layer_module( 2025-12-04T08:56:06.9681461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-12-04T08:56:06.9681985Z intermediate_output = self.intermediate(attention_output) 2025-12-04T08:56:06.9682545Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T08:56:06.9683095Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:56:06.9683286Z 2025-12-04T08:56:06.9683401Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:06.9683805Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:06.9684168Z res = mod(**inputs) 2025-12-04T08:56:06.9684613Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:06.9685078Z outputs = self.mobilebert( 2025-12-04T08:56:06.9685578Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:06.9686130Z encoder_outputs = self.encoder( 2025-12-04T08:56:06.9686627Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:06.9687124Z layer_outputs = layer_module( 2025-12-04T08:56:06.9687627Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T08:56:06.9688238Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T08:56:06.9688863Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-12-04T08:56:06.9689379Z layer_output = self.dense(intermediate_states) 2025-12-04T08:56:06.9689554Z 2025-12-04T08:56:06.9689675Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:06.9690081Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:06.9690468Z res = mod(**inputs) 2025-12-04T08:56:06.9690916Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:06.9691391Z outputs = self.mobilebert( 2025-12-04T08:56:06.9691864Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:06.9692326Z encoder_outputs = self.encoder( 2025-12-04T08:56:06.9692777Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:06.9693234Z layer_outputs = layer_module( 2025-12-04T08:56:06.9693673Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T08:56:06.9694218Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T08:56:06.9694773Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-12-04T08:56:06.9695282Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-12-04T08:56:06.9695786Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:06.9696287Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:06.9696457Z 2025-12-04T08:56:06.9696587Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:06.9696997Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:06.9697361Z res = mod(**inputs) 2025-12-04T08:56:06.9697800Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:06.9698265Z outputs = self.mobilebert( 2025-12-04T08:56:06.9698713Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:06.9699185Z encoder_outputs = self.encoder( 2025-12-04T08:56:06.9699648Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:06.9700120Z layer_outputs = layer_module( 2025-12-04T08:56:06.9700575Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T08:56:06.9701140Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T08:56:06.9701706Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-12-04T08:56:06.9702228Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-12-04T08:56:06.9702759Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-12-04T08:56:06.9703224Z layer_outputs = self.dense(hidden_states) 2025-12-04T08:56:06.9703376Z 2025-12-04T08:56:06.9703498Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:06.9703899Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:06.9704286Z res = mod(**inputs) 2025-12-04T08:56:06.9704710Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:06.9705169Z outputs = self.mobilebert( 2025-12-04T08:56:06.9705605Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:06.9706066Z encoder_outputs = self.encoder( 2025-12-04T08:56:06.9706520Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:06.9706983Z layer_outputs = layer_module( 2025-12-04T08:56:06.9707428Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T08:56:06.9708003Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T08:56:06.9708552Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-12-04T08:56:06.9709077Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-12-04T08:56:06.9709573Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-12-04T08:56:06.9710101Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T08:56:06.9710609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:06.9711074Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:06.9711241Z 2025-12-04T08:56:06.9711353Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:06.9711743Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:06.9712095Z res = mod(**inputs) 2025-12-04T08:56:06.9712504Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:06.9712954Z outputs = self.mobilebert( 2025-12-04T08:56:06.9713388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:06.9713840Z encoder_outputs = self.encoder( 2025-12-04T08:56:06.9714284Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:06.9714749Z layer_outputs = layer_module( 2025-12-04T08:56:06.9715203Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T08:56:06.9715767Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T08:56:06.9716339Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-12-04T08:56:06.9716848Z shared_attention_input = self.attention(hidden_states) 2025-12-04T08:56:06.9717351Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-12-04T08:56:06.9717821Z layer_input = self.dense(hidden_states) 2025-12-04T08:56:06.9717988Z 2025-12-04T08:56:06.9718221Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:06.9718643Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:06.9719012Z res = mod(**inputs) 2025-12-04T08:56:06.9719446Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:06.9719917Z outputs = self.mobilebert( 2025-12-04T08:56:06.9720438Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:06.9721124Z encoder_outputs = self.encoder( 2025-12-04T08:56:06.9721593Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:06.9722058Z layer_outputs = layer_module( 2025-12-04T08:56:06.9722524Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T08:56:06.9723036Z self_attention_outputs = self.attention( 2025-12-04T08:56:06.9723568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T08:56:06.9724126Z self_outputs = self.self( 2025-12-04T08:56:06.9724634Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-12-04T08:56:06.9725122Z self.value(value_tensor) 2025-12-04T08:56:06.9725270Z 2025-12-04T08:56:06.9725387Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:06.9725796Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:06.9726186Z res = mod(**inputs) 2025-12-04T08:56:06.9726626Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:06.9727117Z outputs = self.mobilebert( 2025-12-04T08:56:06.9727582Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:06.9728077Z encoder_outputs = self.encoder( 2025-12-04T08:56:06.9728605Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:06.9729097Z layer_outputs = layer_module( 2025-12-04T08:56:06.9729590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T08:56:06.9730189Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T08:56:06.9730807Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-12-04T08:56:06.9731357Z bottlenecked_hidden_states = self.input(hidden_states) 2025-12-04T08:56:06.9731905Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-12-04T08:56:06.9732398Z layer_input = self.dense(hidden_states) 2025-12-04T08:56:06.9732564Z 2025-12-04T08:56:06.9732678Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:06.9733086Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:06.9733443Z res = mod(**inputs) 2025-12-04T08:56:06.9733877Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:06.9734339Z outputs = self.mobilebert( 2025-12-04T08:56:06.9734786Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:06.9735294Z encoder_outputs = self.encoder( 2025-12-04T08:56:06.9735759Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:06.9736235Z layer_outputs = layer_module( 2025-12-04T08:56:06.9736682Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T08:56:06.9737273Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T08:56:06.9737868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-12-04T08:56:06.9738382Z shared_attention_input = self.attention(hidden_states) 2025-12-04T08:56:06.9738884Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-12-04T08:56:06.9739374Z layer_input = self.LayerNorm(layer_input) 2025-12-04T08:56:06.9739854Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:06.9740347Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:06.9740514Z 2025-12-04T08:56:06.9740655Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:06.9741060Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:06.9741428Z res = mod(**inputs) 2025-12-04T08:56:06.9741855Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:06.9742320Z outputs = self.mobilebert( 2025-12-04T08:56:06.9742767Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:06.9743238Z encoder_outputs = self.encoder( 2025-12-04T08:56:06.9743693Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:06.9744159Z layer_outputs = layer_module( 2025-12-04T08:56:06.9744615Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T08:56:06.9745104Z self_attention_outputs = self.attention( 2025-12-04T08:56:06.9745578Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T08:56:06.9746041Z self_outputs = self.self( 2025-12-04T08:56:06.9746491Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-12-04T08:56:06.9746947Z self.query(query_tensor) 2025-12-04T08:56:06.9747085Z 2025-12-04T08:56:06.9747199Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:06.9747607Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:06.9747972Z res = mod(**inputs) 2025-12-04T08:56:06.9748398Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:06.9748865Z outputs = self.mobilebert( 2025-12-04T08:56:06.9749319Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:06.9749782Z encoder_outputs = self.encoder( 2025-12-04T08:56:06.9750234Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:06.9750715Z layer_outputs = layer_module( 2025-12-04T08:56:06.9751177Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T08:56:06.9751657Z self_attention_outputs = self.attention( 2025-12-04T08:56:06.9752133Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T08:56:06.9752596Z self_outputs = self.self( 2025-12-04T08:56:06.9753076Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-12-04T08:56:06.9753591Z self.key(key_tensor) 2025-12-04T08:56:06.9753717Z 2025-12-04T08:56:06.9753810Z cudagraph partition due to non gpu ops 2025-12-04T08:56:06.9754054Z cudagraph partition due to non gpu ops 2025-12-04T08:56:06.9754314Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:06.9754719Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:06.9755083Z res = mod(**inputs) 2025-12-04T08:56:06.9755526Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:06.9755980Z outputs = self.mobilebert( 2025-12-04T08:56:06.9756439Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:06.9756931Z encoder_outputs = self.encoder( 2025-12-04T08:56:06.9757394Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:06.9757860Z layer_outputs = layer_module( 2025-12-04T08:56:06.9758535Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T08:56:06.9759024Z self_attention_outputs = self.attention( 2025-12-04T08:56:06.9759498Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-12-04T08:56:06.9760031Z attention_output = self.output(self_outputs[0], layer_input) 2025-12-04T08:56:06.9760537Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-12-04T08:56:06.9761006Z layer_outputs = self.dense(hidden_states) 2025-12-04T08:56:06.9761163Z 2025-12-04T08:56:06.9761274Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:06.9761675Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:06.9762038Z res = mod(**inputs) 2025-12-04T08:56:06.9762454Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:06.9762908Z outputs = self.mobilebert( 2025-12-04T08:56:06.9763343Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:06.9763800Z encoder_outputs = self.encoder( 2025-12-04T08:56:06.9764254Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:06.9764735Z layer_outputs = layer_module( 2025-12-04T08:56:06.9765183Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T08:56:06.9765664Z self_attention_outputs = self.attention( 2025-12-04T08:56:06.9766114Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-12-04T08:56:06.9766615Z attention_output = self.output(self_outputs[0], layer_input) 2025-12-04T08:56:06.9767117Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-12-04T08:56:06.9767683Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T08:56:06.9768208Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:06.9768703Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:06.9768876Z 2025-12-04T08:56:06.9768997Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:06.9769419Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:06.9769822Z res = mod(**inputs) 2025-12-04T08:56:06.9770249Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:06.9770704Z outputs = self.mobilebert( 2025-12-04T08:56:06.9771138Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:06.9771610Z encoder_outputs = self.encoder( 2025-12-04T08:56:06.9772061Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:06.9772519Z layer_outputs = layer_module( 2025-12-04T08:56:06.9772963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:06.9773480Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:06.9773954Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T08:56:06.9774440Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T08:56:06.9774929Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T08:56:06.9775388Z hidden_states = self.dense(hidden_states) 2025-12-04T08:56:06.9775538Z 2025-12-04T08:56:06.9775657Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:06.9776049Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:06.9776410Z res = mod(**inputs) 2025-12-04T08:56:06.9776837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:06.9777302Z outputs = self.mobilebert( 2025-12-04T08:56:06.9777742Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:06.9778203Z encoder_outputs = self.encoder( 2025-12-04T08:56:06.9778657Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:06.9779111Z layer_outputs = layer_module( 2025-12-04T08:56:06.9779562Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:06.9780053Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:06.9780537Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T08:56:06.9781041Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T08:56:06.9781543Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T08:56:06.9782051Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:56:06.9782236Z 2025-12-04T08:56:06.9782360Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:06.9782750Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:06.9783125Z res = mod(**inputs) 2025-12-04T08:56:06.9783588Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:06.9784045Z outputs = self.mobilebert( 2025-12-04T08:56:06.9784493Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:06.9784961Z encoder_outputs = self.encoder( 2025-12-04T08:56:06.9785462Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:06.9785938Z layer_outputs = layer_module( 2025-12-04T08:56:06.9786390Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:06.9786878Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:06.9787358Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T08:56:06.9787877Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T08:56:06.9788397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T08:56:06.9788897Z layer_outputs = self.dense(hidden_states) 2025-12-04T08:56:06.9789053Z 2025-12-04T08:56:06.9789170Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:06.9789573Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:06.9789934Z res = mod(**inputs) 2025-12-04T08:56:06.9790366Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:06.9790821Z outputs = self.mobilebert( 2025-12-04T08:56:06.9791274Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:06.9791741Z encoder_outputs = self.encoder( 2025-12-04T08:56:06.9792189Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:06.9792659Z layer_outputs = layer_module( 2025-12-04T08:56:06.9793099Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:06.9793572Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:06.9794036Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T08:56:06.9794545Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T08:56:06.9795064Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T08:56:06.9795587Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T08:56:06.9796100Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:06.9796588Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:06.9796756Z 2025-12-04T08:56:06.9796879Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:06.9797104Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:06.9797185Z res = mod(**inputs) 2025-12-04T08:56:06.9797505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:06.9797585Z outputs = self.mobilebert( 2025-12-04T08:56:06.9797904Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:06.9798012Z encoder_outputs = self.encoder( 2025-12-04T08:56:06.9798416Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:06.9798503Z layer_outputs = layer_module( 2025-12-04T08:56:06.9798817Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:06.9798987Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:06.9799330Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T08:56:06.9799451Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T08:56:06.9799763Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T08:56:06.9799856Z hidden_states = self.dense(hidden_states) 2025-12-04T08:56:06.9799862Z 2025-12-04T08:56:06.9799983Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:06.9800198Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:06.9800270Z res = mod(**inputs) 2025-12-04T08:56:06.9800598Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:06.9800678Z outputs = self.mobilebert( 2025-12-04T08:56:06.9800989Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:06.9801070Z encoder_outputs = self.encoder( 2025-12-04T08:56:06.9801371Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:06.9801457Z layer_outputs = layer_module( 2025-12-04T08:56:06.9801776Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:06.9801877Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:06.9802185Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T08:56:06.9802309Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T08:56:06.9802625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T08:56:06.9802747Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:56:06.9802751Z 2025-12-04T08:56:06.9802863Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:06.9803089Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:06.9803158Z res = mod(**inputs) 2025-12-04T08:56:06.9803472Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:06.9803549Z outputs = self.mobilebert( 2025-12-04T08:56:06.9803852Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:06.9803942Z encoder_outputs = self.encoder( 2025-12-04T08:56:06.9804248Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:06.9804326Z layer_outputs = layer_module( 2025-12-04T08:56:06.9804637Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:06.9804736Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:06.9805078Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T08:56:06.9805216Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T08:56:06.9805521Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T08:56:06.9805622Z layer_outputs = self.dense(hidden_states) 2025-12-04T08:56:06.9805625Z 2025-12-04T08:56:06.9805751Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:06.9806001Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:06.9806071Z res = mod(**inputs) 2025-12-04T08:56:06.9806375Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:06.9806457Z outputs = self.mobilebert( 2025-12-04T08:56:06.9806764Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:06.9806850Z encoder_outputs = self.encoder( 2025-12-04T08:56:06.9807154Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:06.9807810Z layer_outputs = layer_module( 2025-12-04T08:56:06.9808133Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:06.9808238Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:06.9808549Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T08:56:06.9808695Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T08:56:06.9809008Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T08:56:06.9809164Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T08:56:06.9809467Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:06.9809568Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:06.9809571Z 2025-12-04T08:56:06.9809691Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:06.9809911Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:06.9809989Z res = mod(**inputs) 2025-12-04T08:56:06.9810300Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:06.9810378Z outputs = self.mobilebert( 2025-12-04T08:56:06.9810697Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:06.9810780Z encoder_outputs = self.encoder( 2025-12-04T08:56:06.9811092Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:06.9811177Z layer_outputs = layer_module( 2025-12-04T08:56:06.9811491Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:06.9811602Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:06.9811911Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T08:56:06.9812035Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T08:56:06.9812354Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T08:56:06.9812471Z hidden_states = self.dense(hidden_states) 2025-12-04T08:56:06.9812475Z 2025-12-04T08:56:06.9812596Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:06.9812818Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:06.9812889Z res = mod(**inputs) 2025-12-04T08:56:06.9813234Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:06.9813335Z outputs = self.mobilebert( 2025-12-04T08:56:06.9813652Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:06.9813740Z encoder_outputs = self.encoder( 2025-12-04T08:56:06.9814054Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:06.9814141Z layer_outputs = layer_module( 2025-12-04T08:56:06.9814459Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:06.9814561Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:06.9814883Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T08:56:06.9815035Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T08:56:06.9815356Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T08:56:06.9815478Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:56:06.9815482Z 2025-12-04T08:56:06.9815595Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:06.9815822Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:06.9815894Z res = mod(**inputs) 2025-12-04T08:56:06.9816214Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:06.9816292Z outputs = self.mobilebert( 2025-12-04T08:56:06.9816603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:06.9816693Z encoder_outputs = self.encoder( 2025-12-04T08:56:06.9817006Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:06.9817085Z layer_outputs = layer_module( 2025-12-04T08:56:06.9817410Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:06.9817513Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:06.9817833Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T08:56:06.9817968Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T08:56:06.9818282Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T08:56:06.9818384Z layer_outputs = self.dense(hidden_states) 2025-12-04T08:56:06.9818389Z 2025-12-04T08:56:06.9818503Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:06.9818729Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:06.9818801Z res = mod(**inputs) 2025-12-04T08:56:06.9819113Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:06.9819197Z outputs = self.mobilebert( 2025-12-04T08:56:06.9819551Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:06.9819634Z encoder_outputs = self.encoder( 2025-12-04T08:56:06.9819954Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:06.9820037Z layer_outputs = layer_module( 2025-12-04T08:56:06.9820378Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:06.9820501Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:06.9820970Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T08:56:06.9821121Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T08:56:06.9821438Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T08:56:06.9821582Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T08:56:06.9821901Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:06.9822058Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:06.9822062Z 2025-12-04T08:56:06.9822189Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:06.9822412Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:06.9822493Z res = mod(**inputs) 2025-12-04T08:56:06.9822806Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:06.9822885Z outputs = self.mobilebert( 2025-12-04T08:56:06.9823206Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:06.9823286Z encoder_outputs = self.encoder( 2025-12-04T08:56:06.9823598Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:06.9823687Z layer_outputs = layer_module( 2025-12-04T08:56:06.9823998Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-12-04T08:56:06.9824146Z intermediate_output = self.intermediate(attention_output) 2025-12-04T08:56:06.9824460Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T08:56:06.9824554Z hidden_states = self.dense(hidden_states) 2025-12-04T08:56:06.9824557Z 2025-12-04T08:56:06.9824680Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:06.9824903Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:06.9824983Z res = mod(**inputs) 2025-12-04T08:56:06.9825295Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:06.9825376Z outputs = self.mobilebert( 2025-12-04T08:56:06.9825701Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:06.9825784Z encoder_outputs = self.encoder( 2025-12-04T08:56:06.9826225Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:06.9826356Z layer_outputs = layer_module( 2025-12-04T08:56:06.9826799Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-12-04T08:56:06.9827036Z intermediate_output = self.intermediate(attention_output) 2025-12-04T08:56:06.9827386Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T08:56:06.9827510Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:56:06.9827516Z 2025-12-04T08:56:06.9827640Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:06.9827891Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:06.9828012Z res = mod(**inputs) 2025-12-04T08:56:06.9828331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:06.9828410Z outputs = self.mobilebert( 2025-12-04T08:56:06.9828737Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:06.9828822Z encoder_outputs = self.encoder( 2025-12-04T08:56:06.9829140Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:06.9829227Z layer_outputs = layer_module( 2025-12-04T08:56:06.9829546Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T08:56:06.9829758Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T08:56:06.9830074Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-12-04T08:56:06.9830182Z layer_output = self.dense(intermediate_states) 2025-12-04T08:56:06.9830186Z 2025-12-04T08:56:06.9830308Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:06.9830529Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:06.9830609Z res = mod(**inputs) 2025-12-04T08:56:06.9830921Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:06.9831002Z outputs = self.mobilebert( 2025-12-04T08:56:06.9831325Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:06.9831408Z encoder_outputs = self.encoder( 2025-12-04T08:56:06.9831725Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:06.9831811Z layer_outputs = layer_module( 2025-12-04T08:56:06.9832123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T08:56:06.9832306Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T08:56:06.9832621Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-12-04T08:56:06.9832759Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-12-04T08:56:06.9833083Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:06.9833188Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:06.9833194Z 2025-12-04T08:56:06.9833315Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:06.9833535Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:06.9833605Z res = mod(**inputs) 2025-12-04T08:56:06.9833925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:06.9834003Z outputs = self.mobilebert( 2025-12-04T08:56:06.9834370Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:06.9834453Z encoder_outputs = self.encoder( 2025-12-04T08:56:06.9834773Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:06.9834876Z layer_outputs = layer_module( 2025-12-04T08:56:06.9835227Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T08:56:06.9835398Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T08:56:06.9835705Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-12-04T08:56:06.9835836Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-12-04T08:56:06.9836157Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-12-04T08:56:06.9836250Z layer_outputs = self.dense(hidden_states) 2025-12-04T08:56:06.9836253Z 2025-12-04T08:56:06.9836369Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:06.9836617Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:06.9836690Z res = mod(**inputs) 2025-12-04T08:56:06.9837012Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:06.9837089Z outputs = self.mobilebert( 2025-12-04T08:56:06.9837405Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:06.9837495Z encoder_outputs = self.encoder( 2025-12-04T08:56:06.9837810Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:06.9837890Z layer_outputs = layer_module( 2025-12-04T08:56:06.9838279Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T08:56:06.9838462Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T08:56:06.9838782Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-12-04T08:56:06.9838915Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-12-04T08:56:06.9839225Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-12-04T08:56:06.9839365Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T08:56:06.9839684Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:06.9839789Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:06.9839793Z 2025-12-04T08:56:06.9839904Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:06.9840114Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:06.9840195Z res = mod(**inputs) 2025-12-04T08:56:06.9840500Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:06.9840583Z outputs = self.mobilebert( 2025-12-04T08:56:06.9840881Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:06.9840960Z encoder_outputs = self.encoder( 2025-12-04T08:56:06.9841295Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:06.9841374Z layer_outputs = layer_module( 2025-12-04T08:56:06.9841679Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T08:56:06.9841865Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T08:56:06.9842209Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-12-04T08:56:06.9842342Z shared_attention_input = self.attention(hidden_states) 2025-12-04T08:56:06.9842644Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-12-04T08:56:06.9842737Z layer_input = self.dense(hidden_states) 2025-12-04T08:56:06.9842741Z 2025-12-04T08:56:06.9842864Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:06.9843079Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:06.9843158Z res = mod(**inputs) 2025-12-04T08:56:06.9843461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:06.9843567Z outputs = self.mobilebert( 2025-12-04T08:56:06.9843880Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:06.9843959Z encoder_outputs = self.encoder( 2025-12-04T08:56:06.9844259Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:06.9844344Z layer_outputs = layer_module( 2025-12-04T08:56:06.9844649Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T08:56:06.9844751Z self_attention_outputs = self.attention( 2025-12-04T08:56:06.9845055Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T08:56:06.9845137Z self_outputs = self.self( 2025-12-04T08:56:06.9845450Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-12-04T08:56:06.9845530Z self.value(value_tensor) 2025-12-04T08:56:06.9845534Z 2025-12-04T08:56:06.9845649Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:06.9845863Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:06.9845932Z res = mod(**inputs) 2025-12-04T08:56:06.9846256Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:06.9846332Z outputs = self.mobilebert( 2025-12-04T08:56:06.9846634Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:06.9846721Z encoder_outputs = self.encoder( 2025-12-04T08:56:06.9847031Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:06.9847116Z layer_outputs = layer_module( 2025-12-04T08:56:06.9847423Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T08:56:06.9847594Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T08:56:06.9847908Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-12-04T08:56:06.9848054Z bottlenecked_hidden_states = self.input(hidden_states) 2025-12-04T08:56:06.9848366Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-12-04T08:56:06.9848455Z layer_input = self.dense(hidden_states) 2025-12-04T08:56:06.9848461Z 2025-12-04T08:56:06.9848571Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:06.9848816Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:06.9848905Z res = mod(**inputs) 2025-12-04T08:56:06.9849215Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:06.9849289Z outputs = self.mobilebert( 2025-12-04T08:56:06.9849589Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:06.9849673Z encoder_outputs = self.encoder( 2025-12-04T08:56:06.9849980Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:06.9850056Z layer_outputs = layer_module( 2025-12-04T08:56:06.9850363Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T08:56:06.9850559Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T08:56:06.9850876Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-12-04T08:56:06.9850995Z shared_attention_input = self.attention(hidden_states) 2025-12-04T08:56:06.9851300Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-12-04T08:56:06.9851402Z layer_input = self.LayerNorm(layer_input) 2025-12-04T08:56:06.9851706Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:06.9851811Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:06.9851815Z 2025-12-04T08:56:06.9851927Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:06.9852140Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:06.9852218Z res = mod(**inputs) 2025-12-04T08:56:06.9852521Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:06.9852595Z outputs = self.mobilebert( 2025-12-04T08:56:06.9852906Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:06.9852982Z encoder_outputs = self.encoder( 2025-12-04T08:56:06.9853294Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:06.9853370Z layer_outputs = layer_module( 2025-12-04T08:56:06.9853672Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T08:56:06.9853773Z self_attention_outputs = self.attention( 2025-12-04T08:56:06.9854078Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T08:56:06.9854162Z self_outputs = self.self( 2025-12-04T08:56:06.9854465Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-12-04T08:56:06.9854541Z self.query(query_tensor) 2025-12-04T08:56:06.9854545Z 2025-12-04T08:56:06.9854662Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:06.9854904Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:06.9854974Z res = mod(**inputs) 2025-12-04T08:56:06.9855283Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:06.9855358Z outputs = self.mobilebert( 2025-12-04T08:56:06.9855688Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:06.9855787Z encoder_outputs = self.encoder( 2025-12-04T08:56:06.9856095Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:06.9856178Z layer_outputs = layer_module( 2025-12-04T08:56:06.9856483Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T08:56:06.9856591Z self_attention_outputs = self.attention( 2025-12-04T08:56:06.9856882Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T08:56:06.9856954Z self_outputs = self.self( 2025-12-04T08:56:06.9857283Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-12-04T08:56:06.9857360Z self.key(key_tensor) 2025-12-04T08:56:06.9857363Z 2025-12-04T08:56:06.9857453Z cudagraph partition due to non gpu ops 2025-12-04T08:56:06.9857549Z cudagraph partition due to non gpu ops 2025-12-04T08:56:06.9857658Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:06.9857880Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:06.9857950Z res = mod(**inputs) 2025-12-04T08:56:06.9858256Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:06.9858340Z outputs = self.mobilebert( 2025-12-04T08:56:06.9858646Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:06.9858734Z encoder_outputs = self.encoder( 2025-12-04T08:56:06.9859041Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:06.9859120Z layer_outputs = layer_module( 2025-12-04T08:56:06.9859429Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T08:56:06.9859521Z self_attention_outputs = self.attention( 2025-12-04T08:56:06.9859828Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-12-04T08:56:06.9859974Z attention_output = self.output(self_outputs[0], layer_input) 2025-12-04T08:56:06.9860282Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-12-04T08:56:06.9860379Z layer_outputs = self.dense(hidden_states) 2025-12-04T08:56:06.9860384Z 2025-12-04T08:56:06.9860493Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:06.9860705Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:06.9860785Z res = mod(**inputs) 2025-12-04T08:56:06.9861090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:06.9861173Z outputs = self.mobilebert( 2025-12-04T08:56:06.9861475Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:06.9861579Z encoder_outputs = self.encoder( 2025-12-04T08:56:06.9861896Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:06.9861972Z layer_outputs = layer_module( 2025-12-04T08:56:06.9862279Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T08:56:06.9862411Z self_attention_outputs = self.attention( 2025-12-04T08:56:06.9862710Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-12-04T08:56:06.9862842Z attention_output = self.output(self_outputs[0], layer_input) 2025-12-04T08:56:06.9863130Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-12-04T08:56:06.9863266Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T08:56:06.9863579Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:06.9863678Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:06.9863700Z 2025-12-04T08:56:06.9863817Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:06.9864033Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:06.9864104Z res = mod(**inputs) 2025-12-04T08:56:06.9864428Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:06.9864505Z outputs = self.mobilebert( 2025-12-04T08:56:06.9864808Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:06.9864894Z encoder_outputs = self.encoder( 2025-12-04T08:56:06.9865199Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:06.9865282Z layer_outputs = layer_module( 2025-12-04T08:56:06.9865586Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:06.9865692Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:06.9866006Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T08:56:06.9866129Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T08:56:06.9866438Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T08:56:06.9866527Z hidden_states = self.dense(hidden_states) 2025-12-04T08:56:06.9866531Z 2025-12-04T08:56:06.9866640Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:06.9866863Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:06.9866933Z res = mod(**inputs) 2025-12-04T08:56:06.9867237Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:06.9867323Z outputs = self.mobilebert( 2025-12-04T08:56:06.9867629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:06.9867715Z encoder_outputs = self.encoder( 2025-12-04T08:56:06.9868017Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:06.9868094Z layer_outputs = layer_module( 2025-12-04T08:56:06.9868427Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:06.9868531Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:06.9868838Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T08:56:06.9868959Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T08:56:06.9869282Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T08:56:06.9869442Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:56:06.9869446Z 2025-12-04T08:56:06.9869554Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:06.9869776Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:06.9869846Z res = mod(**inputs) 2025-12-04T08:56:06.9870154Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:06.9870236Z outputs = self.mobilebert( 2025-12-04T08:56:06.9870544Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:06.9870644Z encoder_outputs = self.encoder( 2025-12-04T08:56:06.9870957Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:06.9871034Z layer_outputs = layer_module( 2025-12-04T08:56:06.9871345Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:06.9871445Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:06.9871747Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T08:56:06.9871890Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T08:56:06.9872195Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T08:56:06.9872294Z layer_outputs = self.dense(hidden_states) 2025-12-04T08:56:06.9872298Z 2025-12-04T08:56:06.9872409Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:06.9872625Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:06.9872700Z res = mod(**inputs) 2025-12-04T08:56:06.9873006Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:06.9873083Z outputs = self.mobilebert( 2025-12-04T08:56:06.9873395Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:06.9873474Z encoder_outputs = self.encoder( 2025-12-04T08:56:06.9873785Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:06.9873861Z layer_outputs = layer_module( 2025-12-04T08:56:06.9874168Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:06.9874276Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:06.9874580Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T08:56:06.9874722Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T08:56:06.9875026Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T08:56:06.9875180Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T08:56:06.9875497Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:06.9875594Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:06.9875599Z 2025-12-04T08:56:06.9875716Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:06.9875948Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:06.9876036Z res = mod(**inputs) 2025-12-04T08:56:06.9876356Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:06.9876430Z outputs = self.mobilebert( 2025-12-04T08:56:06.9876730Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:06.9876818Z encoder_outputs = self.encoder( 2025-12-04T08:56:06.9877120Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:06.9877204Z layer_outputs = layer_module( 2025-12-04T08:56:06.9877506Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:06.9877650Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:06.9877971Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T08:56:06.9878169Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T08:56:06.9878498Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T08:56:06.9878590Z hidden_states = self.dense(hidden_states) 2025-12-04T08:56:06.9878597Z 2025-12-04T08:56:06.9878711Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:06.9878939Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:06.9879012Z res = mod(**inputs) 2025-12-04T08:56:06.9879340Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:06.9879428Z outputs = self.mobilebert( 2025-12-04T08:56:06.9879732Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:06.9879819Z encoder_outputs = self.encoder( 2025-12-04T08:56:06.9880125Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:06.9880221Z layer_outputs = layer_module( 2025-12-04T08:56:06.9880537Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:06.9880638Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:06.9880951Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T08:56:06.9881074Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T08:56:06.9881383Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T08:56:06.9881512Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:56:06.9881515Z 2025-12-04T08:56:06.9881625Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:06.9881844Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:06.9881913Z res = mod(**inputs) 2025-12-04T08:56:06.9882247Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:06.9882333Z outputs = self.mobilebert( 2025-12-04T08:56:06.9882646Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:06.9882727Z encoder_outputs = self.encoder( 2025-12-04T08:56:06.9883071Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:06.9883169Z layer_outputs = layer_module( 2025-12-04T08:56:06.9883497Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:06.9883600Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:06.9883915Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T08:56:06.9884061Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T08:56:06.9884381Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T08:56:06.9884508Z layer_outputs = self.dense(hidden_states) 2025-12-04T08:56:06.9884512Z 2025-12-04T08:56:06.9884626Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:06.9884847Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:06.9884926Z res = mod(**inputs) 2025-12-04T08:56:06.9885240Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:06.9885316Z outputs = self.mobilebert( 2025-12-04T08:56:06.9885638Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:06.9885718Z encoder_outputs = self.encoder( 2025-12-04T08:56:06.9886038Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:06.9886118Z layer_outputs = layer_module( 2025-12-04T08:56:06.9886432Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:06.9886543Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:06.9886854Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T08:56:06.9886995Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T08:56:06.9887305Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T08:56:06.9887441Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T08:56:06.9887757Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:06.9887857Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:06.9887862Z 2025-12-04T08:56:06.9887987Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:06.9888207Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:06.9888273Z res = mod(**inputs) 2025-12-04T08:56:06.9888582Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:06.9888658Z outputs = self.mobilebert( 2025-12-04T08:56:06.9888959Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:06.9889059Z encoder_outputs = self.encoder( 2025-12-04T08:56:06.9889361Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:06.9889444Z layer_outputs = layer_module( 2025-12-04T08:56:06.9889827Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:06.9889951Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:06.9890272Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T08:56:06.9890393Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T08:56:06.9890706Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T08:56:06.9890809Z hidden_states = self.dense(hidden_states) 2025-12-04T08:56:06.9890813Z 2025-12-04T08:56:06.9890925Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:06.9891151Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:06.9891223Z res = mod(**inputs) 2025-12-04T08:56:06.9891556Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:06.9891642Z outputs = self.mobilebert( 2025-12-04T08:56:06.9891955Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:06.9892041Z encoder_outputs = self.encoder( 2025-12-04T08:56:06.9892357Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:06.9892435Z layer_outputs = layer_module( 2025-12-04T08:56:06.9892759Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:06.9892863Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:06.9893174Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T08:56:06.9893308Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T08:56:06.9893621Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T08:56:06.9893751Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:56:06.9893755Z 2025-12-04T08:56:06.9893867Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:06.9894087Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:06.9894166Z res = mod(**inputs) 2025-12-04T08:56:06.9894480Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:06.9894565Z outputs = self.mobilebert( 2025-12-04T08:56:06.9894878Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:06.9894960Z encoder_outputs = self.encoder( 2025-12-04T08:56:06.9895285Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:06.9895364Z layer_outputs = layer_module( 2025-12-04T08:56:06.9895685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:06.9895788Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:06.9896135Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T08:56:06.9896282Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T08:56:06.9896598Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T08:56:06.9896693Z layer_outputs = self.dense(hidden_states) 2025-12-04T08:56:06.9896730Z 2025-12-04T08:56:06.9896863Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:06.9897083Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:06.9897163Z res = mod(**inputs) 2025-12-04T08:56:06.9897478Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:06.9897553Z outputs = self.mobilebert( 2025-12-04T08:56:06.9897871Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:06.9897950Z encoder_outputs = self.encoder( 2025-12-04T08:56:06.9898265Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:06.9898364Z layer_outputs = layer_module( 2025-12-04T08:56:06.9898682Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:06.9898792Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:06.9899104Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T08:56:06.9899240Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T08:56:06.9899562Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T08:56:06.9899696Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T08:56:06.9900015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:06.9900117Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:06.9900121Z 2025-12-04T08:56:06.9900235Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:06.9900461Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:06.9900532Z res = mod(**inputs) 2025-12-04T08:56:06.9900853Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:06.9900929Z outputs = self.mobilebert( 2025-12-04T08:56:06.9901242Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:06.9901329Z encoder_outputs = self.encoder( 2025-12-04T08:56:06.9901637Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:06.9901716Z layer_outputs = layer_module( 2025-12-04T08:56:06.9902038Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-12-04T08:56:06.9902175Z intermediate_output = self.intermediate(attention_output) 2025-12-04T08:56:06.9902491Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T08:56:06.9902583Z hidden_states = self.dense(hidden_states) 2025-12-04T08:56:06.9902587Z 2025-12-04T08:56:06.9902699Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:06.9902946Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:06.9903021Z res = mod(**inputs) 2025-12-04T08:56:06.9903343Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:06.9903423Z outputs = self.mobilebert( 2025-12-04T08:56:06.9903755Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:06.9903860Z encoder_outputs = self.encoder( 2025-12-04T08:56:06.9904170Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:06.9904255Z layer_outputs = layer_module( 2025-12-04T08:56:06.9904566Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-12-04T08:56:06.9904701Z intermediate_output = self.intermediate(attention_output) 2025-12-04T08:56:06.9905019Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T08:56:06.9905139Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:56:06.9905163Z 2025-12-04T08:56:06.9905278Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:06.9905511Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:06.9905582Z res = mod(**inputs) 2025-12-04T08:56:06.9905900Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:06.9905976Z outputs = self.mobilebert( 2025-12-04T08:56:06.9906287Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:06.9906373Z encoder_outputs = self.encoder( 2025-12-04T08:56:06.9906689Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:06.9906773Z layer_outputs = layer_module( 2025-12-04T08:56:06.9907085Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T08:56:06.9907265Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T08:56:06.9907587Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-12-04T08:56:06.9907691Z layer_output = self.dense(intermediate_states) 2025-12-04T08:56:06.9907695Z 2025-12-04T08:56:06.9907818Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:06.9908039Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:06.9908109Z res = mod(**inputs) 2025-12-04T08:56:06.9908427Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:06.9908507Z outputs = self.mobilebert( 2025-12-04T08:56:06.9908823Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:06.9908914Z encoder_outputs = self.encoder( 2025-12-04T08:56:06.9909228Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:06.9909313Z layer_outputs = layer_module( 2025-12-04T08:56:06.9909628Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T08:56:06.9909803Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T08:56:06.9910146Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-12-04T08:56:06.9910286Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-12-04T08:56:06.9910607Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:06.9910729Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:06.9910751Z 2025-12-04T08:56:06.9910866Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:06.9911090Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:06.9911160Z res = mod(**inputs) 2025-12-04T08:56:06.9911470Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:06.9911555Z outputs = self.mobilebert( 2025-12-04T08:56:06.9911882Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:06.9911970Z encoder_outputs = self.encoder( 2025-12-04T08:56:06.9912287Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:06.9912391Z layer_outputs = layer_module( 2025-12-04T08:56:06.9912716Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T08:56:06.9912889Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T08:56:06.9913210Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-12-04T08:56:06.9913347Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-12-04T08:56:06.9913661Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-12-04T08:56:06.9913763Z layer_outputs = self.dense(hidden_states) 2025-12-04T08:56:06.9913767Z 2025-12-04T08:56:06.9913880Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:06.9914109Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:06.9914183Z res = mod(**inputs) 2025-12-04T08:56:06.9914495Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:06.9914579Z outputs = self.mobilebert( 2025-12-04T08:56:06.9914894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:06.9914974Z encoder_outputs = self.encoder( 2025-12-04T08:56:06.9915297Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:06.9915375Z layer_outputs = layer_module( 2025-12-04T08:56:06.9915693Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T08:56:06.9915868Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T08:56:06.9916184Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-12-04T08:56:06.9916326Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-12-04T08:56:06.9916640Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-12-04T08:56:06.9916782Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T08:56:06.9917124Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:06.9917227Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:06.9917231Z 2025-12-04T08:56:06.9917350Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:06.9917570Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:06.9917657Z res = mod(**inputs) 2025-12-04T08:56:06.9917995Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:06.9918135Z outputs = self.mobilebert( 2025-12-04T08:56:06.9918466Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:06.9918550Z encoder_outputs = self.encoder( 2025-12-04T08:56:06.9918868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:06.9918957Z layer_outputs = layer_module( 2025-12-04T08:56:06.9919274Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T08:56:06.9919487Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T08:56:06.9920010Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-12-04T08:56:06.9920141Z shared_attention_input = self.attention(hidden_states) 2025-12-04T08:56:06.9920465Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-12-04T08:56:06.9920558Z layer_input = self.dense(hidden_states) 2025-12-04T08:56:06.9920563Z 2025-12-04T08:56:06.9920825Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:06.9921060Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:06.9921131Z res = mod(**inputs) 2025-12-04T08:56:06.9921452Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:06.9921534Z outputs = self.mobilebert( 2025-12-04T08:56:06.9921853Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:06.9921944Z encoder_outputs = self.encoder( 2025-12-04T08:56:06.9922257Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:06.9922345Z layer_outputs = layer_module( 2025-12-04T08:56:06.9922673Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T08:56:06.9922770Z self_attention_outputs = self.attention( 2025-12-04T08:56:06.9923093Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T08:56:06.9923175Z self_outputs = self.self( 2025-12-04T08:56:06.9923488Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-12-04T08:56:06.9923578Z self.value(value_tensor) 2025-12-04T08:56:06.9923582Z 2025-12-04T08:56:06.9923696Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:06.9923921Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:06.9923992Z res = mod(**inputs) 2025-12-04T08:56:06.9924302Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:06.9924453Z outputs = self.mobilebert( 2025-12-04T08:56:06.9924766Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:06.9924861Z encoder_outputs = self.encoder( 2025-12-04T08:56:06.9925192Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:06.9925293Z layer_outputs = layer_module( 2025-12-04T08:56:06.9925601Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T08:56:06.9925773Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T08:56:06.9926094Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-12-04T08:56:06.9926214Z bottlenecked_hidden_states = self.input(hidden_states) 2025-12-04T08:56:06.9926516Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-12-04T08:56:06.9926613Z layer_input = self.dense(hidden_states) 2025-12-04T08:56:06.9926645Z 2025-12-04T08:56:06.9926757Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:06.9926973Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:06.9927051Z res = mod(**inputs) 2025-12-04T08:56:06.9927356Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:06.9927439Z outputs = self.mobilebert( 2025-12-04T08:56:06.9927741Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:06.9927819Z encoder_outputs = self.encoder( 2025-12-04T08:56:06.9928132Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:06.9928207Z layer_outputs = layer_module( 2025-12-04T08:56:06.9928516Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T08:56:06.9928689Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T08:56:06.9928995Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-12-04T08:56:06.9929121Z shared_attention_input = self.attention(hidden_states) 2025-12-04T08:56:06.9929424Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-12-04T08:56:06.9929517Z layer_input = self.LayerNorm(layer_input) 2025-12-04T08:56:06.9929834Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:06.9929933Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:06.9929936Z 2025-12-04T08:56:06.9930054Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:06.9930270Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:06.9930342Z res = mod(**inputs) 2025-12-04T08:56:06.9930650Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:06.9930726Z outputs = self.mobilebert( 2025-12-04T08:56:06.9931034Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:06.9931112Z encoder_outputs = self.encoder( 2025-12-04T08:56:06.9931448Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:06.9931535Z layer_outputs = layer_module( 2025-12-04T08:56:06.9931836Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T08:56:06.9931928Z self_attention_outputs = self.attention( 2025-12-04T08:56:06.9932266Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T08:56:06.9932361Z self_outputs = self.self( 2025-12-04T08:56:06.9932678Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-12-04T08:56:06.9932756Z self.query(query_tensor) 2025-12-04T08:56:06.9932759Z 2025-12-04T08:56:06.9932870Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:06.9933091Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:06.9933160Z res = mod(**inputs) 2025-12-04T08:56:06.9933477Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:06.9933572Z outputs = self.mobilebert( 2025-12-04T08:56:06.9933887Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:06.9933975Z encoder_outputs = self.encoder( 2025-12-04T08:56:06.9934285Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:06.9934359Z layer_outputs = layer_module( 2025-12-04T08:56:06.9934679Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T08:56:06.9934771Z self_attention_outputs = self.attention( 2025-12-04T08:56:06.9935089Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T08:56:06.9935164Z self_outputs = self.self( 2025-12-04T08:56:06.9935482Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-12-04T08:56:06.9935567Z self.key(key_tensor) 2025-12-04T08:56:06.9935571Z 2025-12-04T08:56:06.9935658Z cudagraph partition due to non gpu ops 2025-12-04T08:56:06.9935753Z cudagraph partition due to non gpu ops 2025-12-04T08:56:06.9935865Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:06.9936075Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:06.9936151Z res = mod(**inputs) 2025-12-04T08:56:06.9936456Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:06.9936531Z outputs = self.mobilebert( 2025-12-04T08:56:06.9936851Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:06.9936931Z encoder_outputs = self.encoder( 2025-12-04T08:56:06.9937250Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:06.9937326Z layer_outputs = layer_module( 2025-12-04T08:56:06.9937638Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T08:56:06.9937735Z self_attention_outputs = self.attention( 2025-12-04T08:56:06.9938054Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-12-04T08:56:06.9938210Z attention_output = self.output(self_outputs[0], layer_input) 2025-12-04T08:56:06.9938527Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-12-04T08:56:06.9938617Z layer_outputs = self.dense(hidden_states) 2025-12-04T08:56:06.9938623Z 2025-12-04T08:56:06.9938739Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:06.9938970Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:06.9939058Z res = mod(**inputs) 2025-12-04T08:56:06.9939372Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:06.9939449Z outputs = self.mobilebert( 2025-12-04T08:56:06.9939762Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:06.9939842Z encoder_outputs = self.encoder( 2025-12-04T08:56:06.9940144Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:06.9940228Z layer_outputs = layer_module( 2025-12-04T08:56:06.9940552Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T08:56:06.9940652Z self_attention_outputs = self.attention( 2025-12-04T08:56:06.9940955Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-12-04T08:56:06.9941109Z attention_output = self.output(self_outputs[0], layer_input) 2025-12-04T08:56:06.9941424Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-12-04T08:56:06.9941561Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T08:56:06.9941862Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:06.9941967Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:06.9941971Z 2025-12-04T08:56:06.9942083Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:06.9942304Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:06.9942374Z res = mod(**inputs) 2025-12-04T08:56:06.9942680Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:06.9942762Z outputs = self.mobilebert( 2025-12-04T08:56:06.9943063Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:06.9943146Z encoder_outputs = self.encoder( 2025-12-04T08:56:06.9943456Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:06.9943533Z layer_outputs = layer_module( 2025-12-04T08:56:06.9943839Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:06.9943945Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:06.9944250Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T08:56:06.9944377Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T08:56:06.9944688Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T08:56:06.9944784Z hidden_states = self.dense(hidden_states) 2025-12-04T08:56:06.9944788Z 2025-12-04T08:56:06.9944917Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:06.9945130Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:06.9945207Z res = mod(**inputs) 2025-12-04T08:56:06.9945510Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:06.9945612Z outputs = self.mobilebert( 2025-12-04T08:56:06.9945962Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:06.9946039Z encoder_outputs = self.encoder( 2025-12-04T08:56:06.9946352Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:06.9946429Z layer_outputs = layer_module( 2025-12-04T08:56:06.9946740Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:06.9946847Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:06.9947147Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T08:56:06.9947303Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T08:56:06.9947609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T08:56:06.9947731Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:56:06.9947735Z 2025-12-04T08:56:06.9947853Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:06.9948067Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:06.9948143Z res = mod(**inputs) 2025-12-04T08:56:06.9948445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:06.9948520Z outputs = self.mobilebert( 2025-12-04T08:56:06.9948829Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:06.9948909Z encoder_outputs = self.encoder( 2025-12-04T08:56:06.9949225Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:06.9949303Z layer_outputs = layer_module( 2025-12-04T08:56:06.9949604Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:06.9949712Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:06.9950017Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T08:56:06.9950154Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T08:56:06.9950466Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T08:56:06.9950556Z layer_outputs = self.dense(hidden_states) 2025-12-04T08:56:06.9950561Z 2025-12-04T08:56:06.9950679Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:06.9950891Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:06.9950960Z res = mod(**inputs) 2025-12-04T08:56:06.9951268Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:06.9951343Z outputs = self.mobilebert( 2025-12-04T08:56:06.9951653Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:06.9951748Z encoder_outputs = self.encoder( 2025-12-04T08:56:06.9952050Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:06.9952135Z layer_outputs = layer_module( 2025-12-04T08:56:06.9952458Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:06.9952589Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:06.9952899Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T08:56:06.9953031Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T08:56:06.9953342Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T08:56:06.9953474Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T08:56:06.9953776Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:06.9953880Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:06.9953901Z 2025-12-04T08:56:06.9954013Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:06.9954234Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:06.9954305Z res = mod(**inputs) 2025-12-04T08:56:06.9954609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:06.9954692Z outputs = self.mobilebert( 2025-12-04T08:56:06.9954995Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:06.9955081Z encoder_outputs = self.encoder( 2025-12-04T08:56:06.9955387Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:06.9955464Z layer_outputs = layer_module( 2025-12-04T08:56:06.9955773Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:06.9955875Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:06.9956182Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T08:56:06.9956310Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T08:56:06.9956617Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T08:56:06.9956712Z hidden_states = self.dense(hidden_states) 2025-12-04T08:56:06.9956716Z 2025-12-04T08:56:06.9956826Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:06.9957037Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:06.9957113Z res = mod(**inputs) 2025-12-04T08:56:06.9957420Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:06.9957508Z outputs = self.mobilebert( 2025-12-04T08:56:06.9957820Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:06.9957899Z encoder_outputs = self.encoder( 2025-12-04T08:56:06.9958306Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:06.9958393Z layer_outputs = layer_module( 2025-12-04T08:56:06.9958734Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:06.9958848Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:06.9959157Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T08:56:06.9959288Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T08:56:06.9959618Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T08:56:06.9959765Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:56:06.9959769Z 2025-12-04T08:56:06.9959891Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:06.9960113Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:06.9960193Z res = mod(**inputs) 2025-12-04T08:56:06.9960509Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:06.9960588Z outputs = self.mobilebert( 2025-12-04T08:56:06.9960909Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:06.9961007Z encoder_outputs = self.encoder( 2025-12-04T08:56:06.9961318Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:06.9961407Z layer_outputs = layer_module( 2025-12-04T08:56:06.9961722Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:06.9961831Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:06.9962145Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T08:56:06.9962285Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T08:56:06.9962604Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T08:56:06.9962699Z layer_outputs = self.dense(hidden_states) 2025-12-04T08:56:06.9962703Z 2025-12-04T08:56:06.9962824Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:06.9963047Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:06.9963117Z res = mod(**inputs) 2025-12-04T08:56:06.9963435Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:06.9963512Z outputs = self.mobilebert( 2025-12-04T08:56:06.9963837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:06.9963925Z encoder_outputs = self.encoder( 2025-12-04T08:56:06.9964234Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:06.9964322Z layer_outputs = layer_module( 2025-12-04T08:56:06.9964634Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:06.9964739Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:06.9965057Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T08:56:06.9965195Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T08:56:06.9965520Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T08:56:06.9965676Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T08:56:06.9965989Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:06.9966100Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:06.9966105Z 2025-12-04T08:56:06.9966219Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:06.9966496Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:06.9966569Z res = mod(**inputs) 2025-12-04T08:56:06.9966891Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:06.9966978Z outputs = self.mobilebert( 2025-12-04T08:56:06.9967289Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:06.9967371Z encoder_outputs = self.encoder( 2025-12-04T08:56:06.9967692Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:06.9967770Z layer_outputs = layer_module( 2025-12-04T08:56:06.9968108Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:06.9968212Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:06.9968537Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T08:56:06.9968668Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T08:56:06.9968993Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T08:56:06.9969095Z hidden_states = self.dense(hidden_states) 2025-12-04T08:56:06.9969099Z 2025-12-04T08:56:06.9969211Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:06.9969430Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:06.9969509Z res = mod(**inputs) 2025-12-04T08:56:06.9969823Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:06.9969903Z outputs = self.mobilebert( 2025-12-04T08:56:06.9970224Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:06.9970304Z encoder_outputs = self.encoder( 2025-12-04T08:56:06.9970628Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:06.9970706Z layer_outputs = layer_module( 2025-12-04T08:56:06.9971026Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:06.9971137Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:06.9971459Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T08:56:06.9971591Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T08:56:06.9971918Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T08:56:06.9972039Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:56:06.9972043Z 2025-12-04T08:56:06.9972164Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:06.9972385Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:06.9972464Z res = mod(**inputs) 2025-12-04T08:56:06.9972805Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:06.9972884Z outputs = self.mobilebert( 2025-12-04T08:56:06.9973206Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:06.9973307Z encoder_outputs = self.encoder( 2025-12-04T08:56:06.9973667Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:06.9973754Z layer_outputs = layer_module( 2025-12-04T08:56:06.9974075Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:06.9974183Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:06.9974501Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T08:56:06.9974639Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T08:56:06.9974961Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T08:56:06.9975077Z layer_outputs = self.dense(hidden_states) 2025-12-04T08:56:06.9975080Z 2025-12-04T08:56:06.9975204Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:06.9975424Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:06.9975494Z res = mod(**inputs) 2025-12-04T08:56:06.9975810Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:06.9975888Z outputs = self.mobilebert( 2025-12-04T08:56:06.9976210Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:06.9976298Z encoder_outputs = self.encoder( 2025-12-04T08:56:06.9976624Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:06.9976711Z layer_outputs = layer_module( 2025-12-04T08:56:06.9977033Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:06.9977137Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:06.9977455Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T08:56:06.9977588Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T08:56:06.9977906Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T08:56:06.9978042Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T08:56:06.9978354Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:06.9978464Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:06.9978468Z 2025-12-04T08:56:06.9978581Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:06.9978806Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:06.9978877Z res = mod(**inputs) 2025-12-04T08:56:06.9979188Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:06.9979272Z outputs = self.mobilebert( 2025-12-04T08:56:06.9979609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:06.9979690Z encoder_outputs = self.encoder( 2025-12-04T08:56:06.9980010Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:06.9980088Z layer_outputs = layer_module( 2025-12-04T08:56:06.9980427Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-12-04T08:56:06.9980584Z intermediate_output = self.intermediate(attention_output) 2025-12-04T08:56:06.9980899Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T08:56:06.9980998Z hidden_states = self.dense(hidden_states) 2025-12-04T08:56:06.9981002Z 2025-12-04T08:56:06.9981115Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:06.9981345Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:06.9981416Z res = mod(**inputs) 2025-12-04T08:56:06.9981734Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:06.9981817Z outputs = self.mobilebert( 2025-12-04T08:56:06.9982148Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:06.9982229Z encoder_outputs = self.encoder( 2025-12-04T08:56:06.9982547Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:06.9982624Z layer_outputs = layer_module( 2025-12-04T08:56:06.9982940Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-12-04T08:56:06.9983076Z intermediate_output = self.intermediate(attention_output) 2025-12-04T08:56:06.9983384Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T08:56:06.9983513Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:56:06.9983519Z 2025-12-04T08:56:06.9983629Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:06.9983856Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:06.9983929Z res = mod(**inputs) 2025-12-04T08:56:06.9984237Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:06.9984322Z outputs = self.mobilebert( 2025-12-04T08:56:06.9984633Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:06.9984713Z encoder_outputs = self.encoder( 2025-12-04T08:56:06.9985038Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:06.9985114Z layer_outputs = layer_module( 2025-12-04T08:56:06.9985437Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T08:56:06.9985614Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T08:56:06.9985926Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-12-04T08:56:06.9986037Z layer_output = self.dense(intermediate_states) 2025-12-04T08:56:06.9986041Z 2025-12-04T08:56:06.9986155Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:06.9986379Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:06.9986467Z res = mod(**inputs) 2025-12-04T08:56:06.9986780Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:06.9986865Z outputs = self.mobilebert( 2025-12-04T08:56:06.9987176Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:06.9987277Z encoder_outputs = self.encoder( 2025-12-04T08:56:06.9987616Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:06.9987695Z layer_outputs = layer_module( 2025-12-04T08:56:06.9988012Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T08:56:06.9988185Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T08:56:06.9988511Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-12-04T08:56:06.9988655Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-12-04T08:56:06.9988970Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:06.9989106Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:06.9989111Z 2025-12-04T08:56:06.9989225Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:06.9989443Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:06.9989522Z res = mod(**inputs) 2025-12-04T08:56:06.9989834Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:06.9989919Z outputs = self.mobilebert( 2025-12-04T08:56:06.9990234Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:06.9990314Z encoder_outputs = self.encoder( 2025-12-04T08:56:06.9990634Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:06.9990714Z layer_outputs = layer_module( 2025-12-04T08:56:06.9991029Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T08:56:06.9991209Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T08:56:06.9991524Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-12-04T08:56:06.9991668Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-12-04T08:56:06.9991989Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-12-04T08:56:06.9992082Z layer_outputs = self.dense(hidden_states) 2025-12-04T08:56:06.9992086Z 2025-12-04T08:56:06.9992207Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:06.9992432Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:06.9992512Z res = mod(**inputs) 2025-12-04T08:56:06.9992826Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:06.9992902Z outputs = self.mobilebert( 2025-12-04T08:56:06.9993219Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:06.9993297Z encoder_outputs = self.encoder( 2025-12-04T08:56:06.9993630Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:06.9993719Z layer_outputs = layer_module( 2025-12-04T08:56:06.9994038Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T08:56:06.9994224Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T08:56:06.9994574Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-12-04T08:56:06.9994709Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-12-04T08:56:06.9995029Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-12-04T08:56:06.9995164Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T08:56:06.9995484Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:06.9995583Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:06.9995587Z 2025-12-04T08:56:06.9995700Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:06.9995945Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:06.9996018Z res = mod(**inputs) 2025-12-04T08:56:06.9996336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:06.9996412Z outputs = self.mobilebert( 2025-12-04T08:56:06.9996722Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:06.9996807Z encoder_outputs = self.encoder( 2025-12-04T08:56:06.9997119Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:06.9997197Z layer_outputs = layer_module( 2025-12-04T08:56:06.9997515Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T08:56:06.9997697Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T08:56:06.9998017Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-12-04T08:56:06.9998220Z shared_attention_input = self.attention(hidden_states) 2025-12-04T08:56:06.9998536Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-12-04T08:56:06.9998637Z layer_input = self.dense(hidden_states) 2025-12-04T08:56:06.9998643Z 2025-12-04T08:56:06.9998760Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:06.9998985Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:06.9999055Z res = mod(**inputs) 2025-12-04T08:56:06.9999378Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:06.9999466Z outputs = self.mobilebert( 2025-12-04T08:56:06.9999777Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:06.9999858Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0000178Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0000260Z layer_outputs = layer_module( 2025-12-04T08:56:07.0000666Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T08:56:07.0000765Z self_attention_outputs = self.attention( 2025-12-04T08:56:07.0001078Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T08:56:07.0001169Z self_outputs = self.self( 2025-12-04T08:56:07.0001508Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-12-04T08:56:07.0001615Z self.value(value_tensor) 2025-12-04T08:56:07.0001619Z 2025-12-04T08:56:07.0001733Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0001951Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0002029Z res = mod(**inputs) 2025-12-04T08:56:07.0002347Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0002427Z outputs = self.mobilebert( 2025-12-04T08:56:07.0002748Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0002829Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0003169Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0003253Z layer_outputs = layer_module( 2025-12-04T08:56:07.0003568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T08:56:07.0003757Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T08:56:07.0004071Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-12-04T08:56:07.0004206Z bottlenecked_hidden_states = self.input(hidden_states) 2025-12-04T08:56:07.0004526Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-12-04T08:56:07.0004619Z layer_input = self.dense(hidden_states) 2025-12-04T08:56:07.0004624Z 2025-12-04T08:56:07.0004747Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0004965Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0005055Z res = mod(**inputs) 2025-12-04T08:56:07.0005360Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0005435Z outputs = self.mobilebert( 2025-12-04T08:56:07.0005744Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0005823Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0006123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0006209Z layer_outputs = layer_module( 2025-12-04T08:56:07.0006510Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T08:56:07.0006691Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T08:56:07.0006998Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-12-04T08:56:07.0007115Z shared_attention_input = self.attention(hidden_states) 2025-12-04T08:56:07.0007425Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-12-04T08:56:07.0007516Z layer_input = self.LayerNorm(layer_input) 2025-12-04T08:56:07.0007846Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:07.0007944Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:07.0007948Z 2025-12-04T08:56:07.0008060Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0008294Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0008382Z res = mod(**inputs) 2025-12-04T08:56:07.0008707Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0008790Z outputs = self.mobilebert( 2025-12-04T08:56:07.0009100Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0009185Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0009496Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0009572Z layer_outputs = layer_module( 2025-12-04T08:56:07.0009891Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T08:56:07.0010002Z self_attention_outputs = self.attention( 2025-12-04T08:56:07.0010315Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T08:56:07.0010394Z self_outputs = self.self( 2025-12-04T08:56:07.0010696Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-12-04T08:56:07.0010781Z self.query(query_tensor) 2025-12-04T08:56:07.0010784Z 2025-12-04T08:56:07.0010893Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0011107Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0011182Z res = mod(**inputs) 2025-12-04T08:56:07.0011484Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0011569Z outputs = self.mobilebert( 2025-12-04T08:56:07.0011873Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0011951Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0012261Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0012336Z layer_outputs = layer_module( 2025-12-04T08:56:07.0012648Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T08:56:07.0012740Z self_attention_outputs = self.attention( 2025-12-04T08:56:07.0013044Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T08:56:07.0013128Z self_outputs = self.self( 2025-12-04T08:56:07.0013430Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-12-04T08:56:07.0013507Z self.key(key_tensor) 2025-12-04T08:56:07.0013517Z 2025-12-04T08:56:07.0013606Z cudagraph partition due to non gpu ops 2025-12-04T08:56:07.0013693Z cudagraph partition due to non gpu ops 2025-12-04T08:56:07.0013811Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0014023Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0014092Z res = mod(**inputs) 2025-12-04T08:56:07.0014422Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0014498Z outputs = self.mobilebert( 2025-12-04T08:56:07.0014799Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0014886Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0015209Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0015309Z layer_outputs = layer_module( 2025-12-04T08:56:07.0015624Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T08:56:07.0015713Z self_attention_outputs = self.attention( 2025-12-04T08:56:07.0016023Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-12-04T08:56:07.0016158Z attention_output = self.output(self_outputs[0], layer_input) 2025-12-04T08:56:07.0016468Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-12-04T08:56:07.0016556Z layer_outputs = self.dense(hidden_states) 2025-12-04T08:56:07.0016578Z 2025-12-04T08:56:07.0016689Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0016909Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0016980Z res = mod(**inputs) 2025-12-04T08:56:07.0017289Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0017364Z outputs = self.mobilebert( 2025-12-04T08:56:07.0017669Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0017754Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0018059Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0018135Z layer_outputs = layer_module( 2025-12-04T08:56:07.0018445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T08:56:07.0018536Z self_attention_outputs = self.attention( 2025-12-04T08:56:07.0018848Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-12-04T08:56:07.0018981Z attention_output = self.output(self_outputs[0], layer_input) 2025-12-04T08:56:07.0019289Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-12-04T08:56:07.0019429Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T08:56:07.0019734Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:07.0019839Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:07.0019844Z 2025-12-04T08:56:07.0019957Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0020174Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0020252Z res = mod(**inputs) 2025-12-04T08:56:07.0020555Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0020631Z outputs = self.mobilebert( 2025-12-04T08:56:07.0021150Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0021235Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0021608Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0021687Z layer_outputs = layer_module( 2025-12-04T08:56:07.0021989Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.0022132Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.0022478Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T08:56:07.0022609Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T08:56:07.0022915Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T08:56:07.0023006Z hidden_states = self.dense(hidden_states) 2025-12-04T08:56:07.0023010Z 2025-12-04T08:56:07.0023132Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0023350Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0023420Z res = mod(**inputs) 2025-12-04T08:56:07.0023745Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0023857Z outputs = self.mobilebert( 2025-12-04T08:56:07.0024178Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0024256Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0024559Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0024642Z layer_outputs = layer_module( 2025-12-04T08:56:07.0024946Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.0025065Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.0025366Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T08:56:07.0025484Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T08:56:07.0025792Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T08:56:07.0025913Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:56:07.0025918Z 2025-12-04T08:56:07.0026035Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0026248Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0026317Z res = mod(**inputs) 2025-12-04T08:56:07.0026629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0026703Z outputs = self.mobilebert( 2025-12-04T08:56:07.0027004Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0027095Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0027397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0027480Z layer_outputs = layer_module( 2025-12-04T08:56:07.0027781Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.0027881Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.0028204Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T08:56:07.0028372Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T08:56:07.0028682Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T08:56:07.0028772Z layer_outputs = self.dense(hidden_states) 2025-12-04T08:56:07.0028778Z 2025-12-04T08:56:07.0028888Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0029147Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0029218Z res = mod(**inputs) 2025-12-04T08:56:07.0029537Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0029622Z outputs = self.mobilebert( 2025-12-04T08:56:07.0029942Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0030040Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0030344Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0030419Z layer_outputs = layer_module( 2025-12-04T08:56:07.0030756Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.0030856Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.0031168Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T08:56:07.0031303Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T08:56:07.0031606Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T08:56:07.0031746Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T08:56:07.0032053Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:07.0032158Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:07.0032163Z 2025-12-04T08:56:07.0032272Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0032492Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0032568Z res = mod(**inputs) 2025-12-04T08:56:07.0032873Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0032946Z outputs = self.mobilebert( 2025-12-04T08:56:07.0033256Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0033335Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0033645Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0033720Z layer_outputs = layer_module( 2025-12-04T08:56:07.0034030Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.0034141Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.0034445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T08:56:07.0034569Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T08:56:07.0034874Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T08:56:07.0034963Z hidden_states = self.dense(hidden_states) 2025-12-04T08:56:07.0034967Z 2025-12-04T08:56:07.0035101Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0035314Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0035384Z res = mod(**inputs) 2025-12-04T08:56:07.0035698Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0035794Z outputs = self.mobilebert( 2025-12-04T08:56:07.0036129Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0036211Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0036522Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0036609Z layer_outputs = layer_module( 2025-12-04T08:56:07.0036921Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.0037030Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.0037339Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T08:56:07.0037482Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T08:56:07.0037812Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T08:56:07.0037935Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:56:07.0037939Z 2025-12-04T08:56:07.0038106Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0038340Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0038412Z res = mod(**inputs) 2025-12-04T08:56:07.0038733Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0038810Z outputs = self.mobilebert( 2025-12-04T08:56:07.0039121Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0039210Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0039525Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0039611Z layer_outputs = layer_module( 2025-12-04T08:56:07.0039971Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.0040073Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.0040391Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T08:56:07.0040528Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T08:56:07.0040842Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T08:56:07.0040945Z layer_outputs = self.dense(hidden_states) 2025-12-04T08:56:07.0040949Z 2025-12-04T08:56:07.0041064Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0041292Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0041365Z res = mod(**inputs) 2025-12-04T08:56:07.0041676Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0041762Z outputs = self.mobilebert( 2025-12-04T08:56:07.0042097Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0042188Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0042505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0042583Z layer_outputs = layer_module( 2025-12-04T08:56:07.0042926Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.0043050Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.0043366Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T08:56:07.0043511Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T08:56:07.0043825Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T08:56:07.0043968Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T08:56:07.0044281Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:07.0044380Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:07.0044410Z 2025-12-04T08:56:07.0044523Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0044746Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0044828Z res = mod(**inputs) 2025-12-04T08:56:07.0045140Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0045218Z outputs = self.mobilebert( 2025-12-04T08:56:07.0045538Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0045618Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0045936Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0046015Z layer_outputs = layer_module( 2025-12-04T08:56:07.0046333Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.0046446Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.0046756Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T08:56:07.0046881Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T08:56:07.0047198Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T08:56:07.0047291Z hidden_states = self.dense(hidden_states) 2025-12-04T08:56:07.0047297Z 2025-12-04T08:56:07.0047417Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0047635Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0047708Z res = mod(**inputs) 2025-12-04T08:56:07.0048031Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0048110Z outputs = self.mobilebert( 2025-12-04T08:56:07.0048431Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0048511Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0048821Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0048909Z layer_outputs = layer_module( 2025-12-04T08:56:07.0049241Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.0049346Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.0049663Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T08:56:07.0049804Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T08:56:07.0050140Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T08:56:07.0050262Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:56:07.0050266Z 2025-12-04T08:56:07.0050378Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0050610Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0050680Z res = mod(**inputs) 2025-12-04T08:56:07.0051002Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0051080Z outputs = self.mobilebert( 2025-12-04T08:56:07.0051397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0051514Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0051820Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0051896Z layer_outputs = layer_module( 2025-12-04T08:56:07.0052206Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.0052305Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.0052617Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T08:56:07.0052751Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T08:56:07.0053052Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T08:56:07.0053152Z layer_outputs = self.dense(hidden_states) 2025-12-04T08:56:07.0053155Z 2025-12-04T08:56:07.0053267Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0053486Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0053557Z res = mod(**inputs) 2025-12-04T08:56:07.0053862Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0053947Z outputs = self.mobilebert( 2025-12-04T08:56:07.0054254Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0054338Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0054642Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0054718Z layer_outputs = layer_module( 2025-12-04T08:56:07.0055031Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.0055131Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.0055436Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T08:56:07.0055576Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T08:56:07.0055902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T08:56:07.0056043Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T08:56:07.0056347Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:07.0056447Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:07.0056451Z 2025-12-04T08:56:07.0056586Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0056822Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0056898Z res = mod(**inputs) 2025-12-04T08:56:07.0057204Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0057280Z outputs = self.mobilebert( 2025-12-04T08:56:07.0057592Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0057670Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0057975Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0058059Z layer_outputs = layer_module( 2025-12-04T08:56:07.0058386Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-12-04T08:56:07.0058526Z intermediate_output = self.intermediate(attention_output) 2025-12-04T08:56:07.0058834Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T08:56:07.0058924Z hidden_states = self.dense(hidden_states) 2025-12-04T08:56:07.0058928Z 2025-12-04T08:56:07.0059048Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0059260Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0059337Z res = mod(**inputs) 2025-12-04T08:56:07.0059654Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0059731Z outputs = self.mobilebert( 2025-12-04T08:56:07.0060045Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0060123Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0060435Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0060518Z layer_outputs = layer_module( 2025-12-04T08:56:07.0060829Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-12-04T08:56:07.0060966Z intermediate_output = self.intermediate(attention_output) 2025-12-04T08:56:07.0061276Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T08:56:07.0061392Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:56:07.0061395Z 2025-12-04T08:56:07.0061513Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0061724Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0061801Z res = mod(**inputs) 2025-12-04T08:56:07.0062114Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0062189Z outputs = self.mobilebert( 2025-12-04T08:56:07.0062500Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0062576Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0062917Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0062996Z layer_outputs = layer_module( 2025-12-04T08:56:07.0063305Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T08:56:07.0063503Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T08:56:07.0063833Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-12-04T08:56:07.0063937Z layer_output = self.dense(intermediate_states) 2025-12-04T08:56:07.0063948Z 2025-12-04T08:56:07.0064057Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0064268Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0064347Z res = mod(**inputs) 2025-12-04T08:56:07.0064647Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0064722Z outputs = self.mobilebert( 2025-12-04T08:56:07.0065036Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0065132Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0065443Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0065518Z layer_outputs = layer_module( 2025-12-04T08:56:07.0065834Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T08:56:07.0066011Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T08:56:07.0066324Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-12-04T08:56:07.0066457Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-12-04T08:56:07.0066765Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:07.0066866Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:07.0066871Z 2025-12-04T08:56:07.0066989Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0067200Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0067269Z res = mod(**inputs) 2025-12-04T08:56:07.0067587Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0067662Z outputs = self.mobilebert( 2025-12-04T08:56:07.0067981Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0068058Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0068357Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0068443Z layer_outputs = layer_module( 2025-12-04T08:56:07.0068742Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T08:56:07.0068917Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T08:56:07.0069231Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-12-04T08:56:07.0069361Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-12-04T08:56:07.0069696Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-12-04T08:56:07.0069788Z layer_outputs = self.dense(hidden_states) 2025-12-04T08:56:07.0069792Z 2025-12-04T08:56:07.0069903Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0070121Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0070203Z res = mod(**inputs) 2025-12-04T08:56:07.0070534Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0070608Z outputs = self.mobilebert( 2025-12-04T08:56:07.0070919Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0071003Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0071316Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0071398Z layer_outputs = layer_module( 2025-12-04T08:56:07.0071700Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T08:56:07.0071887Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T08:56:07.0072197Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-12-04T08:56:07.0072330Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-12-04T08:56:07.0072637Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-12-04T08:56:07.0072766Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T08:56:07.0073078Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:07.0073185Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:07.0073189Z 2025-12-04T08:56:07.0073299Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0073512Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0073588Z res = mod(**inputs) 2025-12-04T08:56:07.0073892Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0073973Z outputs = self.mobilebert( 2025-12-04T08:56:07.0074286Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0074363Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0074678Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0074754Z layer_outputs = layer_module( 2025-12-04T08:56:07.0075066Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T08:56:07.0075242Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T08:56:07.0075546Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-12-04T08:56:07.0075674Z shared_attention_input = self.attention(hidden_states) 2025-12-04T08:56:07.0075973Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-12-04T08:56:07.0076062Z layer_input = self.dense(hidden_states) 2025-12-04T08:56:07.0076072Z 2025-12-04T08:56:07.0076182Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0076414Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0076490Z res = mod(**inputs) 2025-12-04T08:56:07.0076808Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0076887Z outputs = self.mobilebert( 2025-12-04T08:56:07.0077221Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0077328Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0077639Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0077715Z layer_outputs = layer_module( 2025-12-04T08:56:07.0078019Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T08:56:07.0078210Z self_attention_outputs = self.attention( 2025-12-04T08:56:07.0078535Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T08:56:07.0078617Z self_outputs = self.self( 2025-12-04T08:56:07.0078965Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-12-04T08:56:07.0079048Z self.value(value_tensor) 2025-12-04T08:56:07.0079052Z 2025-12-04T08:56:07.0079177Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0079399Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0079474Z res = mod(**inputs) 2025-12-04T08:56:07.0079806Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0079886Z outputs = self.mobilebert( 2025-12-04T08:56:07.0080201Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0080279Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0080586Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0080677Z layer_outputs = layer_module( 2025-12-04T08:56:07.0080980Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T08:56:07.0081151Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T08:56:07.0081466Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-12-04T08:56:07.0081587Z bottlenecked_hidden_states = self.input(hidden_states) 2025-12-04T08:56:07.0081898Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-12-04T08:56:07.0081986Z layer_input = self.dense(hidden_states) 2025-12-04T08:56:07.0081990Z 2025-12-04T08:56:07.0082103Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0082335Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0082405Z res = mod(**inputs) 2025-12-04T08:56:07.0082721Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0082796Z outputs = self.mobilebert( 2025-12-04T08:56:07.0083101Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0083186Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0083514Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0083598Z layer_outputs = layer_module( 2025-12-04T08:56:07.0083902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T08:56:07.0084093Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T08:56:07.0084422Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-12-04T08:56:07.0084541Z shared_attention_input = self.attention(hidden_states) 2025-12-04T08:56:07.0084845Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-12-04T08:56:07.0084945Z layer_input = self.LayerNorm(layer_input) 2025-12-04T08:56:07.0085248Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:07.0085354Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:07.0085358Z 2025-12-04T08:56:07.0085466Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0085702Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0085779Z res = mod(**inputs) 2025-12-04T08:56:07.0086083Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0086165Z outputs = self.mobilebert( 2025-12-04T08:56:07.0086465Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0086542Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0086854Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0086929Z layer_outputs = layer_module( 2025-12-04T08:56:07.0087231Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T08:56:07.0087332Z self_attention_outputs = self.attention( 2025-12-04T08:56:07.0087636Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T08:56:07.0087723Z self_outputs = self.self( 2025-12-04T08:56:07.0088023Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-12-04T08:56:07.0088099Z self.query(query_tensor) 2025-12-04T08:56:07.0088102Z 2025-12-04T08:56:07.0088218Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0088430Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0088507Z res = mod(**inputs) 2025-12-04T08:56:07.0088809Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0088885Z outputs = self.mobilebert( 2025-12-04T08:56:07.0089197Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0089275Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0089578Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0089661Z layer_outputs = layer_module( 2025-12-04T08:56:07.0089964Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T08:56:07.0090081Z self_attention_outputs = self.attention( 2025-12-04T08:56:07.0090386Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T08:56:07.0090461Z self_outputs = self.self( 2025-12-04T08:56:07.0090776Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-12-04T08:56:07.0090867Z self.key(key_tensor) 2025-12-04T08:56:07.0090888Z 2025-12-04T08:56:07.0090985Z cudagraph partition due to non gpu ops 2025-12-04T08:56:07.0091073Z cudagraph partition due to non gpu ops 2025-12-04T08:56:07.0091185Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0091407Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0091476Z res = mod(**inputs) 2025-12-04T08:56:07.0091788Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0091871Z outputs = self.mobilebert( 2025-12-04T08:56:07.0092176Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0092282Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0092587Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0092665Z layer_outputs = layer_module( 2025-12-04T08:56:07.0092977Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T08:56:07.0093068Z self_attention_outputs = self.attention( 2025-12-04T08:56:07.0093369Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-12-04T08:56:07.0093513Z attention_output = self.output(self_outputs[0], layer_input) 2025-12-04T08:56:07.0093816Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-12-04T08:56:07.0093915Z layer_outputs = self.dense(hidden_states) 2025-12-04T08:56:07.0093921Z 2025-12-04T08:56:07.0094031Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0094244Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0094324Z res = mod(**inputs) 2025-12-04T08:56:07.0094627Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0094710Z outputs = self.mobilebert( 2025-12-04T08:56:07.0095013Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0095093Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0095403Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0095478Z layer_outputs = layer_module( 2025-12-04T08:56:07.0095786Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T08:56:07.0095878Z self_attention_outputs = self.attention( 2025-12-04T08:56:07.0096182Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-12-04T08:56:07.0096330Z attention_output = self.output(self_outputs[0], layer_input) 2025-12-04T08:56:07.0096633Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-12-04T08:56:07.0096767Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T08:56:07.0097096Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:07.0097195Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:07.0097199Z 2025-12-04T08:56:07.0097319Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0097552Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0097639Z res = mod(**inputs) 2025-12-04T08:56:07.0097956Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0098032Z outputs = self.mobilebert( 2025-12-04T08:56:07.0098344Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0098422Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0098723Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0098808Z layer_outputs = layer_module( 2025-12-04T08:56:07.0099107Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.0099235Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.0099550Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T08:56:07.0099670Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T08:56:07.0099981Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T08:56:07.0100070Z hidden_states = self.dense(hidden_states) 2025-12-04T08:56:07.0100074Z 2025-12-04T08:56:07.0100185Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0100405Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0100475Z res = mod(**inputs) 2025-12-04T08:56:07.0100785Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0100863Z outputs = self.mobilebert( 2025-12-04T08:56:07.0101170Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0101256Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0101559Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0101634Z layer_outputs = layer_module( 2025-12-04T08:56:07.0101950Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.0102051Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.0102363Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T08:56:07.0102484Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T08:56:07.0102787Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T08:56:07.0102915Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:56:07.0102919Z 2025-12-04T08:56:07.0103027Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0103256Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0103324Z res = mod(**inputs) 2025-12-04T08:56:07.0103659Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0103742Z outputs = self.mobilebert( 2025-12-04T08:56:07.0104049Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0104134Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0104464Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0104559Z layer_outputs = layer_module( 2025-12-04T08:56:07.0104868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.0104968Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.0105271Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T08:56:07.0105415Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T08:56:07.0105722Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T08:56:07.0105817Z layer_outputs = self.dense(hidden_states) 2025-12-04T08:56:07.0105892Z 2025-12-04T08:56:07.0106001Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0106217Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0106293Z res = mod(**inputs) 2025-12-04T08:56:07.0106595Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0106675Z outputs = self.mobilebert( 2025-12-04T08:56:07.0106980Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0107057Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0107367Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0107443Z layer_outputs = layer_module( 2025-12-04T08:56:07.0107748Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.0107856Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.0108163Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T08:56:07.0108305Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T08:56:07.0108611Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T08:56:07.0108747Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T08:56:07.0109062Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:07.0109163Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:07.0109168Z 2025-12-04T08:56:07.0109288Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0109504Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0109580Z res = mod(**inputs) 2025-12-04T08:56:07.0109901Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0109978Z outputs = self.mobilebert( 2025-12-04T08:56:07.0110296Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0110400Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0110708Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0110792Z layer_outputs = layer_module( 2025-12-04T08:56:07.0111095Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.0111214Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.0111546Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T08:56:07.0111666Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T08:56:07.0111979Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T08:56:07.0112070Z hidden_states = self.dense(hidden_states) 2025-12-04T08:56:07.0112074Z 2025-12-04T08:56:07.0112188Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0112415Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0112484Z res = mod(**inputs) 2025-12-04T08:56:07.0112815Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0112910Z outputs = self.mobilebert( 2025-12-04T08:56:07.0113213Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0113297Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0113599Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0113675Z layer_outputs = layer_module( 2025-12-04T08:56:07.0113985Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.0114084Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.0114392Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T08:56:07.0114512Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T08:56:07.0114823Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T08:56:07.0114956Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:56:07.0114960Z 2025-12-04T08:56:07.0115074Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0115300Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0115371Z res = mod(**inputs) 2025-12-04T08:56:07.0115683Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0115767Z outputs = self.mobilebert( 2025-12-04T08:56:07.0116079Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0116160Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0116478Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0116557Z layer_outputs = layer_module( 2025-12-04T08:56:07.0116875Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.0116977Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.0117308Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T08:56:07.0117457Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T08:56:07.0117772Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T08:56:07.0117872Z layer_outputs = self.dense(hidden_states) 2025-12-04T08:56:07.0117875Z 2025-12-04T08:56:07.0118010Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0118345Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0118428Z res = mod(**inputs) 2025-12-04T08:56:07.0118741Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0118819Z outputs = self.mobilebert( 2025-12-04T08:56:07.0119141Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0119219Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0119539Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0119618Z layer_outputs = layer_module( 2025-12-04T08:56:07.0119954Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.0120069Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.0120381Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T08:56:07.0120527Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T08:56:07.0120976Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T08:56:07.0121121Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T08:56:07.0121442Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:07.0121545Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:07.0121551Z 2025-12-04T08:56:07.0121673Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0121894Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0121969Z res = mod(**inputs) 2025-12-04T08:56:07.0122288Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0122366Z outputs = self.mobilebert( 2025-12-04T08:56:07.0122677Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0122766Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0123084Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0123169Z layer_outputs = layer_module( 2025-12-04T08:56:07.0123483Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.0123588Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.0123908Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T08:56:07.0124031Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T08:56:07.0124348Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T08:56:07.0124440Z hidden_states = self.dense(hidden_states) 2025-12-04T08:56:07.0124495Z 2025-12-04T08:56:07.0124612Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0124840Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0124909Z res = mod(**inputs) 2025-12-04T08:56:07.0125250Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0125363Z outputs = self.mobilebert( 2025-12-04T08:56:07.0125673Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0125761Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0126073Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0126153Z layer_outputs = layer_module( 2025-12-04T08:56:07.0126482Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.0126583Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.0126898Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T08:56:07.0127050Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T08:56:07.0127369Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T08:56:07.0127499Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:56:07.0127503Z 2025-12-04T08:56:07.0127617Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0127855Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0127927Z res = mod(**inputs) 2025-12-04T08:56:07.0128256Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0128341Z outputs = self.mobilebert( 2025-12-04T08:56:07.0128655Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0128736Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0129074Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0129150Z layer_outputs = layer_module( 2025-12-04T08:56:07.0129460Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.0129560Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.0129865Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T08:56:07.0130005Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T08:56:07.0130310Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T08:56:07.0130408Z layer_outputs = self.dense(hidden_states) 2025-12-04T08:56:07.0130411Z 2025-12-04T08:56:07.0130522Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0130736Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0130810Z res = mod(**inputs) 2025-12-04T08:56:07.0131111Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0131186Z outputs = self.mobilebert( 2025-12-04T08:56:07.0131528Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0131608Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0131919Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0131997Z layer_outputs = layer_module( 2025-12-04T08:56:07.0132313Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.0132442Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.0132746Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T08:56:07.0132885Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T08:56:07.0133192Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T08:56:07.0133322Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T08:56:07.0133649Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:07.0133774Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:07.0133777Z 2025-12-04T08:56:07.0133891Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0134121Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0134191Z res = mod(**inputs) 2025-12-04T08:56:07.0134507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0134585Z outputs = self.mobilebert( 2025-12-04T08:56:07.0134898Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0134986Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0135293Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0135379Z layer_outputs = layer_module( 2025-12-04T08:56:07.0135693Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-12-04T08:56:07.0135828Z intermediate_output = self.intermediate(attention_output) 2025-12-04T08:56:07.0136150Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T08:56:07.0136243Z hidden_states = self.dense(hidden_states) 2025-12-04T08:56:07.0136247Z 2025-12-04T08:56:07.0136364Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0136586Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0136657Z res = mod(**inputs) 2025-12-04T08:56:07.0136973Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0137051Z outputs = self.mobilebert( 2025-12-04T08:56:07.0137369Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0137456Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0137768Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0137853Z layer_outputs = layer_module( 2025-12-04T08:56:07.0138164Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-12-04T08:56:07.0138296Z intermediate_output = self.intermediate(attention_output) 2025-12-04T08:56:07.0138637Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T08:56:07.0138763Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:56:07.0138767Z 2025-12-04T08:56:07.0138888Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0139124Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0139223Z res = mod(**inputs) 2025-12-04T08:56:07.0139533Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0139607Z outputs = self.mobilebert( 2025-12-04T08:56:07.0139908Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0139993Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0140294Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0140377Z layer_outputs = layer_module( 2025-12-04T08:56:07.0140677Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T08:56:07.0140876Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T08:56:07.0141187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-12-04T08:56:07.0141288Z layer_output = self.dense(intermediate_states) 2025-12-04T08:56:07.0141292Z 2025-12-04T08:56:07.0141408Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0141619Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0141689Z res = mod(**inputs) 2025-12-04T08:56:07.0141998Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0142071Z outputs = self.mobilebert( 2025-12-04T08:56:07.0142374Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0142464Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0142778Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0142862Z layer_outputs = layer_module( 2025-12-04T08:56:07.0143173Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T08:56:07.0143347Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T08:56:07.0143671Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-12-04T08:56:07.0143806Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-12-04T08:56:07.0144125Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:07.0144229Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:07.0144234Z 2025-12-04T08:56:07.0144347Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0144578Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0144649Z res = mod(**inputs) 2025-12-04T08:56:07.0144969Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0145054Z outputs = self.mobilebert( 2025-12-04T08:56:07.0145390Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0145478Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0145799Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0145875Z layer_outputs = layer_module( 2025-12-04T08:56:07.0146206Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T08:56:07.0146397Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T08:56:07.0146720Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-12-04T08:56:07.0146855Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-12-04T08:56:07.0147173Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-12-04T08:56:07.0147276Z layer_outputs = self.dense(hidden_states) 2025-12-04T08:56:07.0147280Z 2025-12-04T08:56:07.0147394Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0147645Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0147720Z res = mod(**inputs) 2025-12-04T08:56:07.0148034Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0148121Z outputs = self.mobilebert( 2025-12-04T08:56:07.0148446Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0148526Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0148848Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0148926Z layer_outputs = layer_module( 2025-12-04T08:56:07.0149247Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T08:56:07.0149419Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T08:56:07.0149733Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-12-04T08:56:07.0149879Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-12-04T08:56:07.0150193Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-12-04T08:56:07.0150333Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T08:56:07.0150647Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:07.0150749Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:07.0150752Z 2025-12-04T08:56:07.0150876Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0151097Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0151175Z res = mod(**inputs) 2025-12-04T08:56:07.0151489Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0151566Z outputs = self.mobilebert( 2025-12-04T08:56:07.0151885Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0151963Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0152301Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0152390Z layer_outputs = layer_module( 2025-12-04T08:56:07.0152698Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T08:56:07.0152885Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T08:56:07.0153218Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-12-04T08:56:07.0153360Z shared_attention_input = self.attention(hidden_states) 2025-12-04T08:56:07.0153681Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-12-04T08:56:07.0153773Z layer_input = self.dense(hidden_states) 2025-12-04T08:56:07.0153777Z 2025-12-04T08:56:07.0153897Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0154122Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0154193Z res = mod(**inputs) 2025-12-04T08:56:07.0154520Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0154629Z outputs = self.mobilebert( 2025-12-04T08:56:07.0154942Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0155031Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0155340Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0155427Z layer_outputs = layer_module( 2025-12-04T08:56:07.0155742Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T08:56:07.0155838Z self_attention_outputs = self.attention( 2025-12-04T08:56:07.0156158Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T08:56:07.0156237Z self_outputs = self.self( 2025-12-04T08:56:07.0156564Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-12-04T08:56:07.0156647Z self.value(value_tensor) 2025-12-04T08:56:07.0156650Z 2025-12-04T08:56:07.0156763Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0156991Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0157061Z res = mod(**inputs) 2025-12-04T08:56:07.0157417Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0157501Z outputs = self.mobilebert( 2025-12-04T08:56:07.0157808Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0157894Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0158302Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0158392Z layer_outputs = layer_module( 2025-12-04T08:56:07.0158712Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T08:56:07.0158888Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T08:56:07.0159252Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-12-04T08:56:07.0159402Z bottlenecked_hidden_states = self.input(hidden_states) 2025-12-04T08:56:07.0159716Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-12-04T08:56:07.0159815Z layer_input = self.dense(hidden_states) 2025-12-04T08:56:07.0159819Z 2025-12-04T08:56:07.0159935Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0160176Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0160266Z res = mod(**inputs) 2025-12-04T08:56:07.0160632Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0160717Z outputs = self.mobilebert( 2025-12-04T08:56:07.0161087Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0161166Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0161538Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0161616Z layer_outputs = layer_module( 2025-12-04T08:56:07.0161962Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T08:56:07.0162154Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T08:56:07.0162458Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-12-04T08:56:07.0162587Z shared_attention_input = self.attention(hidden_states) 2025-12-04T08:56:07.0162907Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-12-04T08:56:07.0163010Z layer_input = self.LayerNorm(layer_input) 2025-12-04T08:56:07.0163359Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:07.0163457Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:07.0163461Z 2025-12-04T08:56:07.0163580Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0163795Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0163867Z res = mod(**inputs) 2025-12-04T08:56:07.0164183Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0164260Z outputs = self.mobilebert( 2025-12-04T08:56:07.0164577Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0164653Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0164962Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0165048Z layer_outputs = layer_module( 2025-12-04T08:56:07.0165358Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T08:56:07.0165460Z self_attention_outputs = self.attention( 2025-12-04T08:56:07.0165771Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T08:56:07.0165850Z self_outputs = self.self( 2025-12-04T08:56:07.0166168Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-12-04T08:56:07.0166246Z self.query(query_tensor) 2025-12-04T08:56:07.0166250Z 2025-12-04T08:56:07.0166360Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0166606Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0166677Z res = mod(**inputs) 2025-12-04T08:56:07.0166987Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0167064Z outputs = self.mobilebert( 2025-12-04T08:56:07.0167385Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0167488Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0167795Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0167877Z layer_outputs = layer_module( 2025-12-04T08:56:07.0168185Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T08:56:07.0168278Z self_attention_outputs = self.attention( 2025-12-04T08:56:07.0168590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T08:56:07.0168667Z self_outputs = self.self( 2025-12-04T08:56:07.0168971Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-12-04T08:56:07.0169084Z self.key(key_tensor) 2025-12-04T08:56:07.0169089Z 2025-12-04T08:56:07.0169177Z cudagraph partition due to non gpu ops 2025-12-04T08:56:07.0169272Z cudagraph partition due to non gpu ops 2025-12-04T08:56:07.0169383Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0169595Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0169673Z res = mod(**inputs) 2025-12-04T08:56:07.0169989Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0170070Z outputs = self.mobilebert( 2025-12-04T08:56:07.0170376Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0170455Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0170768Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0170847Z layer_outputs = layer_module( 2025-12-04T08:56:07.0171156Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T08:56:07.0171254Z self_attention_outputs = self.attention( 2025-12-04T08:56:07.0171558Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-12-04T08:56:07.0171699Z attention_output = self.output(self_outputs[0], layer_input) 2025-12-04T08:56:07.0172005Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-12-04T08:56:07.0172095Z layer_outputs = self.dense(hidden_states) 2025-12-04T08:56:07.0172100Z 2025-12-04T08:56:07.0172221Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0172435Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0172514Z res = mod(**inputs) 2025-12-04T08:56:07.0172816Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0172891Z outputs = self.mobilebert( 2025-12-04T08:56:07.0173202Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0173300Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0173604Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0173687Z layer_outputs = layer_module( 2025-12-04T08:56:07.0173991Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T08:56:07.0174111Z self_attention_outputs = self.attention( 2025-12-04T08:56:07.0174431Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-12-04T08:56:07.0174563Z attention_output = self.output(self_outputs[0], layer_input) 2025-12-04T08:56:07.0174874Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-12-04T08:56:07.0175007Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T08:56:07.0175319Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:07.0175418Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:07.0175421Z 2025-12-04T08:56:07.0175552Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0175774Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0175845Z res = mod(**inputs) 2025-12-04T08:56:07.0176151Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0176232Z outputs = self.mobilebert( 2025-12-04T08:56:07.0176535Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0176618Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0176920Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0176996Z layer_outputs = layer_module( 2025-12-04T08:56:07.0177308Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.0177411Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.0177724Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T08:56:07.0177844Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T08:56:07.0178145Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T08:56:07.0178243Z hidden_states = self.dense(hidden_states) 2025-12-04T08:56:07.0178247Z 2025-12-04T08:56:07.0178357Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0178576Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0178645Z res = mod(**inputs) 2025-12-04T08:56:07.0178945Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0179030Z outputs = self.mobilebert( 2025-12-04T08:56:07.0179339Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0179415Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0179739Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0179814Z layer_outputs = layer_module( 2025-12-04T08:56:07.0180146Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.0180247Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.0180555Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T08:56:07.0180685Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T08:56:07.0181009Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T08:56:07.0181167Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:56:07.0181171Z 2025-12-04T08:56:07.0181282Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0181504Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0181582Z res = mod(**inputs) 2025-12-04T08:56:07.0181884Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0181962Z outputs = self.mobilebert( 2025-12-04T08:56:07.0182275Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0182373Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0182688Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0182766Z layer_outputs = layer_module( 2025-12-04T08:56:07.0183068Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.0183175Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.0183479Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T08:56:07.0183625Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T08:56:07.0183931Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T08:56:07.0184021Z layer_outputs = self.dense(hidden_states) 2025-12-04T08:56:07.0184026Z 2025-12-04T08:56:07.0184143Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0184356Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0184428Z res = mod(**inputs) 2025-12-04T08:56:07.0184739Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0184812Z outputs = self.mobilebert( 2025-12-04T08:56:07.0185120Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0185200Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0185501Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0185584Z layer_outputs = layer_module( 2025-12-04T08:56:07.0185887Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.0185995Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.0186297Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T08:56:07.0186440Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T08:56:07.0186732Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T08:56:07.0186880Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T08:56:07.0187176Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:07.0187274Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:07.0187279Z 2025-12-04T08:56:07.0187388Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0187629Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0187715Z res = mod(**inputs) 2025-12-04T08:56:07.0188016Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0188100Z outputs = self.mobilebert( 2025-12-04T08:56:07.0188399Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0188485Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0188788Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0188864Z layer_outputs = layer_module( 2025-12-04T08:56:07.0189173Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.0189295Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.0189606Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T08:56:07.0189724Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T08:56:07.0190036Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T08:56:07.0190136Z hidden_states = self.dense(hidden_states) 2025-12-04T08:56:07.0190140Z 2025-12-04T08:56:07.0190249Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0190463Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0190540Z res = mod(**inputs) 2025-12-04T08:56:07.0190841Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0190927Z outputs = self.mobilebert( 2025-12-04T08:56:07.0191228Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0191304Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0191614Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0191690Z layer_outputs = layer_module( 2025-12-04T08:56:07.0192002Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.0192100Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.0192401Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T08:56:07.0192528Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T08:56:07.0192829Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T08:56:07.0192947Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:56:07.0192957Z 2025-12-04T08:56:07.0193065Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0193277Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0193351Z res = mod(**inputs) 2025-12-04T08:56:07.0193677Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0193755Z outputs = self.mobilebert( 2025-12-04T08:56:07.0194068Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0194145Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0194470Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0194566Z layer_outputs = layer_module( 2025-12-04T08:56:07.0194880Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.0194984Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.0195289Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T08:56:07.0195421Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T08:56:07.0195730Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T08:56:07.0195840Z layer_outputs = self.dense(hidden_states) 2025-12-04T08:56:07.0195844Z 2025-12-04T08:56:07.0195961Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0196176Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0196246Z res = mod(**inputs) 2025-12-04T08:56:07.0196553Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0196626Z outputs = self.mobilebert( 2025-12-04T08:56:07.0196941Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0197017Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0197320Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0197406Z layer_outputs = layer_module( 2025-12-04T08:56:07.0197734Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.0197843Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.0198242Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T08:56:07.0198387Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T08:56:07.0198708Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T08:56:07.0198847Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T08:56:07.0199158Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:07.0199269Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:07.0199275Z 2025-12-04T08:56:07.0199389Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0199620Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0199693Z res = mod(**inputs) 2025-12-04T08:56:07.0200015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0200099Z outputs = self.mobilebert( 2025-12-04T08:56:07.0200406Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0200517Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0200824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0200901Z layer_outputs = layer_module( 2025-12-04T08:56:07.0201217Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.0201338Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.0201658Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T08:56:07.0201787Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T08:56:07.0202095Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T08:56:07.0202195Z hidden_states = self.dense(hidden_states) 2025-12-04T08:56:07.0202200Z 2025-12-04T08:56:07.0202311Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0202530Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0202608Z res = mod(**inputs) 2025-12-04T08:56:07.0202931Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0203017Z outputs = self.mobilebert( 2025-12-04T08:56:07.0203322Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0203398Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0203712Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0203789Z layer_outputs = layer_module( 2025-12-04T08:56:07.0204094Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.0204202Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.0204503Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T08:56:07.0204631Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T08:56:07.0204938Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T08:56:07.0205056Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:56:07.0205060Z 2025-12-04T08:56:07.0205176Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0205385Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0205462Z res = mod(**inputs) 2025-12-04T08:56:07.0205765Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0205839Z outputs = self.mobilebert( 2025-12-04T08:56:07.0206152Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0206232Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0206545Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0206624Z layer_outputs = layer_module( 2025-12-04T08:56:07.0206927Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.0207033Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.0207360Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T08:56:07.0207495Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T08:56:07.0207808Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T08:56:07.0207899Z layer_outputs = self.dense(hidden_states) 2025-12-04T08:56:07.0207903Z 2025-12-04T08:56:07.0208049Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0208286Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0208356Z res = mod(**inputs) 2025-12-04T08:56:07.0208667Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0208741Z outputs = self.mobilebert( 2025-12-04T08:56:07.0209049Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0209126Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0209430Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0209535Z layer_outputs = layer_module( 2025-12-04T08:56:07.0209841Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.0209942Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.0210250Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T08:56:07.0210383Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T08:56:07.0210705Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T08:56:07.0210839Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T08:56:07.0211157Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:07.0211263Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:07.0211267Z 2025-12-04T08:56:07.0211378Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0211597Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0211664Z res = mod(**inputs) 2025-12-04T08:56:07.0211963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0212046Z outputs = self.mobilebert( 2025-12-04T08:56:07.0212347Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0212424Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0212731Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0212806Z layer_outputs = layer_module( 2025-12-04T08:56:07.0213119Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-12-04T08:56:07.0213248Z intermediate_output = self.intermediate(attention_output) 2025-12-04T08:56:07.0213554Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T08:56:07.0213651Z hidden_states = self.dense(hidden_states) 2025-12-04T08:56:07.0213655Z 2025-12-04T08:56:07.0213765Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0214004Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0214074Z res = mod(**inputs) 2025-12-04T08:56:07.0214375Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0214457Z outputs = self.mobilebert( 2025-12-04T08:56:07.0214779Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0214940Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0215242Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0215320Z layer_outputs = layer_module( 2025-12-04T08:56:07.0215633Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-12-04T08:56:07.0215761Z intermediate_output = self.intermediate(attention_output) 2025-12-04T08:56:07.0216066Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T08:56:07.0216190Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:56:07.0216194Z 2025-12-04T08:56:07.0216322Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0216542Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0216613Z res = mod(**inputs) 2025-12-04T08:56:07.0216912Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0216990Z outputs = self.mobilebert( 2025-12-04T08:56:07.0217277Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0217363Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0217669Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0217747Z layer_outputs = layer_module( 2025-12-04T08:56:07.0218064Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T08:56:07.0218229Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T08:56:07.0218516Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-12-04T08:56:07.0218621Z layer_output = self.dense(intermediate_states) 2025-12-04T08:56:07.0218624Z 2025-12-04T08:56:07.0218729Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0218937Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0219002Z res = mod(**inputs) 2025-12-04T08:56:07.0219291Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0219370Z outputs = self.mobilebert( 2025-12-04T08:56:07.0219658Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0219743Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0220032Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0220104Z layer_outputs = layer_module( 2025-12-04T08:56:07.0220398Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T08:56:07.0220558Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T08:56:07.0221673Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-12-04T08:56:07.0221881Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-12-04T08:56:07.0222186Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:07.0222335Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:07.0222372Z 2025-12-04T08:56:07.0222482Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0222687Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0222766Z res = mod(**inputs) 2025-12-04T08:56:07.0223059Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0223143Z outputs = self.mobilebert( 2025-12-04T08:56:07.0223434Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0223509Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0223808Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0223913Z layer_outputs = layer_module( 2025-12-04T08:56:07.0224226Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T08:56:07.0224398Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T08:56:07.0224717Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-12-04T08:56:07.0224854Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-12-04T08:56:07.0225139Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-12-04T08:56:07.0225226Z layer_outputs = self.dense(hidden_states) 2025-12-04T08:56:07.0225238Z 2025-12-04T08:56:07.0225343Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0225543Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0225616Z res = mod(**inputs) 2025-12-04T08:56:07.0225903Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0225974Z outputs = self.mobilebert( 2025-12-04T08:56:07.0226268Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0226341Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0226635Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0226708Z layer_outputs = layer_module( 2025-12-04T08:56:07.0227001Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T08:56:07.0227168Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T08:56:07.0227473Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-12-04T08:56:07.0227613Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-12-04T08:56:07.0227921Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-12-04T08:56:07.0228044Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T08:56:07.0228351Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:07.0228446Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:07.0228450Z 2025-12-04T08:56:07.0228554Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0228761Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0228844Z res = mod(**inputs) 2025-12-04T08:56:07.0229152Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0229225Z outputs = self.mobilebert( 2025-12-04T08:56:07.0229509Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0229588Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0229886Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0229963Z layer_outputs = layer_module( 2025-12-04T08:56:07.0230249Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T08:56:07.0230432Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T08:56:07.0230728Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-12-04T08:56:07.0230842Z shared_attention_input = self.attention(hidden_states) 2025-12-04T08:56:07.0231126Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-12-04T08:56:07.0231217Z layer_input = self.dense(hidden_states) 2025-12-04T08:56:07.0231221Z 2025-12-04T08:56:07.0231325Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0231533Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0231599Z res = mod(**inputs) 2025-12-04T08:56:07.0231885Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0231966Z outputs = self.mobilebert( 2025-12-04T08:56:07.0232252Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0232330Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0232619Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0232691Z layer_outputs = layer_module( 2025-12-04T08:56:07.0232996Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T08:56:07.0233090Z self_attention_outputs = self.attention( 2025-12-04T08:56:07.0233391Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T08:56:07.0233473Z self_outputs = self.self( 2025-12-04T08:56:07.0233779Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-12-04T08:56:07.0233865Z self.value(value_tensor) 2025-12-04T08:56:07.0233869Z 2025-12-04T08:56:07.0233978Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0234193Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0234269Z res = mod(**inputs) 2025-12-04T08:56:07.0234568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0234673Z outputs = self.mobilebert( 2025-12-04T08:56:07.0234977Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0235054Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0235368Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0235466Z layer_outputs = layer_module( 2025-12-04T08:56:07.0235796Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T08:56:07.0235979Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T08:56:07.0236293Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-12-04T08:56:07.0236422Z bottlenecked_hidden_states = self.input(hidden_states) 2025-12-04T08:56:07.0236737Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-12-04T08:56:07.0236828Z layer_input = self.dense(hidden_states) 2025-12-04T08:56:07.0236833Z 2025-12-04T08:56:07.0236973Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0237195Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0237276Z res = mod(**inputs) 2025-12-04T08:56:07.0237589Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0237667Z outputs = self.mobilebert( 2025-12-04T08:56:07.0237985Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0238205Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0238540Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0238620Z layer_outputs = layer_module( 2025-12-04T08:56:07.0238935Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T08:56:07.0239122Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T08:56:07.0239449Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-12-04T08:56:07.0239567Z shared_attention_input = self.attention(hidden_states) 2025-12-04T08:56:07.0239878Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-12-04T08:56:07.0239972Z layer_input = self.LayerNorm(layer_input) 2025-12-04T08:56:07.0240321Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:07.0240423Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:07.0240427Z 2025-12-04T08:56:07.0240537Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0240764Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0240840Z res = mod(**inputs) 2025-12-04T08:56:07.0241152Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0241229Z outputs = self.mobilebert( 2025-12-04T08:56:07.0241540Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0241627Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0241957Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0242035Z layer_outputs = layer_module( 2025-12-04T08:56:07.0242361Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T08:56:07.0242453Z self_attention_outputs = self.attention( 2025-12-04T08:56:07.0242821Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T08:56:07.0242933Z self_outputs = self.self( 2025-12-04T08:56:07.0243236Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-12-04T08:56:07.0243322Z self.query(query_tensor) 2025-12-04T08:56:07.0243326Z 2025-12-04T08:56:07.0243438Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0243658Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0243727Z res = mod(**inputs) 2025-12-04T08:56:07.0244032Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0244135Z outputs = self.mobilebert( 2025-12-04T08:56:07.0244441Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0244519Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0244832Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0244908Z layer_outputs = layer_module( 2025-12-04T08:56:07.0245217Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T08:56:07.0245312Z self_attention_outputs = self.attention( 2025-12-04T08:56:07.0245616Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T08:56:07.0245702Z self_outputs = self.self( 2025-12-04T08:56:07.0246011Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-12-04T08:56:07.0246095Z self.key(key_tensor) 2025-12-04T08:56:07.0246100Z 2025-12-04T08:56:07.0246190Z cudagraph partition due to non gpu ops 2025-12-04T08:56:07.0246278Z cudagraph partition due to non gpu ops 2025-12-04T08:56:07.0246396Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0246607Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0246675Z res = mod(**inputs) 2025-12-04T08:56:07.0246984Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0247059Z outputs = self.mobilebert( 2025-12-04T08:56:07.0247375Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0247454Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0247759Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0247842Z layer_outputs = layer_module( 2025-12-04T08:56:07.0248156Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T08:56:07.0248251Z self_attention_outputs = self.attention( 2025-12-04T08:56:07.0248570Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-12-04T08:56:07.0248723Z attention_output = self.output(self_outputs[0], layer_input) 2025-12-04T08:56:07.0249036Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-12-04T08:56:07.0249127Z layer_outputs = self.dense(hidden_states) 2025-12-04T08:56:07.0249132Z 2025-12-04T08:56:07.0249241Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0249477Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0249565Z res = mod(**inputs) 2025-12-04T08:56:07.0249875Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0249952Z outputs = self.mobilebert( 2025-12-04T08:56:07.0250252Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0250342Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0250655Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0250738Z layer_outputs = layer_module( 2025-12-04T08:56:07.0251048Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T08:56:07.0251164Z self_attention_outputs = self.attention( 2025-12-04T08:56:07.0251477Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-12-04T08:56:07.0251610Z attention_output = self.output(self_outputs[0], layer_input) 2025-12-04T08:56:07.0251917Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-12-04T08:56:07.0252058Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T08:56:07.0252368Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:07.0252475Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:07.0252479Z 2025-12-04T08:56:07.0252588Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0252806Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0252884Z res = mod(**inputs) 2025-12-04T08:56:07.0253186Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0253271Z outputs = self.mobilebert( 2025-12-04T08:56:07.0253576Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0253653Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0253970Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0254048Z layer_outputs = layer_module( 2025-12-04T08:56:07.0254347Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.0254462Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.0254772Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T08:56:07.0254901Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T08:56:07.0255217Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T08:56:07.0255309Z hidden_states = self.dense(hidden_states) 2025-12-04T08:56:07.0255313Z 2025-12-04T08:56:07.0255450Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0255665Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0255743Z res = mod(**inputs) 2025-12-04T08:56:07.0256045Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0256124Z outputs = self.mobilebert( 2025-12-04T08:56:07.0256455Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0256549Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0256852Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0256937Z layer_outputs = layer_module( 2025-12-04T08:56:07.0257255Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.0257364Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.0257676Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T08:56:07.0257813Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T08:56:07.0258126Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T08:56:07.0258250Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:56:07.0258254Z 2025-12-04T08:56:07.0258371Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0258586Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0258653Z res = mod(**inputs) 2025-12-04T08:56:07.0258968Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0259045Z outputs = self.mobilebert( 2025-12-04T08:56:07.0259355Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0259434Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0259739Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0259825Z layer_outputs = layer_module( 2025-12-04T08:56:07.0260128Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.0260226Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.0260542Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T08:56:07.0260680Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T08:56:07.0261053Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T08:56:07.0261146Z layer_outputs = self.dense(hidden_states) 2025-12-04T08:56:07.0261151Z 2025-12-04T08:56:07.0261261Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0261492Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0261564Z res = mod(**inputs) 2025-12-04T08:56:07.0261888Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0261963Z outputs = self.mobilebert( 2025-12-04T08:56:07.0262265Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0262371Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0262674Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0262750Z layer_outputs = layer_module( 2025-12-04T08:56:07.0263063Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.0263199Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.0263510Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T08:56:07.0263644Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T08:56:07.0263946Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T08:56:07.0264086Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T08:56:07.0264395Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:07.0264503Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:07.0264525Z 2025-12-04T08:56:07.0264640Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0264863Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0264942Z res = mod(**inputs) 2025-12-04T08:56:07.0265251Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0265334Z outputs = self.mobilebert( 2025-12-04T08:56:07.0265648Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0265728Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0266047Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0266126Z layer_outputs = layer_module( 2025-12-04T08:56:07.0266452Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.0266562Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.0266863Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T08:56:07.0266988Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T08:56:07.0267289Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T08:56:07.0267382Z hidden_states = self.dense(hidden_states) 2025-12-04T08:56:07.0267386Z 2025-12-04T08:56:07.0267508Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0267729Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0267807Z res = mod(**inputs) 2025-12-04T08:56:07.0268119Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0268199Z outputs = self.mobilebert( 2025-12-04T08:56:07.0268523Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0268601Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0268919Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0269004Z layer_outputs = layer_module( 2025-12-04T08:56:07.0269339Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.0269450Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.0269763Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T08:56:07.0269886Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T08:56:07.0270232Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T08:56:07.0270374Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:56:07.0270378Z 2025-12-04T08:56:07.0270498Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0270719Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0270789Z res = mod(**inputs) 2025-12-04T08:56:07.0271111Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0271189Z outputs = self.mobilebert( 2025-12-04T08:56:07.0271503Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0271611Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0271935Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0272023Z layer_outputs = layer_module( 2025-12-04T08:56:07.0272346Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.0272450Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.0272785Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T08:56:07.0272924Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T08:56:07.0273252Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T08:56:07.0273346Z layer_outputs = self.dense(hidden_states) 2025-12-04T08:56:07.0273349Z 2025-12-04T08:56:07.0273464Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0273701Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0273771Z res = mod(**inputs) 2025-12-04T08:56:07.0274093Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0274178Z outputs = self.mobilebert( 2025-12-04T08:56:07.0274508Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0274592Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0274915Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0274993Z layer_outputs = layer_module( 2025-12-04T08:56:07.0275328Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.0275434Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.0275761Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T08:56:07.0275900Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T08:56:07.0276220Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T08:56:07.0276398Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T08:56:07.0276712Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:07.0276819Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:07.0276824Z 2025-12-04T08:56:07.0276938Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0277174Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0277269Z res = mod(**inputs) 2025-12-04T08:56:07.0277582Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0277659Z outputs = self.mobilebert( 2025-12-04T08:56:07.0277976Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0278157Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0278487Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0278566Z layer_outputs = layer_module( 2025-12-04T08:56:07.0278874Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.0279011Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.0279332Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T08:56:07.0279465Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T08:56:07.0279780Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T08:56:07.0279872Z hidden_states = self.dense(hidden_states) 2025-12-04T08:56:07.0279877Z 2025-12-04T08:56:07.0280000Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0280221Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0280294Z res = mod(**inputs) 2025-12-04T08:56:07.0280619Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0280703Z outputs = self.mobilebert( 2025-12-04T08:56:07.0281027Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0281111Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0281422Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0281509Z layer_outputs = layer_module( 2025-12-04T08:56:07.0281832Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.0281942Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.0282257Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T08:56:07.0282381Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T08:56:07.0282704Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T08:56:07.0282827Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:56:07.0282830Z 2025-12-04T08:56:07.0282949Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0283172Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0283243Z res = mod(**inputs) 2025-12-04T08:56:07.0283589Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0283669Z outputs = self.mobilebert( 2025-12-04T08:56:07.0283985Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0284073Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0284407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0284508Z layer_outputs = layer_module( 2025-12-04T08:56:07.0284810Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.0284907Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.0285218Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T08:56:07.0285349Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T08:56:07.0285659Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T08:56:07.0285767Z layer_outputs = self.dense(hidden_states) 2025-12-04T08:56:07.0285771Z 2025-12-04T08:56:07.0285882Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0286102Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0286170Z res = mod(**inputs) 2025-12-04T08:56:07.0286469Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0286552Z outputs = self.mobilebert( 2025-12-04T08:56:07.0286854Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0286937Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0287239Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0287316Z layer_outputs = layer_module( 2025-12-04T08:56:07.0287628Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.0287727Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.0288041Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T08:56:07.0288173Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T08:56:07.0288471Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T08:56:07.0288609Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T08:56:07.0288911Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:07.0289009Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:07.0289021Z 2025-12-04T08:56:07.0289132Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0289344Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0289422Z res = mod(**inputs) 2025-12-04T08:56:07.0289722Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0289798Z outputs = self.mobilebert( 2025-12-04T08:56:07.0290106Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0290204Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0290514Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0290590Z layer_outputs = layer_module( 2025-12-04T08:56:07.0290914Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-12-04T08:56:07.0291069Z intermediate_output = self.intermediate(attention_output) 2025-12-04T08:56:07.0291374Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T08:56:07.0291469Z hidden_states = self.dense(hidden_states) 2025-12-04T08:56:07.0291473Z 2025-12-04T08:56:07.0291583Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0291797Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0291875Z res = mod(**inputs) 2025-12-04T08:56:07.0292187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0292264Z outputs = self.mobilebert( 2025-12-04T08:56:07.0292597Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0292676Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0292989Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0293065Z layer_outputs = layer_module( 2025-12-04T08:56:07.0293368Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-12-04T08:56:07.0293502Z intermediate_output = self.intermediate(attention_output) 2025-12-04T08:56:07.0293808Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T08:56:07.0293934Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:56:07.0293937Z 2025-12-04T08:56:07.0294048Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0294261Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0294337Z res = mod(**inputs) 2025-12-04T08:56:07.0294639Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0294714Z outputs = self.mobilebert( 2025-12-04T08:56:07.0295023Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0295098Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0295409Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0295484Z layer_outputs = layer_module( 2025-12-04T08:56:07.0295787Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T08:56:07.0295969Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T08:56:07.0296272Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-12-04T08:56:07.0296380Z layer_output = self.dense(intermediate_states) 2025-12-04T08:56:07.0296384Z 2025-12-04T08:56:07.0296492Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0296704Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0296780Z res = mod(**inputs) 2025-12-04T08:56:07.0297113Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0297191Z outputs = self.mobilebert( 2025-12-04T08:56:07.0297501Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0297596Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0297926Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0298002Z layer_outputs = layer_module( 2025-12-04T08:56:07.0298304Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T08:56:07.0298481Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T08:56:07.0298791Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-12-04T08:56:07.0298929Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-12-04T08:56:07.0299232Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:07.0299353Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:07.0299358Z 2025-12-04T08:56:07.0299476Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0299687Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0299761Z res = mod(**inputs) 2025-12-04T08:56:07.0300067Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0300144Z outputs = self.mobilebert( 2025-12-04T08:56:07.0300459Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0300537Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0300853Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0300940Z layer_outputs = layer_module( 2025-12-04T08:56:07.0301244Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T08:56:07.0301418Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T08:56:07.0301736Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-12-04T08:56:07.0301867Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-12-04T08:56:07.0302180Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-12-04T08:56:07.0302271Z layer_outputs = self.dense(hidden_states) 2025-12-04T08:56:07.0302275Z 2025-12-04T08:56:07.0302390Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0302606Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0302676Z res = mod(**inputs) 2025-12-04T08:56:07.0302986Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0303061Z outputs = self.mobilebert( 2025-12-04T08:56:07.0303365Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0303449Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0303771Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0303855Z layer_outputs = layer_module( 2025-12-04T08:56:07.0304156Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T08:56:07.0304326Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T08:56:07.0304658Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-12-04T08:56:07.0304808Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-12-04T08:56:07.0305120Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-12-04T08:56:07.0305250Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T08:56:07.0305554Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:07.0305661Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:07.0305665Z 2025-12-04T08:56:07.0305773Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0306016Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0306086Z res = mod(**inputs) 2025-12-04T08:56:07.0306391Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0306476Z outputs = self.mobilebert( 2025-12-04T08:56:07.0306777Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0306855Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0307168Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0307245Z layer_outputs = layer_module( 2025-12-04T08:56:07.0307552Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T08:56:07.0307728Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T08:56:07.0308034Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-12-04T08:56:07.0308163Z shared_attention_input = self.attention(hidden_states) 2025-12-04T08:56:07.0308466Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-12-04T08:56:07.0308564Z layer_input = self.dense(hidden_states) 2025-12-04T08:56:07.0308568Z 2025-12-04T08:56:07.0308677Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0308894Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0308971Z res = mod(**inputs) 2025-12-04T08:56:07.0309276Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0309354Z outputs = self.mobilebert( 2025-12-04T08:56:07.0309669Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0309749Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0310064Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0310139Z layer_outputs = layer_module( 2025-12-04T08:56:07.0310441Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T08:56:07.0310560Z self_attention_outputs = self.attention( 2025-12-04T08:56:07.0310860Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T08:56:07.0310944Z self_outputs = self.self( 2025-12-04T08:56:07.0311272Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-12-04T08:56:07.0311366Z self.value(value_tensor) 2025-12-04T08:56:07.0311370Z 2025-12-04T08:56:07.0311506Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0311717Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0311787Z res = mod(**inputs) 2025-12-04T08:56:07.0312098Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0312175Z outputs = self.mobilebert( 2025-12-04T08:56:07.0312484Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0312561Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0312860Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0312965Z layer_outputs = layer_module( 2025-12-04T08:56:07.0313271Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T08:56:07.0313452Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T08:56:07.0313762Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-12-04T08:56:07.0313882Z bottlenecked_hidden_states = self.input(hidden_states) 2025-12-04T08:56:07.0314198Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-12-04T08:56:07.0314290Z layer_input = self.dense(hidden_states) 2025-12-04T08:56:07.0314294Z 2025-12-04T08:56:07.0314411Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0314640Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0314710Z res = mod(**inputs) 2025-12-04T08:56:07.0315027Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0315103Z outputs = self.mobilebert( 2025-12-04T08:56:07.0315414Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0315501Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0315811Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0315894Z layer_outputs = layer_module( 2025-12-04T08:56:07.0316207Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T08:56:07.0316386Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T08:56:07.0316713Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-12-04T08:56:07.0316832Z shared_attention_input = self.attention(hidden_states) 2025-12-04T08:56:07.0317152Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-12-04T08:56:07.0317246Z layer_input = self.LayerNorm(layer_input) 2025-12-04T08:56:07.0317578Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:07.0317685Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:07.0317689Z 2025-12-04T08:56:07.0317801Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0318122Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0318250Z res = mod(**inputs) 2025-12-04T08:56:07.0318567Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0318654Z outputs = self.mobilebert( 2025-12-04T08:56:07.0318967Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0319048Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0319373Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0319453Z layer_outputs = layer_module( 2025-12-04T08:56:07.0319781Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T08:56:07.0319912Z self_attention_outputs = self.attention( 2025-12-04T08:56:07.0320218Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T08:56:07.0320306Z self_outputs = self.self( 2025-12-04T08:56:07.0320616Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-12-04T08:56:07.0320878Z self.query(query_tensor) 2025-12-04T08:56:07.0320892Z 2025-12-04T08:56:07.0350732Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0351121Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0351202Z res = mod(**inputs) 2025-12-04T08:56:07.0351537Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0351623Z outputs = self.mobilebert( 2025-12-04T08:56:07.0351933Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0352018Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0352314Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0352394Z layer_outputs = layer_module( 2025-12-04T08:56:07.0352681Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T08:56:07.0352778Z self_attention_outputs = self.attention( 2025-12-04T08:56:07.0353067Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T08:56:07.0353148Z self_outputs = self.self( 2025-12-04T08:56:07.0353438Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-12-04T08:56:07.0353514Z self.key(key_tensor) 2025-12-04T08:56:07.0353522Z 2025-12-04T08:56:07.0353621Z cudagraph partition due to non gpu ops 2025-12-04T08:56:07.0353703Z cudagraph partition due to non gpu ops 2025-12-04T08:56:07.0353823Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0354044Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0354116Z res = mod(**inputs) 2025-12-04T08:56:07.0354616Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0354703Z outputs = self.mobilebert( 2025-12-04T08:56:07.0355011Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0355102Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0355461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0355577Z layer_outputs = layer_module( 2025-12-04T08:56:07.0355895Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T08:56:07.0355993Z self_attention_outputs = self.attention( 2025-12-04T08:56:07.0356304Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-12-04T08:56:07.0356447Z attention_output = self.output(self_outputs[0], layer_input) 2025-12-04T08:56:07.0356757Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-12-04T08:56:07.0356853Z layer_outputs = self.dense(hidden_states) 2025-12-04T08:56:07.0356902Z 2025-12-04T08:56:07.0357018Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0357245Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0357316Z res = mod(**inputs) 2025-12-04T08:56:07.0357618Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0357707Z outputs = self.mobilebert( 2025-12-04T08:56:07.0358019Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0358219Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0358539Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0358618Z layer_outputs = layer_module( 2025-12-04T08:56:07.0358936Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T08:56:07.0359031Z self_attention_outputs = self.attention( 2025-12-04T08:56:07.0359350Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-12-04T08:56:07.0359488Z attention_output = self.output(self_outputs[0], layer_input) 2025-12-04T08:56:07.0359857Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-12-04T08:56:07.0359996Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T08:56:07.0360280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:07.0360377Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:07.0360381Z 2025-12-04T08:56:07.0360489Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0360699Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0360770Z res = mod(**inputs) 2025-12-04T08:56:07.0361067Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0361138Z outputs = self.mobilebert( 2025-12-04T08:56:07.0361416Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0361488Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0361796Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0361872Z layer_outputs = layer_module( 2025-12-04T08:56:07.0362149Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.0362255Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.0362551Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T08:56:07.0362687Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T08:56:07.0362977Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T08:56:07.0363064Z hidden_states = self.dense(hidden_states) 2025-12-04T08:56:07.0363068Z 2025-12-04T08:56:07.0363181Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0363383Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0363448Z res = mod(**inputs) 2025-12-04T08:56:07.0363733Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0363824Z outputs = self.mobilebert( 2025-12-04T08:56:07.0364102Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0364182Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0364455Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0364537Z layer_outputs = layer_module( 2025-12-04T08:56:07.0364816Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.0364908Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.0365195Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T08:56:07.0365311Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T08:56:07.0365602Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T08:56:07.0365715Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:56:07.0365718Z 2025-12-04T08:56:07.0365825Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0366034Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0366101Z res = mod(**inputs) 2025-12-04T08:56:07.0366394Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0366468Z outputs = self.mobilebert( 2025-12-04T08:56:07.0366760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0366842Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0367132Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0367208Z layer_outputs = layer_module( 2025-12-04T08:56:07.0367496Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.0367596Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.0367906Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T08:56:07.0368068Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T08:56:07.0368366Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T08:56:07.0368466Z layer_outputs = self.dense(hidden_states) 2025-12-04T08:56:07.0368473Z 2025-12-04T08:56:07.0368584Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0368822Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0368910Z res = mod(**inputs) 2025-12-04T08:56:07.0369212Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0369296Z outputs = self.mobilebert( 2025-12-04T08:56:07.0369597Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0369677Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0369985Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0370061Z layer_outputs = layer_module( 2025-12-04T08:56:07.0370369Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.0370494Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.0370801Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T08:56:07.0370946Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T08:56:07.0371244Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T08:56:07.0371385Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T08:56:07.0371688Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:07.0371791Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:07.0371795Z 2025-12-04T08:56:07.0371912Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0372129Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0372203Z res = mod(**inputs) 2025-12-04T08:56:07.0372513Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0372591Z outputs = self.mobilebert( 2025-12-04T08:56:07.0372898Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0372977Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0373280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0373369Z layer_outputs = layer_module( 2025-12-04T08:56:07.0373668Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.0373781Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.0374090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T08:56:07.0374212Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T08:56:07.0374531Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T08:56:07.0374627Z hidden_states = self.dense(hidden_states) 2025-12-04T08:56:07.0374630Z 2025-12-04T08:56:07.0374767Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0374989Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0375060Z res = mod(**inputs) 2025-12-04T08:56:07.0375373Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0375468Z outputs = self.mobilebert( 2025-12-04T08:56:07.0375805Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0375893Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0376195Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0376279Z layer_outputs = layer_module( 2025-12-04T08:56:07.0376577Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.0376677Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.0376980Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T08:56:07.0377122Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T08:56:07.0377432Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T08:56:07.0377557Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:56:07.0377561Z 2025-12-04T08:56:07.0377674Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0377899Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0377968Z res = mod(**inputs) 2025-12-04T08:56:07.0378281Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0378366Z outputs = self.mobilebert( 2025-12-04T08:56:07.0378662Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0378751Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0379056Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0379135Z layer_outputs = layer_module( 2025-12-04T08:56:07.0379445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.0379547Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.0379857Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T08:56:07.0379994Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T08:56:07.0380295Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T08:56:07.0380399Z layer_outputs = self.dense(hidden_states) 2025-12-04T08:56:07.0380402Z 2025-12-04T08:56:07.0380515Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0380736Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0380807Z res = mod(**inputs) 2025-12-04T08:56:07.0381112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0381191Z outputs = self.mobilebert( 2025-12-04T08:56:07.0381493Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0381594Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0381906Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0381986Z layer_outputs = layer_module( 2025-12-04T08:56:07.0382322Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.0382437Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.0382743Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T08:56:07.0382888Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T08:56:07.0383194Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T08:56:07.0383334Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T08:56:07.0383640Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:07.0383740Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:07.0383762Z 2025-12-04T08:56:07.0383883Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0384099Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0384170Z res = mod(**inputs) 2025-12-04T08:56:07.0384484Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0384559Z outputs = self.mobilebert( 2025-12-04T08:56:07.0384870Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0384950Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0385253Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0385340Z layer_outputs = layer_module( 2025-12-04T08:56:07.0385641Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.0385752Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.0386059Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T08:56:07.0386180Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T08:56:07.0386490Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T08:56:07.0386582Z hidden_states = self.dense(hidden_states) 2025-12-04T08:56:07.0386585Z 2025-12-04T08:56:07.0386693Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0386902Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0386968Z res = mod(**inputs) 2025-12-04T08:56:07.0387272Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0387350Z outputs = self.mobilebert( 2025-12-04T08:56:07.0387666Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0387752Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0388070Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0388147Z layer_outputs = layer_module( 2025-12-04T08:56:07.0388480Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.0388580Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.0388894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T08:56:07.0389014Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T08:56:07.0389357Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T08:56:07.0389504Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:56:07.0389508Z 2025-12-04T08:56:07.0389617Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0389828Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0389905Z res = mod(**inputs) 2025-12-04T08:56:07.0390208Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0390291Z outputs = self.mobilebert( 2025-12-04T08:56:07.0390614Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0390710Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0391023Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0391102Z layer_outputs = layer_module( 2025-12-04T08:56:07.0391411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.0391509Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.0391821Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T08:56:07.0391962Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T08:56:07.0392274Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T08:56:07.0392367Z layer_outputs = self.dense(hidden_states) 2025-12-04T08:56:07.0392378Z 2025-12-04T08:56:07.0392490Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0392702Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0392777Z res = mod(**inputs) 2025-12-04T08:56:07.0393078Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0393153Z outputs = self.mobilebert( 2025-12-04T08:56:07.0393465Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0393543Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0393859Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0393938Z layer_outputs = layer_module( 2025-12-04T08:56:07.0394251Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.0394357Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.0394676Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T08:56:07.0394814Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T08:56:07.0395132Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T08:56:07.0395282Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T08:56:07.0395609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:07.0395710Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:07.0395716Z 2025-12-04T08:56:07.0395827Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0396082Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0396152Z res = mod(**inputs) 2025-12-04T08:56:07.0396465Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0396544Z outputs = self.mobilebert( 2025-12-04T08:56:07.0396858Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0396949Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0397265Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0397352Z layer_outputs = layer_module( 2025-12-04T08:56:07.0397682Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-12-04T08:56:07.0397819Z intermediate_output = self.intermediate(attention_output) 2025-12-04T08:56:07.0398254Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T08:56:07.0398352Z hidden_states = self.dense(hidden_states) 2025-12-04T08:56:07.0398356Z 2025-12-04T08:56:07.0398468Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0398690Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0398766Z res = mod(**inputs) 2025-12-04T08:56:07.0399097Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0399177Z outputs = self.mobilebert( 2025-12-04T08:56:07.0399501Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0399596Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0399917Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0400005Z layer_outputs = layer_module( 2025-12-04T08:56:07.0400332Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-12-04T08:56:07.0400478Z intermediate_output = self.intermediate(attention_output) 2025-12-04T08:56:07.0400802Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T08:56:07.0400922Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:56:07.0400926Z 2025-12-04T08:56:07.0401054Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0401263Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0401332Z res = mod(**inputs) 2025-12-04T08:56:07.0401634Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0401707Z outputs = self.mobilebert( 2025-12-04T08:56:07.0401999Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0402080Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0402413Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0402496Z layer_outputs = layer_module( 2025-12-04T08:56:07.0402811Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T08:56:07.0403017Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T08:56:07.0403348Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-12-04T08:56:07.0403450Z layer_output = self.dense(intermediate_states) 2025-12-04T08:56:07.0403455Z 2025-12-04T08:56:07.0403569Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0403784Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0403851Z res = mod(**inputs) 2025-12-04T08:56:07.0404171Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0404250Z outputs = self.mobilebert( 2025-12-04T08:56:07.0404568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0404666Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0404975Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0405053Z layer_outputs = layer_module( 2025-12-04T08:56:07.0405362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T08:56:07.0405533Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T08:56:07.0405852Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-12-04T08:56:07.0405987Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-12-04T08:56:07.0406308Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:07.0406411Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:07.0406418Z 2025-12-04T08:56:07.0406529Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0406752Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0406820Z res = mod(**inputs) 2025-12-04T08:56:07.0407128Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0407211Z outputs = self.mobilebert( 2025-12-04T08:56:07.0407520Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0407600Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0407909Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0407986Z layer_outputs = layer_module( 2025-12-04T08:56:07.0408302Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T08:56:07.0408476Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T08:56:07.0408791Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-12-04T08:56:07.0408925Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-12-04T08:56:07.0409254Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-12-04T08:56:07.0409353Z layer_outputs = self.dense(hidden_states) 2025-12-04T08:56:07.0409357Z 2025-12-04T08:56:07.0409471Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0409697Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0409785Z res = mod(**inputs) 2025-12-04T08:56:07.0410114Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0410195Z outputs = self.mobilebert( 2025-12-04T08:56:07.0410507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0410583Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0410900Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0410979Z layer_outputs = layer_module( 2025-12-04T08:56:07.0411296Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T08:56:07.0411486Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T08:56:07.0411796Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-12-04T08:56:07.0411941Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-12-04T08:56:07.0412252Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-12-04T08:56:07.0412394Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T08:56:07.0412709Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:07.0412809Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:07.0412813Z 2025-12-04T08:56:07.0412930Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0413149Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0413220Z res = mod(**inputs) 2025-12-04T08:56:07.0413539Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0413616Z outputs = self.mobilebert( 2025-12-04T08:56:07.0413933Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0414014Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0414325Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0414411Z layer_outputs = layer_module( 2025-12-04T08:56:07.0414727Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T08:56:07.0414916Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T08:56:07.0415233Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-12-04T08:56:07.0415359Z shared_attention_input = self.attention(hidden_states) 2025-12-04T08:56:07.0415676Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-12-04T08:56:07.0415768Z layer_input = self.dense(hidden_states) 2025-12-04T08:56:07.0415772Z 2025-12-04T08:56:07.0415894Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0416134Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0416209Z res = mod(**inputs) 2025-12-04T08:56:07.0416532Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0416612Z outputs = self.mobilebert( 2025-12-04T08:56:07.0416942Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0417061Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0417378Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0417464Z layer_outputs = layer_module( 2025-12-04T08:56:07.0417783Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T08:56:07.0417882Z self_attention_outputs = self.attention( 2025-12-04T08:56:07.0418210Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T08:56:07.0418290Z self_outputs = self.self( 2025-12-04T08:56:07.0418631Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-12-04T08:56:07.0418711Z self.value(value_tensor) 2025-12-04T08:56:07.0418715Z 2025-12-04T08:56:07.0418828Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0419051Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0419119Z res = mod(**inputs) 2025-12-04T08:56:07.0419428Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0419514Z outputs = self.mobilebert( 2025-12-04T08:56:07.0419825Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0419909Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0420218Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0420297Z layer_outputs = layer_module( 2025-12-04T08:56:07.0420615Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T08:56:07.0421084Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T08:56:07.0421409Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-12-04T08:56:07.0421534Z bottlenecked_hidden_states = self.input(hidden_states) 2025-12-04T08:56:07.0421851Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-12-04T08:56:07.0421948Z layer_input = self.dense(hidden_states) 2025-12-04T08:56:07.0421952Z 2025-12-04T08:56:07.0422062Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0422283Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0422364Z res = mod(**inputs) 2025-12-04T08:56:07.0422690Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0422778Z outputs = self.mobilebert( 2025-12-04T08:56:07.0423084Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0423161Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0423538Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0423615Z layer_outputs = layer_module( 2025-12-04T08:56:07.0423934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T08:56:07.0424142Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T08:56:07.0424484Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-12-04T08:56:07.0424610Z shared_attention_input = self.attention(hidden_states) 2025-12-04T08:56:07.0424921Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-12-04T08:56:07.0425020Z layer_input = self.LayerNorm(layer_input) 2025-12-04T08:56:07.0425333Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:07.0425436Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:07.0425440Z 2025-12-04T08:56:07.0425559Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0425804Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0425879Z res = mod(**inputs) 2025-12-04T08:56:07.0426201Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0426277Z outputs = self.mobilebert( 2025-12-04T08:56:07.0426596Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0426675Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0426985Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0427067Z layer_outputs = layer_module( 2025-12-04T08:56:07.0427381Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T08:56:07.0427485Z self_attention_outputs = self.attention( 2025-12-04T08:56:07.0427798Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T08:56:07.0427879Z self_outputs = self.self( 2025-12-04T08:56:07.0428195Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-12-04T08:56:07.0428274Z self.query(query_tensor) 2025-12-04T08:56:07.0428278Z 2025-12-04T08:56:07.0428389Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0428617Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0428685Z res = mod(**inputs) 2025-12-04T08:56:07.0429001Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0429079Z outputs = self.mobilebert( 2025-12-04T08:56:07.0429389Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0429474Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0429796Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0429879Z layer_outputs = layer_module( 2025-12-04T08:56:07.0430191Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T08:56:07.0430320Z self_attention_outputs = self.attention( 2025-12-04T08:56:07.0430641Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T08:56:07.0430719Z self_outputs = self.self( 2025-12-04T08:56:07.0431025Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-12-04T08:56:07.0431129Z self.key(key_tensor) 2025-12-04T08:56:07.0431150Z 2025-12-04T08:56:07.0431243Z cudagraph partition due to non gpu ops 2025-12-04T08:56:07.0431341Z cudagraph partition due to non gpu ops 2025-12-04T08:56:07.0431456Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0431676Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0431754Z res = mod(**inputs) 2025-12-04T08:56:07.0432067Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0432145Z outputs = self.mobilebert( 2025-12-04T08:56:07.0432461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0432542Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0432883Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0432961Z layer_outputs = layer_module( 2025-12-04T08:56:07.0433278Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T08:56:07.0433378Z self_attention_outputs = self.attention( 2025-12-04T08:56:07.0433695Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-12-04T08:56:07.0433838Z attention_output = self.output(self_outputs[0], layer_input) 2025-12-04T08:56:07.0434150Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-12-04T08:56:07.0434244Z layer_outputs = self.dense(hidden_states) 2025-12-04T08:56:07.0434249Z 2025-12-04T08:56:07.0434368Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0434589Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0434659Z res = mod(**inputs) 2025-12-04T08:56:07.0434976Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0435054Z outputs = self.mobilebert( 2025-12-04T08:56:07.0435373Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0435453Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0435763Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0435848Z layer_outputs = layer_module( 2025-12-04T08:56:07.0436170Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T08:56:07.0436272Z self_attention_outputs = self.attention( 2025-12-04T08:56:07.0436586Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-12-04T08:56:07.0436720Z attention_output = self.output(self_outputs[0], layer_input) 2025-12-04T08:56:07.0437044Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-12-04T08:56:07.0437182Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T08:56:07.0437523Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:07.0437626Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:07.0437630Z 2025-12-04T08:56:07.0437744Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0437986Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0438139Z res = mod(**inputs) 2025-12-04T08:56:07.0438461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0438547Z outputs = self.mobilebert( 2025-12-04T08:56:07.0438867Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0438952Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0439277Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0439354Z layer_outputs = layer_module( 2025-12-04T08:56:07.0439671Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.0439800Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.0440123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T08:56:07.0440249Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T08:56:07.0440560Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T08:56:07.0440662Z hidden_states = self.dense(hidden_states) 2025-12-04T08:56:07.0440666Z 2025-12-04T08:56:07.0440782Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0441003Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0441084Z res = mod(**inputs) 2025-12-04T08:56:07.0441392Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0441476Z outputs = self.mobilebert( 2025-12-04T08:56:07.0441789Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0441870Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0442187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0442267Z layer_outputs = layer_module( 2025-12-04T08:56:07.0442596Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.0442701Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.0443012Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T08:56:07.0443142Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T08:56:07.0443454Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T08:56:07.0443579Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:56:07.0443588Z 2025-12-04T08:56:07.0443699Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0443919Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0443995Z res = mod(**inputs) 2025-12-04T08:56:07.0444329Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0444408Z outputs = self.mobilebert( 2025-12-04T08:56:07.0444737Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0444816Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0445154Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0445249Z layer_outputs = layer_module( 2025-12-04T08:56:07.0445559Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.0445663Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.0445973Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T08:56:07.0446109Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T08:56:07.0446430Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T08:56:07.0446517Z layer_outputs = self.dense(hidden_states) 2025-12-04T08:56:07.0446541Z 2025-12-04T08:56:07.0446656Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0446868Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0446937Z res = mod(**inputs) 2025-12-04T08:56:07.0447246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0447320Z outputs = self.mobilebert( 2025-12-04T08:56:07.0447638Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0447714Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0448015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0448098Z layer_outputs = layer_module( 2025-12-04T08:56:07.0448409Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.0448511Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.0448824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T08:56:07.0448955Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T08:56:07.0449264Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T08:56:07.0449395Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T08:56:07.0449697Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:07.0449799Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:07.0449803Z 2025-12-04T08:56:07.0449916Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0450138Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0450207Z res = mod(**inputs) 2025-12-04T08:56:07.0450510Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0450590Z outputs = self.mobilebert( 2025-12-04T08:56:07.0450894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0450978Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0451302Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0451381Z layer_outputs = layer_module( 2025-12-04T08:56:07.0451686Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.0451806Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.0452123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T08:56:07.0452250Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T08:56:07.0452552Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T08:56:07.0452651Z hidden_states = self.dense(hidden_states) 2025-12-04T08:56:07.0452655Z 2025-12-04T08:56:07.0452766Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0452981Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0453053Z res = mod(**inputs) 2025-12-04T08:56:07.0453353Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0453466Z outputs = self.mobilebert( 2025-12-04T08:56:07.0453766Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0453843Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0454154Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0454231Z layer_outputs = layer_module( 2025-12-04T08:56:07.0454543Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.0454648Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.0454958Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T08:56:07.0455089Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T08:56:07.0455402Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T08:56:07.0455525Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:56:07.0455528Z 2025-12-04T08:56:07.0455645Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0455864Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0455939Z res = mod(**inputs) 2025-12-04T08:56:07.0456248Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0456324Z outputs = self.mobilebert( 2025-12-04T08:56:07.0456644Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0456731Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0457038Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0457116Z layer_outputs = layer_module( 2025-12-04T08:56:07.0457417Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.0457525Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.0457829Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T08:56:07.0457982Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T08:56:07.0458290Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T08:56:07.0458377Z layer_outputs = self.dense(hidden_states) 2025-12-04T08:56:07.0458382Z 2025-12-04T08:56:07.0458496Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0458751Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0458821Z res = mod(**inputs) 2025-12-04T08:56:07.0459129Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0459204Z outputs = self.mobilebert( 2025-12-04T08:56:07.0459514Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0459591Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0459890Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0459974Z layer_outputs = layer_module( 2025-12-04T08:56:07.0460300Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.0460407Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.0460710Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T08:56:07.0460839Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T08:56:07.0461150Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T08:56:07.0461280Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T08:56:07.0461580Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:07.0461682Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:07.0461688Z 2025-12-04T08:56:07.0461795Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0462010Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0462080Z res = mod(**inputs) 2025-12-04T08:56:07.0462379Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0462462Z outputs = self.mobilebert( 2025-12-04T08:56:07.0462771Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0462856Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0463168Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0463243Z layer_outputs = layer_module( 2025-12-04T08:56:07.0463565Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.0463671Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.0463988Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T08:56:07.0464113Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T08:56:07.0464426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T08:56:07.0464524Z hidden_states = self.dense(hidden_states) 2025-12-04T08:56:07.0464527Z 2025-12-04T08:56:07.0464655Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0464871Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0464946Z res = mod(**inputs) 2025-12-04T08:56:07.0465246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0465344Z outputs = self.mobilebert( 2025-12-04T08:56:07.0465661Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0465738Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0466051Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0466127Z layer_outputs = layer_module( 2025-12-04T08:56:07.0466435Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.0466535Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.0466834Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T08:56:07.0466988Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T08:56:07.0467295Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T08:56:07.0467415Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:56:07.0467426Z 2025-12-04T08:56:07.0467535Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0467749Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0467829Z res = mod(**inputs) 2025-12-04T08:56:07.0468136Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0468217Z outputs = self.mobilebert( 2025-12-04T08:56:07.0468530Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0468611Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0468927Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0469008Z layer_outputs = layer_module( 2025-12-04T08:56:07.0469313Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.0469421Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.0469734Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T08:56:07.0469873Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T08:56:07.0470194Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T08:56:07.0470293Z layer_outputs = self.dense(hidden_states) 2025-12-04T08:56:07.0470297Z 2025-12-04T08:56:07.0470430Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0470645Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0470715Z res = mod(**inputs) 2025-12-04T08:56:07.0471024Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0471103Z outputs = self.mobilebert( 2025-12-04T08:56:07.0471432Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0471511Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0471809Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0471887Z layer_outputs = layer_module( 2025-12-04T08:56:07.0472211Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.0472330Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.0472636Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T08:56:07.0472767Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T08:56:07.0473075Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T08:56:07.0473206Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T08:56:07.0473512Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:07.0473620Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:07.0473642Z 2025-12-04T08:56:07.0473755Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0473981Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0474050Z res = mod(**inputs) 2025-12-04T08:56:07.0474362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0474445Z outputs = self.mobilebert( 2025-12-04T08:56:07.0474752Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0474836Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0475146Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0475220Z layer_outputs = layer_module( 2025-12-04T08:56:07.0475537Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-12-04T08:56:07.0475671Z intermediate_output = self.intermediate(attention_output) 2025-12-04T08:56:07.0475983Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T08:56:07.0476080Z hidden_states = self.dense(hidden_states) 2025-12-04T08:56:07.0476084Z 2025-12-04T08:56:07.0476194Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0476416Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0476488Z res = mod(**inputs) 2025-12-04T08:56:07.0476798Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0476882Z outputs = self.mobilebert( 2025-12-04T08:56:07.0477194Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0477281Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0477591Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0477665Z layer_outputs = layer_module( 2025-12-04T08:56:07.0477983Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-12-04T08:56:07.0478207Z intermediate_output = self.intermediate(attention_output) 2025-12-04T08:56:07.0478544Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T08:56:07.0478676Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:56:07.0478681Z 2025-12-04T08:56:07.0478794Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0479040Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0479139Z res = mod(**inputs) 2025-12-04T08:56:07.0479452Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0479539Z outputs = self.mobilebert( 2025-12-04T08:56:07.0479871Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0479950Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0480264Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0480343Z layer_outputs = layer_module( 2025-12-04T08:56:07.0480665Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T08:56:07.0480864Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T08:56:07.0481180Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-12-04T08:56:07.0481293Z layer_output = self.dense(intermediate_states) 2025-12-04T08:56:07.0481297Z 2025-12-04T08:56:07.0481412Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0481640Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0481710Z res = mod(**inputs) 2025-12-04T08:56:07.0482021Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0482107Z outputs = self.mobilebert( 2025-12-04T08:56:07.0482419Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0482507Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0482820Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0482898Z layer_outputs = layer_module( 2025-12-04T08:56:07.0483216Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T08:56:07.0483389Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T08:56:07.0483721Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-12-04T08:56:07.0483858Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-12-04T08:56:07.0484170Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:07.0484283Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:07.0484290Z 2025-12-04T08:56:07.0484405Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0484623Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0484701Z res = mod(**inputs) 2025-12-04T08:56:07.0485013Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0485099Z outputs = self.mobilebert( 2025-12-04T08:56:07.0485433Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0485515Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0485839Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0485921Z layer_outputs = layer_module( 2025-12-04T08:56:07.0486272Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T08:56:07.0486467Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T08:56:07.0486787Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-12-04T08:56:07.0486928Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-12-04T08:56:07.0487240Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-12-04T08:56:07.0487342Z layer_outputs = self.dense(hidden_states) 2025-12-04T08:56:07.0487346Z 2025-12-04T08:56:07.0487458Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0487697Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0487776Z res = mod(**inputs) 2025-12-04T08:56:07.0488097Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0488173Z outputs = self.mobilebert( 2025-12-04T08:56:07.0488503Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0488581Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0488911Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0488989Z layer_outputs = layer_module( 2025-12-04T08:56:07.0489296Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T08:56:07.0489477Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T08:56:07.0489802Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-12-04T08:56:07.0489945Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-12-04T08:56:07.0490257Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-12-04T08:56:07.0490402Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T08:56:07.0490713Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:07.0490812Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:07.0490815Z 2025-12-04T08:56:07.0490931Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0491144Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0491215Z res = mod(**inputs) 2025-12-04T08:56:07.0491528Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0491604Z outputs = self.mobilebert( 2025-12-04T08:56:07.0491903Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0491988Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0492311Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0492394Z layer_outputs = layer_module( 2025-12-04T08:56:07.0492698Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T08:56:07.0492874Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T08:56:07.0493202Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-12-04T08:56:07.0493338Z shared_attention_input = self.attention(hidden_states) 2025-12-04T08:56:07.0493647Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-12-04T08:56:07.0493736Z layer_input = self.dense(hidden_states) 2025-12-04T08:56:07.0493740Z 2025-12-04T08:56:07.0493849Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0494086Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0494154Z res = mod(**inputs) 2025-12-04T08:56:07.0494465Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0494559Z outputs = self.mobilebert( 2025-12-04T08:56:07.0494862Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0494951Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0495252Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0495335Z layer_outputs = layer_module( 2025-12-04T08:56:07.0495636Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T08:56:07.0495729Z self_attention_outputs = self.attention( 2025-12-04T08:56:07.0496039Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T08:56:07.0496115Z self_outputs = self.self( 2025-12-04T08:56:07.0496419Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-12-04T08:56:07.0496504Z self.value(value_tensor) 2025-12-04T08:56:07.0496508Z 2025-12-04T08:56:07.0496617Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0496837Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0496905Z res = mod(**inputs) 2025-12-04T08:56:07.0497207Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0497291Z outputs = self.mobilebert( 2025-12-04T08:56:07.0497592Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0497677Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0497977Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0498057Z layer_outputs = layer_module( 2025-12-04T08:56:07.0498364Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T08:56:07.0498534Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T08:56:07.0498844Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-12-04T08:56:07.0498970Z bottlenecked_hidden_states = self.input(hidden_states) 2025-12-04T08:56:07.0499291Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-12-04T08:56:07.0499391Z layer_input = self.dense(hidden_states) 2025-12-04T08:56:07.0499395Z 2025-12-04T08:56:07.0499508Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0499747Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0499841Z res = mod(**inputs) 2025-12-04T08:56:07.0500141Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0500225Z outputs = self.mobilebert( 2025-12-04T08:56:07.0500525Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0500604Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0500916Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0500992Z layer_outputs = layer_module( 2025-12-04T08:56:07.0501299Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T08:56:07.0501492Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T08:56:07.0501805Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-12-04T08:56:07.0501927Z shared_attention_input = self.attention(hidden_states) 2025-12-04T08:56:07.0502239Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-12-04T08:56:07.0502330Z layer_input = self.LayerNorm(layer_input) 2025-12-04T08:56:07.0502643Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:07.0502740Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:07.0502744Z 2025-12-04T08:56:07.0502859Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0503073Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0503142Z res = mod(**inputs) 2025-12-04T08:56:07.0503453Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0503527Z outputs = self.mobilebert( 2025-12-04T08:56:07.0503847Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0503922Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0504238Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0504320Z layer_outputs = layer_module( 2025-12-04T08:56:07.0504635Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T08:56:07.0504727Z self_attention_outputs = self.attention( 2025-12-04T08:56:07.0505041Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T08:56:07.0505121Z self_outputs = self.self( 2025-12-04T08:56:07.0505440Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-12-04T08:56:07.0505518Z self.query(query_tensor) 2025-12-04T08:56:07.0505521Z 2025-12-04T08:56:07.0505631Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0505869Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0505939Z res = mod(**inputs) 2025-12-04T08:56:07.0506253Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0506331Z outputs = self.mobilebert( 2025-12-04T08:56:07.0506661Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0506766Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0507074Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0507150Z layer_outputs = layer_module( 2025-12-04T08:56:07.0507476Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T08:56:07.0507568Z self_attention_outputs = self.attention( 2025-12-04T08:56:07.0507884Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T08:56:07.0507961Z self_outputs = self.self( 2025-12-04T08:56:07.0508267Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-12-04T08:56:07.0508370Z self.key(key_tensor) 2025-12-04T08:56:07.0508375Z 2025-12-04T08:56:07.0508464Z cudagraph partition due to non gpu ops 2025-12-04T08:56:07.0508560Z cudagraph partition due to non gpu ops 2025-12-04T08:56:07.0508670Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0508882Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0508957Z res = mod(**inputs) 2025-12-04T08:56:07.0509261Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0509335Z outputs = self.mobilebert( 2025-12-04T08:56:07.0509645Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0509722Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0510036Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0510112Z layer_outputs = layer_module( 2025-12-04T08:56:07.0510414Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T08:56:07.0510512Z self_attention_outputs = self.attention( 2025-12-04T08:56:07.0510823Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-12-04T08:56:07.0510959Z attention_output = self.output(self_outputs[0], layer_input) 2025-12-04T08:56:07.0511279Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-12-04T08:56:07.0511369Z layer_outputs = self.dense(hidden_states) 2025-12-04T08:56:07.0511373Z 2025-12-04T08:56:07.0511491Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0511705Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0511776Z res = mod(**inputs) 2025-12-04T08:56:07.0512094Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0512169Z outputs = self.mobilebert( 2025-12-04T08:56:07.0512488Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0512566Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0512904Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0512989Z layer_outputs = layer_module( 2025-12-04T08:56:07.0513301Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T08:56:07.0513414Z self_attention_outputs = self.attention( 2025-12-04T08:56:07.0513751Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-12-04T08:56:07.0513884Z attention_output = self.output(self_outputs[0], layer_input) 2025-12-04T08:56:07.0514200Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-12-04T08:56:07.0514334Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T08:56:07.0514640Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:07.0514747Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:07.0514750Z 2025-12-04T08:56:07.0514860Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0515104Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0515176Z res = mod(**inputs) 2025-12-04T08:56:07.0515491Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0515574Z outputs = self.mobilebert( 2025-12-04T08:56:07.0515887Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0515970Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0516283Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0516359Z layer_outputs = layer_module( 2025-12-04T08:56:07.0516678Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.0516783Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.0517090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T08:56:07.0517220Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T08:56:07.0517535Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T08:56:07.0517633Z hidden_states = self.dense(hidden_states) 2025-12-04T08:56:07.0517636Z 2025-12-04T08:56:07.0517744Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0517964Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0518118Z res = mod(**inputs) 2025-12-04T08:56:07.0518450Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0518541Z outputs = self.mobilebert( 2025-12-04T08:56:07.0518866Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0518946Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0519264Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0519342Z layer_outputs = layer_module( 2025-12-04T08:56:07.0519736Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.0519847Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.0520157Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T08:56:07.0520287Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T08:56:07.0520626Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T08:56:07.0521043Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:56:07.0521049Z 2025-12-04T08:56:07.0521176Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0521398Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0521478Z res = mod(**inputs) 2025-12-04T08:56:07.0521794Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0521874Z outputs = self.mobilebert( 2025-12-04T08:56:07.0522193Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0522329Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0522650Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0522730Z layer_outputs = layer_module( 2025-12-04T08:56:07.0523042Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.0523153Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.0523468Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T08:56:07.0523607Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T08:56:07.0523926Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T08:56:07.0524019Z layer_outputs = self.dense(hidden_states) 2025-12-04T08:56:07.0524025Z 2025-12-04T08:56:07.0524145Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0524364Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0524437Z res = mod(**inputs) 2025-12-04T08:56:07.0524755Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0524834Z outputs = self.mobilebert( 2025-12-04T08:56:07.0525153Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0525235Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0525547Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0525634Z layer_outputs = layer_module( 2025-12-04T08:56:07.0525959Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.0526066Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.0526388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T08:56:07.0526526Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T08:56:07.0526847Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T08:56:07.0526983Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T08:56:07.0527354Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:07.0527465Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:07.0527469Z 2025-12-04T08:56:07.0527583Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0527840Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0527950Z res = mod(**inputs) 2025-12-04T08:56:07.0528264Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0528352Z outputs = self.mobilebert( 2025-12-04T08:56:07.0528671Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0528751Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0529089Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0529168Z layer_outputs = layer_module( 2025-12-04T08:56:07.0529489Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.0529615Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.0529928Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T08:56:07.0530060Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T08:56:07.0530382Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T08:56:07.0530482Z hidden_states = self.dense(hidden_states) 2025-12-04T08:56:07.0530486Z 2025-12-04T08:56:07.0530599Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0530818Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0530895Z res = mod(**inputs) 2025-12-04T08:56:07.0531221Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0531306Z outputs = self.mobilebert( 2025-12-04T08:56:07.0531627Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0531707Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0532029Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0532109Z layer_outputs = layer_module( 2025-12-04T08:56:07.0532425Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.0532538Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.0532852Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T08:56:07.0532983Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T08:56:07.0533299Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T08:56:07.0533425Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:56:07.0533428Z 2025-12-04T08:56:07.0533550Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0533775Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0533853Z res = mod(**inputs) 2025-12-04T08:56:07.0534195Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0534274Z outputs = self.mobilebert( 2025-12-04T08:56:07.0534592Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0534674Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0535014Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0535121Z layer_outputs = layer_module( 2025-12-04T08:56:07.0535432Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.0535541Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.0535850Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T08:56:07.0535989Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T08:56:07.0536307Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T08:56:07.0536400Z layer_outputs = self.dense(hidden_states) 2025-12-04T08:56:07.0536425Z 2025-12-04T08:56:07.0536547Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0536768Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0536838Z res = mod(**inputs) 2025-12-04T08:56:07.0537169Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0537246Z outputs = self.mobilebert( 2025-12-04T08:56:07.0537556Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0537642Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0537954Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0538037Z layer_outputs = layer_module( 2025-12-04T08:56:07.0538351Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.0538453Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.0538766Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T08:56:07.0538901Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T08:56:07.0539219Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T08:56:07.0539356Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T08:56:07.0539669Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:07.0539779Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:07.0539784Z 2025-12-04T08:56:07.0539897Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0540123Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0540196Z res = mod(**inputs) 2025-12-04T08:56:07.0540505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0540589Z outputs = self.mobilebert( 2025-12-04T08:56:07.0540904Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0541006Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0541327Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0541405Z layer_outputs = layer_module( 2025-12-04T08:56:07.0541723Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.0541850Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.0542184Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T08:56:07.0542315Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T08:56:07.0542626Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T08:56:07.0542727Z hidden_states = self.dense(hidden_states) 2025-12-04T08:56:07.0542730Z 2025-12-04T08:56:07.0542846Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0543064Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0543145Z res = mod(**inputs) 2025-12-04T08:56:07.0543458Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0543557Z outputs = self.mobilebert( 2025-12-04T08:56:07.0543877Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0543957Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0544277Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0544355Z layer_outputs = layer_module( 2025-12-04T08:56:07.0544679Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.0544790Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.0545102Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T08:56:07.0545234Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T08:56:07.0545547Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T08:56:07.0545672Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:56:07.0545676Z 2025-12-04T08:56:07.0545797Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0546021Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0546090Z res = mod(**inputs) 2025-12-04T08:56:07.0546414Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0546491Z outputs = self.mobilebert( 2025-12-04T08:56:07.0546809Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0546892Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0547206Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0547293Z layer_outputs = layer_module( 2025-12-04T08:56:07.0547604Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.0547713Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.0548063Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T08:56:07.0548197Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T08:56:07.0548503Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T08:56:07.0548595Z layer_outputs = self.dense(hidden_states) 2025-12-04T08:56:07.0548599Z 2025-12-04T08:56:07.0548731Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0548963Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0549033Z res = mod(**inputs) 2025-12-04T08:56:07.0549348Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0549425Z outputs = self.mobilebert( 2025-12-04T08:56:07.0549738Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0549828Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0550138Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0550224Z layer_outputs = layer_module( 2025-12-04T08:56:07.0550562Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.0550665Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.0550981Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T08:56:07.0551122Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T08:56:07.0551441Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T08:56:07.0551582Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T08:56:07.0551897Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:07.0552005Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:07.0552010Z 2025-12-04T08:56:07.0552125Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0552345Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0552425Z res = mod(**inputs) 2025-12-04T08:56:07.0552737Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0552823Z outputs = self.mobilebert( 2025-12-04T08:56:07.0553135Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0553216Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0553538Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0553616Z layer_outputs = layer_module( 2025-12-04T08:56:07.0553937Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-12-04T08:56:07.0554077Z intermediate_output = self.intermediate(attention_output) 2025-12-04T08:56:07.0554389Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T08:56:07.0554490Z hidden_states = self.dense(hidden_states) 2025-12-04T08:56:07.0554493Z 2025-12-04T08:56:07.0554606Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0554831Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0554927Z res = mod(**inputs) 2025-12-04T08:56:07.0555244Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0555328Z outputs = self.mobilebert( 2025-12-04T08:56:07.0555648Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0555766Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0556087Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0556165Z layer_outputs = layer_module( 2025-12-04T08:56:07.0556484Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-12-04T08:56:07.0556616Z intermediate_output = self.intermediate(attention_output) 2025-12-04T08:56:07.0556931Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T08:56:07.0557061Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:56:07.0557065Z 2025-12-04T08:56:07.0557178Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0557434Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0557505Z res = mod(**inputs) 2025-12-04T08:56:07.0557816Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0557900Z outputs = self.mobilebert( 2025-12-04T08:56:07.0558295Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0558382Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0558706Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0558785Z layer_outputs = layer_module( 2025-12-04T08:56:07.0559104Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T08:56:07.0559285Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T08:56:07.0559598Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-12-04T08:56:07.0559711Z layer_output = self.dense(intermediate_states) 2025-12-04T08:56:07.0559715Z 2025-12-04T08:56:07.0559841Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0560065Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0560135Z res = mod(**inputs) 2025-12-04T08:56:07.0560439Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0560524Z outputs = self.mobilebert( 2025-12-04T08:56:07.0561068Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0561200Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0561673Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0561759Z layer_outputs = layer_module( 2025-12-04T08:56:07.0562071Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T08:56:07.0562241Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T08:56:07.0562588Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-12-04T08:56:07.0562731Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-12-04T08:56:07.0563045Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:07.0563154Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:07.0563178Z 2025-12-04T08:56:07.0563310Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0563524Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0563602Z res = mod(**inputs) 2025-12-04T08:56:07.0563918Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0563994Z outputs = self.mobilebert( 2025-12-04T08:56:07.0564321Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0564402Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0564724Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0564827Z layer_outputs = layer_module( 2025-12-04T08:56:07.0565157Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T08:56:07.0565340Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T08:56:07.0565655Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-12-04T08:56:07.0565799Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-12-04T08:56:07.0566116Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-12-04T08:56:07.0566211Z layer_outputs = self.dense(hidden_states) 2025-12-04T08:56:07.0566215Z 2025-12-04T08:56:07.0566339Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0566559Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0566639Z res = mod(**inputs) 2025-12-04T08:56:07.0566967Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0567047Z outputs = self.mobilebert( 2025-12-04T08:56:07.0567372Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0567451Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0567770Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0567855Z layer_outputs = layer_module( 2025-12-04T08:56:07.0568170Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T08:56:07.0568352Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T08:56:07.0568677Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-12-04T08:56:07.0568814Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-12-04T08:56:07.0569138Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-12-04T08:56:07.0569270Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T08:56:07.0569612Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:07.0569717Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:07.0569721Z 2025-12-04T08:56:07.0569836Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0570065Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0570138Z res = mod(**inputs) 2025-12-04T08:56:07.0570477Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0570574Z outputs = self.mobilebert( 2025-12-04T08:56:07.0570889Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0570976Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0571317Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0571394Z layer_outputs = layer_module( 2025-12-04T08:56:07.0571717Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T08:56:07.0571898Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T08:56:07.0572243Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-12-04T08:56:07.0572367Z shared_attention_input = self.attention(hidden_states) 2025-12-04T08:56:07.0572680Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-12-04T08:56:07.0572780Z layer_input = self.dense(hidden_states) 2025-12-04T08:56:07.0572784Z 2025-12-04T08:56:07.0572897Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0573124Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0573194Z res = mod(**inputs) 2025-12-04T08:56:07.0573510Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0573598Z outputs = self.mobilebert( 2025-12-04T08:56:07.0573909Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0573992Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0574308Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0574386Z layer_outputs = layer_module( 2025-12-04T08:56:07.0574704Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T08:56:07.0574800Z self_attention_outputs = self.attention( 2025-12-04T08:56:07.0575112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T08:56:07.0575196Z self_outputs = self.self( 2025-12-04T08:56:07.0575507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-12-04T08:56:07.0575596Z self.value(value_tensor) 2025-12-04T08:56:07.0575600Z 2025-12-04T08:56:07.0575713Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0575930Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0576008Z res = mod(**inputs) 2025-12-04T08:56:07.0576320Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0576398Z outputs = self.mobilebert( 2025-12-04T08:56:07.0576736Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0576818Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0577134Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0577232Z layer_outputs = layer_module( 2025-12-04T08:56:07.0577561Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T08:56:07.0577744Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T08:56:07.0578057Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-12-04T08:56:07.0578186Z bottlenecked_hidden_states = self.input(hidden_states) 2025-12-04T08:56:07.0578501Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-12-04T08:56:07.0578592Z layer_input = self.dense(hidden_states) 2025-12-04T08:56:07.0578596Z 2025-12-04T08:56:07.0578718Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0578961Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0579041Z res = mod(**inputs) 2025-12-04T08:56:07.0579358Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0579436Z outputs = self.mobilebert( 2025-12-04T08:56:07.0579752Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0579831Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0580142Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0580230Z layer_outputs = layer_module( 2025-12-04T08:56:07.0580542Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T08:56:07.0580727Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T08:56:07.0581045Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-12-04T08:56:07.0581167Z shared_attention_input = self.attention(hidden_states) 2025-12-04T08:56:07.0581486Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-12-04T08:56:07.0581583Z layer_input = self.LayerNorm(layer_input) 2025-12-04T08:56:07.0581906Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:07.0582005Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:07.0582009Z 2025-12-04T08:56:07.0582122Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0582355Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0582428Z res = mod(**inputs) 2025-12-04T08:56:07.0582740Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0582827Z outputs = self.mobilebert( 2025-12-04T08:56:07.0583136Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0583224Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0583570Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0583651Z layer_outputs = layer_module( 2025-12-04T08:56:07.0583967Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T08:56:07.0584064Z self_attention_outputs = self.attention( 2025-12-04T08:56:07.0584401Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T08:56:07.0584508Z self_outputs = self.self( 2025-12-04T08:56:07.0584827Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-12-04T08:56:07.0584914Z self.query(query_tensor) 2025-12-04T08:56:07.0584918Z 2025-12-04T08:56:07.0585032Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0585253Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0585332Z res = mod(**inputs) 2025-12-04T08:56:07.0585649Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0585736Z outputs = self.mobilebert( 2025-12-04T08:56:07.0586075Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0586155Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0586471Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0586549Z layer_outputs = layer_module( 2025-12-04T08:56:07.0586866Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T08:56:07.0586961Z self_attention_outputs = self.attention( 2025-12-04T08:56:07.0587280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T08:56:07.0587366Z self_outputs = self.self( 2025-12-04T08:56:07.0587674Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-12-04T08:56:07.0587750Z self.key(key_tensor) 2025-12-04T08:56:07.0587764Z 2025-12-04T08:56:07.0587855Z cudagraph partition due to non gpu ops 2025-12-04T08:56:07.0587946Z cudagraph partition due to non gpu ops 2025-12-04T08:56:07.0588067Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0588287Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0588357Z res = mod(**inputs) 2025-12-04T08:56:07.0588681Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0588759Z outputs = self.mobilebert( 2025-12-04T08:56:07.0589068Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0589156Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0589468Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0589553Z layer_outputs = layer_module( 2025-12-04T08:56:07.0589864Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T08:56:07.0589955Z self_attention_outputs = self.attention( 2025-12-04T08:56:07.0590288Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-12-04T08:56:07.0590440Z attention_output = self.output(self_outputs[0], layer_input) 2025-12-04T08:56:07.0590762Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-12-04T08:56:07.0590855Z layer_outputs = self.dense(hidden_states) 2025-12-04T08:56:07.0590859Z 2025-12-04T08:56:07.0590976Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0591227Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0591329Z res = mod(**inputs) 2025-12-04T08:56:07.0591642Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0591718Z outputs = self.mobilebert( 2025-12-04T08:56:07.0592029Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0592114Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0592424Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0592501Z layer_outputs = layer_module( 2025-12-04T08:56:07.0592812Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T08:56:07.0592922Z self_attention_outputs = self.attention( 2025-12-04T08:56:07.0593229Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-12-04T08:56:07.0593360Z attention_output = self.output(self_outputs[0], layer_input) 2025-12-04T08:56:07.0593661Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-12-04T08:56:07.0593803Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T08:56:07.0594106Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:07.0594209Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:07.0594212Z 2025-12-04T08:56:07.0594322Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0594533Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0594615Z res = mod(**inputs) 2025-12-04T08:56:07.0594919Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0594994Z outputs = self.mobilebert( 2025-12-04T08:56:07.0595301Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0595377Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0595687Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0595764Z layer_outputs = layer_module( 2025-12-04T08:56:07.0596069Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.0596180Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.0596485Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T08:56:07.0596614Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T08:56:07.0596918Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T08:56:07.0597008Z hidden_states = self.dense(hidden_states) 2025-12-04T08:56:07.0597011Z 2025-12-04T08:56:07.0597128Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0597358Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0597430Z res = mod(**inputs) 2025-12-04T08:56:07.0597740Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0597819Z outputs = self.mobilebert( 2025-12-04T08:56:07.0598238Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0598343Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0598657Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0598746Z layer_outputs = layer_module( 2025-12-04T08:56:07.0599059Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.0599175Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.0599488Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T08:56:07.0599613Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T08:56:07.0599959Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T08:56:07.0600088Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:56:07.0600092Z 2025-12-04T08:56:07.0600214Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0600435Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0600506Z res = mod(**inputs) 2025-12-04T08:56:07.0600839Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0600915Z outputs = self.mobilebert( 2025-12-04T08:56:07.0601217Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0601306Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0601612Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0601696Z layer_outputs = layer_module( 2025-12-04T08:56:07.0601999Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.0602100Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.0602413Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T08:56:07.0602551Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T08:56:07.0602861Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T08:56:07.0602951Z layer_outputs = self.dense(hidden_states) 2025-12-04T08:56:07.0602956Z 2025-12-04T08:56:07.0603066Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0603292Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0603363Z res = mod(**inputs) 2025-12-04T08:56:07.0603679Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0603763Z outputs = self.mobilebert( 2025-12-04T08:56:07.0604066Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0604170Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0604471Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0604547Z layer_outputs = layer_module( 2025-12-04T08:56:07.0604857Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.0604978Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.0605303Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T08:56:07.0605439Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T08:56:07.0605742Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T08:56:07.0605884Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T08:56:07.0606188Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:07.0606286Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:07.0606297Z 2025-12-04T08:56:07.0606425Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0606640Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0606718Z res = mod(**inputs) 2025-12-04T08:56:07.0607022Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0607098Z outputs = self.mobilebert( 2025-12-04T08:56:07.0607415Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0607492Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0607803Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0607880Z layer_outputs = layer_module( 2025-12-04T08:56:07.0608185Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.0608297Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.0608606Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T08:56:07.0608729Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T08:56:07.0609042Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T08:56:07.0609133Z hidden_states = self.dense(hidden_states) 2025-12-04T08:56:07.0609136Z 2025-12-04T08:56:07.0609255Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0609470Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0609539Z res = mod(**inputs) 2025-12-04T08:56:07.0609865Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0609941Z outputs = self.mobilebert( 2025-12-04T08:56:07.0610254Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0610332Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0610634Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0610718Z layer_outputs = layer_module( 2025-12-04T08:56:07.0611045Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.0611153Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.0611452Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T08:56:07.0611573Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T08:56:07.0611906Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T08:56:07.0612045Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:56:07.0612049Z 2025-12-04T08:56:07.0612158Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0612376Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0612443Z res = mod(**inputs) 2025-12-04T08:56:07.0612753Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0612829Z outputs = self.mobilebert( 2025-12-04T08:56:07.0613130Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0613234Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0613546Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0613630Z layer_outputs = layer_module( 2025-12-04T08:56:07.0613934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.0614033Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.0614341Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T08:56:07.0614476Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T08:56:07.0614780Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T08:56:07.0614878Z layer_outputs = self.dense(hidden_states) 2025-12-04T08:56:07.0614883Z 2025-12-04T08:56:07.0614994Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0615218Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0615288Z res = mod(**inputs) 2025-12-04T08:56:07.0615596Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0615679Z outputs = self.mobilebert( 2025-12-04T08:56:07.0615997Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0616082Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0616385Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0616462Z layer_outputs = layer_module( 2025-12-04T08:56:07.0616770Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.0616874Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.0617181Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T08:56:07.0617321Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T08:56:07.0617625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T08:56:07.0617789Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T08:56:07.0618088Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:07.0618186Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:07.0618192Z 2025-12-04T08:56:07.0618311Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0618539Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0618634Z res = mod(**inputs) 2025-12-04T08:56:07.0618935Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0619011Z outputs = self.mobilebert( 2025-12-04T08:56:07.0619320Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0619396Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0619704Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0619780Z layer_outputs = layer_module( 2025-12-04T08:56:07.0620081Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.0620213Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.0620516Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T08:56:07.0620634Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T08:56:07.0621095Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T08:56:07.0621194Z hidden_states = self.dense(hidden_states) 2025-12-04T08:56:07.0621198Z 2025-12-04T08:56:07.0621324Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0621544Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0621615Z res = mod(**inputs) 2025-12-04T08:56:07.0621931Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0622014Z outputs = self.mobilebert( 2025-12-04T08:56:07.0622326Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0622406Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0622707Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0622793Z layer_outputs = layer_module( 2025-12-04T08:56:07.0623098Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.0623197Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.0623503Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T08:56:07.0623626Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T08:56:07.0623949Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T08:56:07.0624070Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:56:07.0624073Z 2025-12-04T08:56:07.0624182Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0624403Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0624472Z res = mod(**inputs) 2025-12-04T08:56:07.0624842Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0624920Z outputs = self.mobilebert( 2025-12-04T08:56:07.0625219Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0625309Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0625634Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0625737Z layer_outputs = layer_module( 2025-12-04T08:56:07.0626053Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.0626154Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.0626479Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T08:56:07.0626615Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T08:56:07.0626920Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T08:56:07.0627047Z layer_outputs = self.dense(hidden_states) 2025-12-04T08:56:07.0627050Z 2025-12-04T08:56:07.0627163Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0627387Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0627457Z res = mod(**inputs) 2025-12-04T08:56:07.0627761Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0627848Z outputs = self.mobilebert( 2025-12-04T08:56:07.0628166Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0628250Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0628552Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0628627Z layer_outputs = layer_module( 2025-12-04T08:56:07.0628939Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.0629042Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.0629344Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T08:56:07.0629486Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T08:56:07.0629802Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T08:56:07.0629944Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T08:56:07.0630249Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:07.0630347Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:07.0630353Z 2025-12-04T08:56:07.0630470Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0630686Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0630765Z res = mod(**inputs) 2025-12-04T08:56:07.0631068Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0631142Z outputs = self.mobilebert( 2025-12-04T08:56:07.0631453Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0631547Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0631849Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0631930Z layer_outputs = layer_module( 2025-12-04T08:56:07.0632232Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-12-04T08:56:07.0632404Z intermediate_output = self.intermediate(attention_output) 2025-12-04T08:56:07.0632709Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T08:56:07.0632799Z hidden_states = self.dense(hidden_states) 2025-12-04T08:56:07.0632803Z 2025-12-04T08:56:07.0632919Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0633128Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0633205Z res = mod(**inputs) 2025-12-04T08:56:07.0633506Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0633582Z outputs = self.mobilebert( 2025-12-04T08:56:07.0633894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0633992Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0634313Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0634400Z layer_outputs = layer_module( 2025-12-04T08:56:07.0634712Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-12-04T08:56:07.0634855Z intermediate_output = self.intermediate(attention_output) 2025-12-04T08:56:07.0635181Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T08:56:07.0635303Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:56:07.0635307Z 2025-12-04T08:56:07.0635429Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0635653Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0635733Z res = mod(**inputs) 2025-12-04T08:56:07.0636055Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0636132Z outputs = self.mobilebert( 2025-12-04T08:56:07.0636451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0636531Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0636863Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0636940Z layer_outputs = layer_module( 2025-12-04T08:56:07.0637261Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T08:56:07.0637448Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T08:56:07.0637757Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-12-04T08:56:07.0637863Z layer_output = self.dense(intermediate_states) 2025-12-04T08:56:07.0637873Z 2025-12-04T08:56:07.0637987Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0638276Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0638358Z res = mod(**inputs) 2025-12-04T08:56:07.0638715Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0638794Z outputs = self.mobilebert( 2025-12-04T08:56:07.0639123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0639206Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0639543Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0639639Z layer_outputs = layer_module( 2025-12-04T08:56:07.0639950Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T08:56:07.0640130Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T08:56:07.0640518Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-12-04T08:56:07.0640653Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-12-04T08:56:07.0640969Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:07.0641089Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:07.0641096Z 2025-12-04T08:56:07.0641215Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0641435Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0641503Z res = mod(**inputs) 2025-12-04T08:56:07.0641832Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0641908Z outputs = self.mobilebert( 2025-12-04T08:56:07.0642240Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0642316Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0642631Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0642716Z layer_outputs = layer_module( 2025-12-04T08:56:07.0643033Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T08:56:07.0643214Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T08:56:07.0643527Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-12-04T08:56:07.0643664Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-12-04T08:56:07.0643984Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-12-04T08:56:07.0644074Z layer_outputs = self.dense(hidden_states) 2025-12-04T08:56:07.0644078Z 2025-12-04T08:56:07.0644188Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0644410Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0644481Z res = mod(**inputs) 2025-12-04T08:56:07.0644806Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0644884Z outputs = self.mobilebert( 2025-12-04T08:56:07.0645196Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0645279Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0645612Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0645696Z layer_outputs = layer_module( 2025-12-04T08:56:07.0646006Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T08:56:07.0646179Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T08:56:07.0646508Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-12-04T08:56:07.0646661Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-12-04T08:56:07.0646981Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-12-04T08:56:07.0647119Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T08:56:07.0647433Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:07.0647540Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:07.0647543Z 2025-12-04T08:56:07.0647654Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0647869Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0647959Z res = mod(**inputs) 2025-12-04T08:56:07.0648281Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0648363Z outputs = self.mobilebert( 2025-12-04T08:56:07.0648695Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0648771Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0649094Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0649170Z layer_outputs = layer_module( 2025-12-04T08:56:07.0649491Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T08:56:07.0649669Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T08:56:07.0649989Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-12-04T08:56:07.0650117Z shared_attention_input = self.attention(hidden_states) 2025-12-04T08:56:07.0650435Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-12-04T08:56:07.0650525Z layer_input = self.dense(hidden_states) 2025-12-04T08:56:07.0650533Z 2025-12-04T08:56:07.0650644Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0650868Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0650942Z res = mod(**inputs) 2025-12-04T08:56:07.0651258Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0651337Z outputs = self.mobilebert( 2025-12-04T08:56:07.0651664Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0651743Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0652066Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0652141Z layer_outputs = layer_module( 2025-12-04T08:56:07.0652470Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T08:56:07.0652589Z self_attention_outputs = self.attention( 2025-12-04T08:56:07.0652901Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T08:56:07.0652978Z self_outputs = self.self( 2025-12-04T08:56:07.0653299Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-12-04T08:56:07.0653414Z self.value(value_tensor) 2025-12-04T08:56:07.0653418Z 2025-12-04T08:56:07.0653537Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0653756Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0653824Z res = mod(**inputs) 2025-12-04T08:56:07.0654152Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0654225Z outputs = self.mobilebert( 2025-12-04T08:56:07.0654539Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0654615Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0654917Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0655019Z layer_outputs = layer_module( 2025-12-04T08:56:07.0655325Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T08:56:07.0655497Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T08:56:07.0655800Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-12-04T08:56:07.0655917Z bottlenecked_hidden_states = self.input(hidden_states) 2025-12-04T08:56:07.0656221Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-12-04T08:56:07.0656306Z layer_input = self.dense(hidden_states) 2025-12-04T08:56:07.0656310Z 2025-12-04T08:56:07.0656417Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0656633Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0656702Z res = mod(**inputs) 2025-12-04T08:56:07.0657010Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0657087Z outputs = self.mobilebert( 2025-12-04T08:56:07.0657386Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0657467Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0657770Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0657844Z layer_outputs = layer_module( 2025-12-04T08:56:07.0658149Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T08:56:07.0658322Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T08:56:07.0658630Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-12-04T08:56:07.0658743Z shared_attention_input = self.attention(hidden_states) 2025-12-04T08:56:07.0659052Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-12-04T08:56:07.0659152Z layer_input = self.LayerNorm(layer_input) 2025-12-04T08:56:07.0659470Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:07.0659571Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:07.0659575Z 2025-12-04T08:56:07.0659682Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0659894Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0659986Z res = mod(**inputs) 2025-12-04T08:56:07.0660309Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0660390Z outputs = self.mobilebert( 2025-12-04T08:56:07.0660703Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0660779Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0661096Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0661175Z layer_outputs = layer_module( 2025-12-04T08:56:07.0661484Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T08:56:07.0661613Z self_attention_outputs = self.attention( 2025-12-04T08:56:07.0661925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T08:56:07.0662009Z self_outputs = self.self( 2025-12-04T08:56:07.0662323Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-12-04T08:56:07.0662399Z self.query(query_tensor) 2025-12-04T08:56:07.0662403Z 2025-12-04T08:56:07.0662521Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0662739Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0662813Z res = mod(**inputs) 2025-12-04T08:56:07.0663131Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0663206Z outputs = self.mobilebert( 2025-12-04T08:56:07.0663546Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0663622Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0663921Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0664000Z layer_outputs = layer_module( 2025-12-04T08:56:07.0664297Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T08:56:07.0664392Z self_attention_outputs = self.attention( 2025-12-04T08:56:07.0664699Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T08:56:07.0664777Z self_outputs = self.self( 2025-12-04T08:56:07.0665089Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-12-04T08:56:07.0665164Z self.key(key_tensor) 2025-12-04T08:56:07.0665171Z 2025-12-04T08:56:07.0665267Z cudagraph partition due to non gpu ops 2025-12-04T08:56:07.0665353Z cudagraph partition due to non gpu ops 2025-12-04T08:56:07.0665464Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0665684Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0665754Z res = mod(**inputs) 2025-12-04T08:56:07.0666095Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0666181Z outputs = self.mobilebert( 2025-12-04T08:56:07.0666501Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0666582Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0666902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0666995Z layer_outputs = layer_module( 2025-12-04T08:56:07.0667296Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T08:56:07.0667385Z self_attention_outputs = self.attention( 2025-12-04T08:56:07.0667687Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-12-04T08:56:07.0667828Z attention_output = self.output(self_outputs[0], layer_input) 2025-12-04T08:56:07.0668136Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-12-04T08:56:07.0668230Z layer_outputs = self.dense(hidden_states) 2025-12-04T08:56:07.0668234Z 2025-12-04T08:56:07.0668365Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0668585Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0668667Z res = mod(**inputs) 2025-12-04T08:56:07.0668977Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0669060Z outputs = self.mobilebert( 2025-12-04T08:56:07.0669396Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0669474Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0669801Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0669879Z layer_outputs = layer_module( 2025-12-04T08:56:07.0670189Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T08:56:07.0670286Z self_attention_outputs = self.attention( 2025-12-04T08:56:07.0670599Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-12-04T08:56:07.0670739Z attention_output = self.output(self_outputs[0], layer_input) 2025-12-04T08:56:07.0671057Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-12-04T08:56:07.0671191Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T08:56:07.0671518Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:07.0671618Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:07.0671622Z 2025-12-04T08:56:07.0671738Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0671957Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0672029Z res = mod(**inputs) 2025-12-04T08:56:07.0672347Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0672425Z outputs = self.mobilebert( 2025-12-04T08:56:07.0672750Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0672828Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0673163Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0673245Z layer_outputs = layer_module( 2025-12-04T08:56:07.0673555Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.0673659Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.0673993Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T08:56:07.0674135Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T08:56:07.0674456Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T08:56:07.0674548Z hidden_states = self.dense(hidden_states) 2025-12-04T08:56:07.0674551Z 2025-12-04T08:56:07.0674662Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0674888Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0674960Z res = mod(**inputs) 2025-12-04T08:56:07.0675276Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0675372Z outputs = self.mobilebert( 2025-12-04T08:56:07.0675683Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0675767Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0676072Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0676148Z layer_outputs = layer_module( 2025-12-04T08:56:07.0676459Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.0676565Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.0676878Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T08:56:07.0677000Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T08:56:07.0677310Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T08:56:07.0677440Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:56:07.0677444Z 2025-12-04T08:56:07.0677555Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0677782Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0677849Z res = mod(**inputs) 2025-12-04T08:56:07.0678253Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0678344Z outputs = self.mobilebert( 2025-12-04T08:56:07.0678653Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0678731Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0679051Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0679129Z layer_outputs = layer_module( 2025-12-04T08:56:07.0679443Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.0679543Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.0679857Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T08:56:07.0680019Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T08:56:07.0680328Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T08:56:07.0680423Z layer_outputs = self.dense(hidden_states) 2025-12-04T08:56:07.0680429Z 2025-12-04T08:56:07.0680540Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0680777Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0680869Z res = mod(**inputs) 2025-12-04T08:56:07.0681183Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0681264Z outputs = self.mobilebert( 2025-12-04T08:56:07.0681573Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0681652Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0681967Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0682043Z layer_outputs = layer_module( 2025-12-04T08:56:07.0682352Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.0682481Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.0682795Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T08:56:07.0682935Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T08:56:07.0683245Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T08:56:07.0683378Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T08:56:07.0683697Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:07.0683795Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:07.0683798Z 2025-12-04T08:56:07.0683913Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0684133Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0684203Z res = mod(**inputs) 2025-12-04T08:56:07.0684520Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0684596Z outputs = self.mobilebert( 2025-12-04T08:56:07.0684906Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0684987Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0685299Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0685380Z layer_outputs = layer_module( 2025-12-04T08:56:07.0685690Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.0685791Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.0686112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T08:56:07.0686238Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T08:56:07.0686555Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T08:56:07.0686647Z hidden_states = self.dense(hidden_states) 2025-12-04T08:56:07.0686651Z 2025-12-04T08:56:07.0686778Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0687008Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0687076Z res = mod(**inputs) 2025-12-04T08:56:07.0687398Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0687475Z outputs = self.mobilebert( 2025-12-04T08:56:07.0687802Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0687911Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0688222Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0688299Z layer_outputs = layer_module( 2025-12-04T08:56:07.0688622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.0688726Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.0689046Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T08:56:07.0689197Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T08:56:07.0689509Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T08:56:07.0689645Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:56:07.0689649Z 2025-12-04T08:56:07.0689760Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0689985Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0690056Z res = mod(**inputs) 2025-12-04T08:56:07.0690377Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0690462Z outputs = self.mobilebert( 2025-12-04T08:56:07.0690774Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0690854Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0691174Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0691254Z layer_outputs = layer_module( 2025-12-04T08:56:07.0691570Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.0691673Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.0691994Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T08:56:07.0692141Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T08:56:07.0692464Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T08:56:07.0692570Z layer_outputs = self.dense(hidden_states) 2025-12-04T08:56:07.0692575Z 2025-12-04T08:56:07.0692688Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0692906Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0692985Z res = mod(**inputs) 2025-12-04T08:56:07.0693296Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0693374Z outputs = self.mobilebert( 2025-12-04T08:56:07.0693696Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0693798Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0694120Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0694197Z layer_outputs = layer_module( 2025-12-04T08:56:07.0694509Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.0694654Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.0694973Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T08:56:07.0695119Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T08:56:07.0695434Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T08:56:07.0695571Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T08:56:07.0695892Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:07.0695991Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:07.0696016Z 2025-12-04T08:56:07.0696138Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0696361Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0696433Z res = mod(**inputs) 2025-12-04T08:56:07.0696754Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0696832Z outputs = self.mobilebert( 2025-12-04T08:56:07.0697141Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0697229Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0697541Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0697627Z layer_outputs = layer_module( 2025-12-04T08:56:07.0697939Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.0698046Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.0698369Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T08:56:07.0698491Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T08:56:07.0698814Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T08:56:07.0698906Z hidden_states = self.dense(hidden_states) 2025-12-04T08:56:07.0698910Z 2025-12-04T08:56:07.0699025Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0699250Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0699320Z res = mod(**inputs) 2025-12-04T08:56:07.0699633Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0699721Z outputs = self.mobilebert( 2025-12-04T08:56:07.0700037Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0700122Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0700433Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0700511Z layer_outputs = layer_module( 2025-12-04T08:56:07.0700856Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.0700959Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.0701282Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T08:56:07.0701407Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T08:56:07.0701741Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T08:56:07.0701892Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:56:07.0701896Z 2025-12-04T08:56:07.0702009Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0702228Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0702306Z res = mod(**inputs) 2025-12-04T08:56:07.0702621Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0702709Z outputs = self.mobilebert( 2025-12-04T08:56:07.0703024Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0703127Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0703450Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0703530Z layer_outputs = layer_module( 2025-12-04T08:56:07.0703851Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.0703953Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.0704267Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T08:56:07.0704412Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T08:56:07.0704725Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T08:56:07.0704827Z layer_outputs = self.dense(hidden_states) 2025-12-04T08:56:07.0704831Z 2025-12-04T08:56:07.0704946Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0705168Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0705244Z res = mod(**inputs) 2025-12-04T08:56:07.0705560Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0705637Z outputs = self.mobilebert( 2025-12-04T08:56:07.0705961Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0706040Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0706360Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0706438Z layer_outputs = layer_module( 2025-12-04T08:56:07.0706755Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.0706864Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.0707178Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T08:56:07.0707321Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T08:56:07.0707636Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T08:56:07.0707797Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T08:56:07.0708115Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:07.0708217Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:07.0708223Z 2025-12-04T08:56:07.0708338Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0708583Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0708671Z res = mod(**inputs) 2025-12-04T08:56:07.0708988Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0709066Z outputs = self.mobilebert( 2025-12-04T08:56:07.0709385Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0709475Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0709784Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0709870Z layer_outputs = layer_module( 2025-12-04T08:56:07.0710184Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-12-04T08:56:07.0710344Z intermediate_output = self.intermediate(attention_output) 2025-12-04T08:56:07.0710670Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T08:56:07.0710776Z hidden_states = self.dense(hidden_states) 2025-12-04T08:56:07.0710779Z 2025-12-04T08:56:07.0710899Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0711119Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0711191Z res = mod(**inputs) 2025-12-04T08:56:07.0711513Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0711589Z outputs = self.mobilebert( 2025-12-04T08:56:07.0711916Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0712007Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0712323Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0712410Z layer_outputs = layer_module( 2025-12-04T08:56:07.0712732Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-12-04T08:56:07.0712865Z intermediate_output = self.intermediate(attention_output) 2025-12-04T08:56:07.0713193Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T08:56:07.0713315Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:56:07.0713319Z 2025-12-04T08:56:07.0713438Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0713662Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0713737Z res = mod(**inputs) 2025-12-04T08:56:07.0714058Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0714136Z outputs = self.mobilebert( 2025-12-04T08:56:07.0714448Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0714537Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0714875Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0714962Z layer_outputs = layer_module( 2025-12-04T08:56:07.0715273Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T08:56:07.0715449Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T08:56:07.0715880Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-12-04T08:56:07.0715987Z layer_output = self.dense(intermediate_states) 2025-12-04T08:56:07.0715991Z 2025-12-04T08:56:07.0716110Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0716328Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0716396Z res = mod(**inputs) 2025-12-04T08:56:07.0716722Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0716800Z outputs = self.mobilebert( 2025-12-04T08:56:07.0717112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0717220Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0717537Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0717620Z layer_outputs = layer_module( 2025-12-04T08:56:07.0717933Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T08:56:07.0718178Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T08:56:07.0718510Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-12-04T08:56:07.0718645Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-12-04T08:56:07.0718962Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:07.0719062Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:07.0719066Z 2025-12-04T08:56:07.0719184Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0719408Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0719479Z res = mod(**inputs) 2025-12-04T08:56:07.0719793Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0719872Z outputs = self.mobilebert( 2025-12-04T08:56:07.0720183Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0720269Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0720581Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0720661Z layer_outputs = layer_module( 2025-12-04T08:56:07.0721200Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T08:56:07.0721384Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T08:56:07.0721697Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-12-04T08:56:07.0721830Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-12-04T08:56:07.0722198Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-12-04T08:56:07.0722302Z layer_outputs = self.dense(hidden_states) 2025-12-04T08:56:07.0722306Z 2025-12-04T08:56:07.0722418Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0722646Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0722718Z res = mod(**inputs) 2025-12-04T08:56:07.0723053Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0723162Z outputs = self.mobilebert( 2025-12-04T08:56:07.0723477Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0723554Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0723879Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0723960Z layer_outputs = layer_module( 2025-12-04T08:56:07.0724276Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T08:56:07.0724446Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T08:56:07.0724783Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-12-04T08:56:07.0724927Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-12-04T08:56:07.0725237Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-12-04T08:56:07.0725374Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T08:56:07.0725690Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:07.0725793Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:07.0725797Z 2025-12-04T08:56:07.0725919Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0726135Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0726213Z res = mod(**inputs) 2025-12-04T08:56:07.0726529Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0726611Z outputs = self.mobilebert( 2025-12-04T08:56:07.0726936Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0727015Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0727340Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0727427Z layer_outputs = layer_module( 2025-12-04T08:56:07.0727741Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T08:56:07.0727930Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T08:56:07.0728251Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-12-04T08:56:07.0728374Z shared_attention_input = self.attention(hidden_states) 2025-12-04T08:56:07.0728697Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-12-04T08:56:07.0728789Z layer_input = self.dense(hidden_states) 2025-12-04T08:56:07.0728792Z 2025-12-04T08:56:07.0728913Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0729159Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0729229Z res = mod(**inputs) 2025-12-04T08:56:07.0729565Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0729644Z outputs = self.mobilebert( 2025-12-04T08:56:07.0729974Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0730078Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0730389Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0730473Z layer_outputs = layer_module( 2025-12-04T08:56:07.0730795Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T08:56:07.0730893Z self_attention_outputs = self.attention( 2025-12-04T08:56:07.0731212Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T08:56:07.0731292Z self_outputs = self.self( 2025-12-04T08:56:07.0731619Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-12-04T08:56:07.0731730Z self.value(value_tensor) 2025-12-04T08:56:07.0731735Z 2025-12-04T08:56:07.0731845Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0732064Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0732133Z res = mod(**inputs) 2025-12-04T08:56:07.0732459Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0732541Z outputs = self.mobilebert( 2025-12-04T08:56:07.0732857Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0732943Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0733243Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0733321Z layer_outputs = layer_module( 2025-12-04T08:56:07.0733634Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T08:56:07.0733807Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T08:56:07.0734126Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-12-04T08:56:07.0734245Z bottlenecked_hidden_states = self.input(hidden_states) 2025-12-04T08:56:07.0734559Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-12-04T08:56:07.0734656Z layer_input = self.dense(hidden_states) 2025-12-04T08:56:07.0734660Z 2025-12-04T08:56:07.0734770Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0734991Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0735063Z res = mod(**inputs) 2025-12-04T08:56:07.0735365Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0735447Z outputs = self.mobilebert( 2025-12-04T08:56:07.0735756Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0735832Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0736158Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0736237Z layer_outputs = layer_module( 2025-12-04T08:56:07.0736547Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T08:56:07.0736724Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T08:56:07.0737071Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-12-04T08:56:07.0737201Z shared_attention_input = self.attention(hidden_states) 2025-12-04T08:56:07.0737512Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-12-04T08:56:07.0737614Z layer_input = self.LayerNorm(layer_input) 2025-12-04T08:56:07.0737923Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:07.0738022Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:07.0738026Z 2025-12-04T08:56:07.0738147Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0738385Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0738457Z res = mod(**inputs) 2025-12-04T08:56:07.0738783Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0738861Z outputs = self.mobilebert( 2025-12-04T08:56:07.0739182Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0739260Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0739584Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0739670Z layer_outputs = layer_module( 2025-12-04T08:56:07.0739984Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T08:56:07.0740090Z self_attention_outputs = self.attention( 2025-12-04T08:56:07.0740404Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T08:56:07.0740486Z self_outputs = self.self( 2025-12-04T08:56:07.0740806Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-12-04T08:56:07.0740883Z self.query(query_tensor) 2025-12-04T08:56:07.0740887Z 2025-12-04T08:56:07.0740999Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0741228Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0741298Z res = mod(**inputs) 2025-12-04T08:56:07.0741620Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0741696Z outputs = self.mobilebert( 2025-12-04T08:56:07.0742002Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0742089Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0742391Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0742474Z layer_outputs = layer_module( 2025-12-04T08:56:07.0742778Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T08:56:07.0742869Z self_attention_outputs = self.attention( 2025-12-04T08:56:07.0743211Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T08:56:07.0743290Z self_outputs = self.self( 2025-12-04T08:56:07.0743596Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-12-04T08:56:07.0743678Z self.key(key_tensor) 2025-12-04T08:56:07.0743700Z 2025-12-04T08:56:07.0743808Z cudagraph partition due to non gpu ops 2025-12-04T08:56:07.0743903Z cudagraph partition due to non gpu ops 2025-12-04T08:56:07.0744015Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0744228Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0744306Z res = mod(**inputs) 2025-12-04T08:56:07.0744611Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0744687Z outputs = self.mobilebert( 2025-12-04T08:56:07.0744998Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0745076Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0745462Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0745541Z layer_outputs = layer_module( 2025-12-04T08:56:07.0745841Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T08:56:07.0745940Z self_attention_outputs = self.attention( 2025-12-04T08:56:07.0746243Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-12-04T08:56:07.0746385Z attention_output = self.output(self_outputs[0], layer_input) 2025-12-04T08:56:07.0746687Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-12-04T08:56:07.0746778Z layer_outputs = self.dense(hidden_states) 2025-12-04T08:56:07.0746781Z 2025-12-04T08:56:07.0746903Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0747117Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0747189Z res = mod(**inputs) 2025-12-04T08:56:07.0747503Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0747577Z outputs = self.mobilebert( 2025-12-04T08:56:07.0747894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0747974Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0748297Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0748385Z layer_outputs = layer_module( 2025-12-04T08:56:07.0748702Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T08:56:07.0748800Z self_attention_outputs = self.attention( 2025-12-04T08:56:07.0749106Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-12-04T08:56:07.0749236Z attention_output = self.output(self_outputs[0], layer_input) 2025-12-04T08:56:07.0749544Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-12-04T08:56:07.0749680Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T08:56:07.0750035Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:07.0750137Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:07.0750141Z 2025-12-04T08:56:07.0750251Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0750473Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0750558Z res = mod(**inputs) 2025-12-04T08:56:07.0750878Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0750960Z outputs = self.mobilebert( 2025-12-04T08:56:07.0751272Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0751355Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0751669Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0751744Z layer_outputs = layer_module( 2025-12-04T08:56:07.0752053Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.0752172Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.0752486Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T08:56:07.0752609Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T08:56:07.0752911Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T08:56:07.0753009Z hidden_states = self.dense(hidden_states) 2025-12-04T08:56:07.0753013Z 2025-12-04T08:56:07.0753122Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0753337Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0753412Z res = mod(**inputs) 2025-12-04T08:56:07.0753716Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0753799Z outputs = self.mobilebert( 2025-12-04T08:56:07.0754106Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0754185Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0754495Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0754570Z layer_outputs = layer_module( 2025-12-04T08:56:07.0754890Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.0754993Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.0755297Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T08:56:07.0755423Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T08:56:07.0755728Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T08:56:07.0755850Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:56:07.0755860Z 2025-12-04T08:56:07.0755968Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0756179Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0756252Z res = mod(**inputs) 2025-12-04T08:56:07.0756575Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0756652Z outputs = self.mobilebert( 2025-12-04T08:56:07.0756962Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0757040Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0757365Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0757457Z layer_outputs = layer_module( 2025-12-04T08:56:07.0757761Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.0757870Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.0758357Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T08:56:07.0758510Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T08:56:07.0758818Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T08:56:07.0758913Z layer_outputs = self.dense(hidden_states) 2025-12-04T08:56:07.0758941Z 2025-12-04T08:56:07.0759061Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0759284Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0759358Z res = mod(**inputs) 2025-12-04T08:56:07.0759691Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0759769Z outputs = self.mobilebert( 2025-12-04T08:56:07.0760082Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0760161Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0760467Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0760554Z layer_outputs = layer_module( 2025-12-04T08:56:07.0760862Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.0760972Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.0761280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T08:56:07.0761415Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T08:56:07.0761728Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T08:56:07.0761861Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T08:56:07.0762172Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:07.0762277Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:07.0762281Z 2025-12-04T08:56:07.0762392Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0762616Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0762688Z res = mod(**inputs) 2025-12-04T08:56:07.0762991Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0763075Z outputs = self.mobilebert( 2025-12-04T08:56:07.0763390Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0763475Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0763800Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0763878Z layer_outputs = layer_module( 2025-12-04T08:56:07.0764189Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.0764290Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.0764607Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T08:56:07.0764751Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T08:56:07.0765054Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T08:56:07.0765151Z hidden_states = self.dense(hidden_states) 2025-12-04T08:56:07.0765154Z 2025-12-04T08:56:07.0765267Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0765480Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0765557Z res = mod(**inputs) 2025-12-04T08:56:07.0765862Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0765975Z outputs = self.mobilebert( 2025-12-04T08:56:07.0766280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0766359Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0766669Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0766745Z layer_outputs = layer_module( 2025-12-04T08:56:07.0767049Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.0767157Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.0767461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T08:56:07.0767590Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T08:56:07.0767893Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T08:56:07.0768015Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:56:07.0768018Z 2025-12-04T08:56:07.0768136Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0768350Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0768427Z res = mod(**inputs) 2025-12-04T08:56:07.0768734Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0768810Z outputs = self.mobilebert( 2025-12-04T08:56:07.0769122Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0769201Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0769514Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0769591Z layer_outputs = layer_module( 2025-12-04T08:56:07.0769893Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.0769999Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.0770315Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T08:56:07.0770469Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T08:56:07.0770778Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T08:56:07.0770869Z layer_outputs = self.dense(hidden_states) 2025-12-04T08:56:07.0770875Z 2025-12-04T08:56:07.0770992Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0771221Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0771310Z res = mod(**inputs) 2025-12-04T08:56:07.0771619Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0771693Z outputs = self.mobilebert( 2025-12-04T08:56:07.0772010Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0772089Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0772389Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0772472Z layer_outputs = layer_module( 2025-12-04T08:56:07.0772782Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.0772901Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.0773210Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T08:56:07.0773342Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T08:56:07.0773655Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T08:56:07.0773789Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T08:56:07.0774085Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:07.0774190Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:07.0774193Z 2025-12-04T08:56:07.0774306Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0774534Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0774605Z res = mod(**inputs) 2025-12-04T08:56:07.0774906Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0774990Z outputs = self.mobilebert( 2025-12-04T08:56:07.0775310Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0775396Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0775726Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0775803Z layer_outputs = layer_module( 2025-12-04T08:56:07.0776110Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.0776213Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.0779262Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T08:56:07.0779415Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T08:56:07.0779735Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T08:56:07.0779828Z hidden_states = self.dense(hidden_states) 2025-12-04T08:56:07.0779832Z 2025-12-04T08:56:07.0779958Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0780179Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0780257Z res = mod(**inputs) 2025-12-04T08:56:07.0780571Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0780675Z outputs = self.mobilebert( 2025-12-04T08:56:07.0781002Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0781122Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0781425Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0781508Z layer_outputs = layer_module( 2025-12-04T08:56:07.0781809Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.0781911Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.0782220Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T08:56:07.0782358Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T08:56:07.0782668Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T08:56:07.0782795Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:56:07.0782799Z 2025-12-04T08:56:07.0782914Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0783145Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0783217Z res = mod(**inputs) 2025-12-04T08:56:07.0783539Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0783618Z outputs = self.mobilebert( 2025-12-04T08:56:07.0783927Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0784017Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0784330Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0784410Z layer_outputs = layer_module( 2025-12-04T08:56:07.0784738Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.0784841Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.0785150Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T08:56:07.0785298Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T08:56:07.0785608Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T08:56:07.0785711Z layer_outputs = self.dense(hidden_states) 2025-12-04T08:56:07.0785715Z 2025-12-04T08:56:07.0785831Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0786057Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0786200Z res = mod(**inputs) 2025-12-04T08:56:07.0786513Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0786601Z outputs = self.mobilebert( 2025-12-04T08:56:07.0786909Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0786997Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0787305Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0787385Z layer_outputs = layer_module( 2025-12-04T08:56:07.0787723Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.0787841Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.0788153Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T08:56:07.0788298Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T08:56:07.0788608Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T08:56:07.0788749Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T08:56:07.0789075Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:07.0789175Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:07.0789203Z 2025-12-04T08:56:07.0789320Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0789543Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0789626Z res = mod(**inputs) 2025-12-04T08:56:07.0789938Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0790016Z outputs = self.mobilebert( 2025-12-04T08:56:07.0790336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0790415Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0790733Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0790811Z layer_outputs = layer_module( 2025-12-04T08:56:07.0791125Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-12-04T08:56:07.0791270Z intermediate_output = self.intermediate(attention_output) 2025-12-04T08:56:07.0791586Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T08:56:07.0791679Z hidden_states = self.dense(hidden_states) 2025-12-04T08:56:07.0791689Z 2025-12-04T08:56:07.0791802Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0792018Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0792095Z res = mod(**inputs) 2025-12-04T08:56:07.0792407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0792486Z outputs = self.mobilebert( 2025-12-04T08:56:07.0792806Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0792888Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0793248Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0793328Z layer_outputs = layer_module( 2025-12-04T08:56:07.0793640Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-12-04T08:56:07.0793777Z intermediate_output = self.intermediate(attention_output) 2025-12-04T08:56:07.0794090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T08:56:07.0794211Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:56:07.0794222Z 2025-12-04T08:56:07.0794335Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0794555Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0794676Z res = mod(**inputs) 2025-12-04T08:56:07.0794989Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0795066Z outputs = self.mobilebert( 2025-12-04T08:56:07.0795384Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0795463Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0795781Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0795858Z layer_outputs = layer_module( 2025-12-04T08:56:07.0796167Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T08:56:07.0796381Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T08:56:07.0796696Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-12-04T08:56:07.0796808Z layer_output = self.dense(intermediate_states) 2025-12-04T08:56:07.0796812Z 2025-12-04T08:56:07.0796926Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0797145Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0797224Z res = mod(**inputs) 2025-12-04T08:56:07.0797533Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0797611Z outputs = self.mobilebert( 2025-12-04T08:56:07.0797930Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0798012Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0798425Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0798514Z layer_outputs = layer_module( 2025-12-04T08:56:07.0798828Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T08:56:07.0799013Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T08:56:07.0799327Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-12-04T08:56:07.0799471Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-12-04T08:56:07.0799785Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:07.0799889Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:07.0799893Z 2025-12-04T08:56:07.0800021Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0800271Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0800343Z res = mod(**inputs) 2025-12-04T08:56:07.0800663Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0800742Z outputs = self.mobilebert( 2025-12-04T08:56:07.0801070Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0801151Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0801461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0801553Z layer_outputs = layer_module( 2025-12-04T08:56:07.0801881Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T08:56:07.0802091Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T08:56:07.0802405Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-12-04T08:56:07.0802538Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-12-04T08:56:07.0802848Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-12-04T08:56:07.0802940Z layer_outputs = self.dense(hidden_states) 2025-12-04T08:56:07.0802943Z 2025-12-04T08:56:07.0803060Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0803279Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0803367Z res = mod(**inputs) 2025-12-04T08:56:07.0803674Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0803753Z outputs = self.mobilebert( 2025-12-04T08:56:07.0804058Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0804142Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0804445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0804528Z layer_outputs = layer_module( 2025-12-04T08:56:07.0804826Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T08:56:07.0804993Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T08:56:07.0805304Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-12-04T08:56:07.0805439Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-12-04T08:56:07.0805750Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-12-04T08:56:07.0805882Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T08:56:07.0806183Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:07.0806288Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:07.0806292Z 2025-12-04T08:56:07.0806403Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0806629Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0806700Z res = mod(**inputs) 2025-12-04T08:56:07.0807004Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0807109Z outputs = self.mobilebert( 2025-12-04T08:56:07.0807410Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0807488Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0807797Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0807873Z layer_outputs = layer_module( 2025-12-04T08:56:07.0808181Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T08:56:07.0808353Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T08:56:07.0808673Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-12-04T08:56:07.0808821Z shared_attention_input = self.attention(hidden_states) 2025-12-04T08:56:07.0809125Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-12-04T08:56:07.0809223Z layer_input = self.dense(hidden_states) 2025-12-04T08:56:07.0809227Z 2025-12-04T08:56:07.0809336Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0809548Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0809622Z res = mod(**inputs) 2025-12-04T08:56:07.0809930Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0810025Z outputs = self.mobilebert( 2025-12-04T08:56:07.0810345Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0810423Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0810736Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0810813Z layer_outputs = layer_module( 2025-12-04T08:56:07.0811116Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T08:56:07.0811215Z self_attention_outputs = self.attention( 2025-12-04T08:56:07.0811526Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T08:56:07.0811609Z self_outputs = self.self( 2025-12-04T08:56:07.0811926Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-12-04T08:56:07.0812005Z self.value(value_tensor) 2025-12-04T08:56:07.0812010Z 2025-12-04T08:56:07.0812129Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0812344Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0812413Z res = mod(**inputs) 2025-12-04T08:56:07.0812733Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0812808Z outputs = self.mobilebert( 2025-12-04T08:56:07.0813125Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0813203Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0813516Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0813602Z layer_outputs = layer_module( 2025-12-04T08:56:07.0813908Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T08:56:07.0814108Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T08:56:07.0814414Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-12-04T08:56:07.0814533Z bottlenecked_hidden_states = self.input(hidden_states) 2025-12-04T08:56:07.0814847Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-12-04T08:56:07.0814934Z layer_input = self.dense(hidden_states) 2025-12-04T08:56:07.0814938Z 2025-12-04T08:56:07.0815049Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0815276Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0815361Z res = mod(**inputs) 2025-12-04T08:56:07.0815694Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0815769Z outputs = self.mobilebert( 2025-12-04T08:56:07.0816069Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0816155Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0816472Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0816556Z layer_outputs = layer_module( 2025-12-04T08:56:07.0816867Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T08:56:07.0817056Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T08:56:07.0817377Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-12-04T08:56:07.0817498Z shared_attention_input = self.attention(hidden_states) 2025-12-04T08:56:07.0817808Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-12-04T08:56:07.0817901Z layer_input = self.LayerNorm(layer_input) 2025-12-04T08:56:07.0818213Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:07.0818315Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:07.0818319Z 2025-12-04T08:56:07.0818429Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0818643Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0818718Z res = mod(**inputs) 2025-12-04T08:56:07.0819032Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0819120Z outputs = self.mobilebert( 2025-12-04T08:56:07.0819437Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0819516Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0819840Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0819915Z layer_outputs = layer_module( 2025-12-04T08:56:07.0820229Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T08:56:07.0820323Z self_attention_outputs = self.attention( 2025-12-04T08:56:07.0820629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T08:56:07.0820938Z self_outputs = self.self( 2025-12-04T08:56:07.0821301Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-12-04T08:56:07.0821382Z self.query(query_tensor) 2025-12-04T08:56:07.0821386Z 2025-12-04T08:56:07.0821513Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0821734Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0821814Z res = mod(**inputs) 2025-12-04T08:56:07.0822126Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0822203Z outputs = self.mobilebert( 2025-12-04T08:56:07.0822569Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0822673Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0823000Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0823081Z layer_outputs = layer_module( 2025-12-04T08:56:07.0823396Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T08:56:07.0823498Z self_attention_outputs = self.attention( 2025-12-04T08:56:07.0823813Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T08:56:07.0823890Z self_outputs = self.self( 2025-12-04T08:56:07.0824207Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-12-04T08:56:07.0824309Z self.key(key_tensor) 2025-12-04T08:56:07.0824315Z 2025-12-04T08:56:07.0824416Z cudagraph partition due to non gpu ops 2025-12-04T08:56:07.0824505Z cudagraph partition due to non gpu ops 2025-12-04T08:56:07.0824625Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0824856Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0824926Z res = mod(**inputs) 2025-12-04T08:56:07.0825246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0825330Z outputs = self.mobilebert( 2025-12-04T08:56:07.0825650Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0825738Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0826062Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0826141Z layer_outputs = layer_module( 2025-12-04T08:56:07.0826469Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T08:56:07.0826562Z self_attention_outputs = self.attention( 2025-12-04T08:56:07.0826888Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-12-04T08:56:07.0827025Z attention_output = self.output(self_outputs[0], layer_input) 2025-12-04T08:56:07.0827345Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-12-04T08:56:07.0827446Z layer_outputs = self.dense(hidden_states) 2025-12-04T08:56:07.0827450Z 2025-12-04T08:56:07.0827566Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0827788Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0827869Z res = mod(**inputs) 2025-12-04T08:56:07.0828211Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0828298Z outputs = self.mobilebert( 2025-12-04T08:56:07.0828610Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0828689Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0829007Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0829084Z layer_outputs = layer_module( 2025-12-04T08:56:07.0829403Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T08:56:07.0829498Z self_attention_outputs = self.attention( 2025-12-04T08:56:07.0829843Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-12-04T08:56:07.0829988Z attention_output = self.output(self_outputs[0], layer_input) 2025-12-04T08:56:07.0830305Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-12-04T08:56:07.0830444Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T08:56:07.0830765Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:07.0830866Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:07.0830870Z 2025-12-04T08:56:07.0830989Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0831227Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0831300Z res = mod(**inputs) 2025-12-04T08:56:07.0831622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0831699Z outputs = self.mobilebert( 2025-12-04T08:56:07.0832016Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0832095Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0832406Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0832493Z layer_outputs = layer_module( 2025-12-04T08:56:07.0832803Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.0832917Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.0833236Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T08:56:07.0833363Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T08:56:07.0833672Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T08:56:07.0833762Z hidden_states = self.dense(hidden_states) 2025-12-04T08:56:07.0833766Z 2025-12-04T08:56:07.0833874Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0834095Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0834164Z res = mod(**inputs) 2025-12-04T08:56:07.0834480Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0834558Z outputs = self.mobilebert( 2025-12-04T08:56:07.0834859Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0834947Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0835269Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0835354Z layer_outputs = layer_module( 2025-12-04T08:56:07.0835657Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.0835757Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.0836066Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T08:56:07.0836185Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T08:56:07.0836507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T08:56:07.0836661Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:56:07.0836665Z 2025-12-04T08:56:07.0836781Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0837009Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0837080Z res = mod(**inputs) 2025-12-04T08:56:07.0837393Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0837478Z outputs = self.mobilebert( 2025-12-04T08:56:07.0837790Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0837876Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0838268Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0838355Z layer_outputs = layer_module( 2025-12-04T08:56:07.0838677Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.0838782Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.0839093Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T08:56:07.0839241Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T08:56:07.0839552Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T08:56:07.0839653Z layer_outputs = self.dense(hidden_states) 2025-12-04T08:56:07.0839657Z 2025-12-04T08:56:07.0839773Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0840012Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0840095Z res = mod(**inputs) 2025-12-04T08:56:07.0840396Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0840481Z outputs = self.mobilebert( 2025-12-04T08:56:07.0840783Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0840863Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0841172Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0841250Z layer_outputs = layer_module( 2025-12-04T08:56:07.0841566Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.0841679Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.0842006Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T08:56:07.0842149Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T08:56:07.0842451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T08:56:07.0842582Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T08:56:07.0842892Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:07.0842989Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:07.0842993Z 2025-12-04T08:56:07.0843111Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0843345Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0843431Z res = mod(**inputs) 2025-12-04T08:56:07.0843747Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0843822Z outputs = self.mobilebert( 2025-12-04T08:56:07.0844141Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0844220Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0844528Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0844614Z layer_outputs = layer_module( 2025-12-04T08:56:07.0844925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.0845053Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.0845375Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T08:56:07.0845500Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T08:56:07.0845832Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T08:56:07.0845923Z hidden_states = self.dense(hidden_states) 2025-12-04T08:56:07.0845927Z 2025-12-04T08:56:07.0846035Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0846255Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0846325Z res = mod(**inputs) 2025-12-04T08:56:07.0846635Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0846712Z outputs = self.mobilebert( 2025-12-04T08:56:07.0847018Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0847109Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0847419Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0847496Z layer_outputs = layer_module( 2025-12-04T08:56:07.0847815Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.0847917Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.0848246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T08:56:07.0848365Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T08:56:07.0848668Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T08:56:07.0848798Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:56:07.0848833Z 2025-12-04T08:56:07.0848946Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0849170Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0849240Z res = mod(**inputs) 2025-12-04T08:56:07.0849544Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0849628Z outputs = self.mobilebert( 2025-12-04T08:56:07.0849932Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0850011Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0850338Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0850432Z layer_outputs = layer_module( 2025-12-04T08:56:07.0850745Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.0850845Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.0851149Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T08:56:07.0851291Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T08:56:07.0851594Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T08:56:07.0851690Z layer_outputs = self.dense(hidden_states) 2025-12-04T08:56:07.0851711Z 2025-12-04T08:56:07.0851822Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0852038Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0852116Z res = mod(**inputs) 2025-12-04T08:56:07.0852420Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0852503Z outputs = self.mobilebert( 2025-12-04T08:56:07.0852808Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0852885Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0853203Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0853278Z layer_outputs = layer_module( 2025-12-04T08:56:07.0853586Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.0853696Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.0854005Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T08:56:07.0854147Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T08:56:07.0854450Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T08:56:07.0854581Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T08:56:07.0854905Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:07.0855004Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:07.0855008Z 2025-12-04T08:56:07.0855130Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0855348Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0855420Z res = mod(**inputs) 2025-12-04T08:56:07.0855752Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0855829Z outputs = self.mobilebert( 2025-12-04T08:56:07.0856130Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0856217Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0856525Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0856609Z layer_outputs = layer_module( 2025-12-04T08:56:07.0856920Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.0857039Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.0857419Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T08:56:07.0857550Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T08:56:07.0857870Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T08:56:07.0857959Z hidden_states = self.dense(hidden_states) 2025-12-04T08:56:07.0857963Z 2025-12-04T08:56:07.0858072Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0858294Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0858363Z res = mod(**inputs) 2025-12-04T08:56:07.0858671Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0858766Z outputs = self.mobilebert( 2025-12-04T08:56:07.0859071Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0859156Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0859456Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0859531Z layer_outputs = layer_module( 2025-12-04T08:56:07.0859853Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.0859953Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.0860271Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T08:56:07.0860393Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T08:56:07.0860699Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T08:56:07.0860833Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:56:07.0860837Z 2025-12-04T08:56:07.0860947Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0861170Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0861240Z res = mod(**inputs) 2025-12-04T08:56:07.0861541Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0861624Z outputs = self.mobilebert( 2025-12-04T08:56:07.0861926Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0862004Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0862318Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0862396Z layer_outputs = layer_module( 2025-12-04T08:56:07.0862723Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.0862826Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.0863128Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T08:56:07.0863270Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T08:56:07.0863568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T08:56:07.0863664Z layer_outputs = self.dense(hidden_states) 2025-12-04T08:56:07.0863670Z 2025-12-04T08:56:07.0863793Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0864037Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0864116Z res = mod(**inputs) 2025-12-04T08:56:07.0864422Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0864498Z outputs = self.mobilebert( 2025-12-04T08:56:07.0864812Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0864889Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0865205Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0865280Z layer_outputs = layer_module( 2025-12-04T08:56:07.0865605Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.0865712Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.0866019Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T08:56:07.0866158Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T08:56:07.0866459Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T08:56:07.0866590Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T08:56:07.0866899Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:07.0866997Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:07.0867002Z 2025-12-04T08:56:07.0867119Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0867333Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0867403Z res = mod(**inputs) 2025-12-04T08:56:07.0867711Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0867785Z outputs = self.mobilebert( 2025-12-04T08:56:07.0868085Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0868168Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0868468Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0868550Z layer_outputs = layer_module( 2025-12-04T08:56:07.0868850Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-12-04T08:56:07.0868984Z intermediate_output = self.intermediate(attention_output) 2025-12-04T08:56:07.0869313Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T08:56:07.0869404Z hidden_states = self.dense(hidden_states) 2025-12-04T08:56:07.0869408Z 2025-12-04T08:56:07.0869525Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0869738Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0869808Z res = mod(**inputs) 2025-12-04T08:56:07.0870119Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0870194Z outputs = self.mobilebert( 2025-12-04T08:56:07.0870496Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0870598Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0870920Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0871004Z layer_outputs = layer_module( 2025-12-04T08:56:07.0871304Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-12-04T08:56:07.0871432Z intermediate_output = self.intermediate(attention_output) 2025-12-04T08:56:07.0871744Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T08:56:07.0871865Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:56:07.0871868Z 2025-12-04T08:56:07.0871987Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0872226Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0872307Z res = mod(**inputs) 2025-12-04T08:56:07.0872607Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0872677Z outputs = self.mobilebert( 2025-12-04T08:56:07.0872968Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0873049Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0873340Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0873420Z layer_outputs = layer_module( 2025-12-04T08:56:07.0873726Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T08:56:07.0873901Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T08:56:07.0874229Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-12-04T08:56:07.0874334Z layer_output = self.dense(intermediate_states) 2025-12-04T08:56:07.0874338Z 2025-12-04T08:56:07.0874455Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0874669Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0874737Z res = mod(**inputs) 2025-12-04T08:56:07.0875065Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0875140Z outputs = self.mobilebert( 2025-12-04T08:56:07.0875443Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0875530Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0875832Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0875939Z layer_outputs = layer_module( 2025-12-04T08:56:07.0876244Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T08:56:07.0876415Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T08:56:07.0876788Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-12-04T08:56:07.0876925Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-12-04T08:56:07.0877255Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:07.0877359Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:07.0877362Z 2025-12-04T08:56:07.0877492Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0877738Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0877810Z res = mod(**inputs) 2025-12-04T08:56:07.0878202Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0878285Z outputs = self.mobilebert( 2025-12-04T08:56:07.0878610Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0878699Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0879031Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0879145Z layer_outputs = layer_module( 2025-12-04T08:56:07.0879468Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T08:56:07.0879640Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T08:56:07.0879952Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-12-04T08:56:07.0880086Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-12-04T08:56:07.0880401Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-12-04T08:56:07.0880502Z layer_outputs = self.dense(hidden_states) 2025-12-04T08:56:07.0880506Z 2025-12-04T08:56:07.0880619Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0880838Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0880908Z res = mod(**inputs) 2025-12-04T08:56:07.0881212Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0881302Z outputs = self.mobilebert( 2025-12-04T08:56:07.0881613Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0881690Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0882000Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0882074Z layer_outputs = layer_module( 2025-12-04T08:56:07.0882387Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T08:56:07.0882551Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T08:56:07.0882854Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-12-04T08:56:07.0882994Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-12-04T08:56:07.0883319Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-12-04T08:56:07.0883456Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T08:56:07.0883766Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:07.0883861Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:07.0883865Z 2025-12-04T08:56:07.0883976Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0884176Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0884250Z res = mod(**inputs) 2025-12-04T08:56:07.0884570Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0884663Z outputs = self.mobilebert( 2025-12-04T08:56:07.0884985Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0885063Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0885371Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0885454Z layer_outputs = layer_module( 2025-12-04T08:56:07.0885764Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T08:56:07.0885944Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T08:56:07.0886269Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-12-04T08:56:07.0886388Z shared_attention_input = self.attention(hidden_states) 2025-12-04T08:56:07.0886695Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-12-04T08:56:07.0886782Z layer_input = self.dense(hidden_states) 2025-12-04T08:56:07.0886786Z 2025-12-04T08:56:07.0886902Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0887111Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0887179Z res = mod(**inputs) 2025-12-04T08:56:07.0887487Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0887566Z outputs = self.mobilebert( 2025-12-04T08:56:07.0887873Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0887952Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0888253Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0888336Z layer_outputs = layer_module( 2025-12-04T08:56:07.0888633Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T08:56:07.0888726Z self_attention_outputs = self.attention( 2025-12-04T08:56:07.0889032Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T08:56:07.0889109Z self_outputs = self.self( 2025-12-04T08:56:07.0889415Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-12-04T08:56:07.0889495Z self.value(value_tensor) 2025-12-04T08:56:07.0889501Z 2025-12-04T08:56:07.0889612Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0889851Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0889921Z res = mod(**inputs) 2025-12-04T08:56:07.0890234Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0890310Z outputs = self.mobilebert( 2025-12-04T08:56:07.0890614Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0890700Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0891002Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0891080Z layer_outputs = layer_module( 2025-12-04T08:56:07.0891411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T08:56:07.0891604Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T08:56:07.0891917Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-12-04T08:56:07.0892037Z bottlenecked_hidden_states = self.input(hidden_states) 2025-12-04T08:56:07.0892339Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-12-04T08:56:07.0892435Z layer_input = self.dense(hidden_states) 2025-12-04T08:56:07.0892439Z 2025-12-04T08:56:07.0892550Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0892796Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0892868Z res = mod(**inputs) 2025-12-04T08:56:07.0893181Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0893266Z outputs = self.mobilebert( 2025-12-04T08:56:07.0893567Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0893646Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0893966Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0894041Z layer_outputs = layer_module( 2025-12-04T08:56:07.0894356Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T08:56:07.0894528Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T08:56:07.0894832Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-12-04T08:56:07.0894960Z shared_attention_input = self.attention(hidden_states) 2025-12-04T08:56:07.0895263Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-12-04T08:56:07.0895362Z layer_input = self.LayerNorm(layer_input) 2025-12-04T08:56:07.0895673Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:07.0895772Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:07.0895775Z 2025-12-04T08:56:07.0895894Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0896105Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0896184Z res = mod(**inputs) 2025-12-04T08:56:07.0896489Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0896609Z outputs = self.mobilebert( 2025-12-04T08:56:07.0896929Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0897005Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0897320Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0897404Z layer_outputs = layer_module( 2025-12-04T08:56:07.0897715Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T08:56:07.0897818Z self_attention_outputs = self.attention( 2025-12-04T08:56:07.0898139Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T08:56:07.0898246Z self_outputs = self.self( 2025-12-04T08:56:07.0898562Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-12-04T08:56:07.0898639Z self.query(query_tensor) 2025-12-04T08:56:07.0898643Z 2025-12-04T08:56:07.0898759Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0898973Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0899042Z res = mod(**inputs) 2025-12-04T08:56:07.0899355Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0899429Z outputs = self.mobilebert( 2025-12-04T08:56:07.0899749Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0899837Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0900138Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0900221Z layer_outputs = layer_module( 2025-12-04T08:56:07.0900522Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T08:56:07.0900614Z self_attention_outputs = self.attention( 2025-12-04T08:56:07.0900929Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T08:56:07.0901005Z self_outputs = self.self( 2025-12-04T08:56:07.0901315Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-12-04T08:56:07.0901396Z self.key(key_tensor) 2025-12-04T08:56:07.0901400Z 2025-12-04T08:56:07.0901490Z cudagraph partition due to non gpu ops 2025-12-04T08:56:07.0901584Z cudagraph partition due to non gpu ops 2025-12-04T08:56:07.0901697Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0901906Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0901984Z res = mod(**inputs) 2025-12-04T08:56:07.0902284Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0902366Z outputs = self.mobilebert( 2025-12-04T08:56:07.0902674Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0902750Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0903059Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0903137Z layer_outputs = layer_module( 2025-12-04T08:56:07.0903457Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T08:56:07.0903557Z self_attention_outputs = self.attention( 2025-12-04T08:56:07.0903875Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-12-04T08:56:07.0904015Z attention_output = self.output(self_outputs[0], layer_input) 2025-12-04T08:56:07.0904316Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-12-04T08:56:07.0904406Z layer_outputs = self.dense(hidden_states) 2025-12-04T08:56:07.0904410Z 2025-12-04T08:56:07.0904526Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0904759Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0904853Z res = mod(**inputs) 2025-12-04T08:56:07.0905156Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0905232Z outputs = self.mobilebert( 2025-12-04T08:56:07.0905541Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0905618Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0905919Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0906001Z layer_outputs = layer_module( 2025-12-04T08:56:07.0906302Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T08:56:07.0906417Z self_attention_outputs = self.attention( 2025-12-04T08:56:07.0906721Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-12-04T08:56:07.0906857Z attention_output = self.output(self_outputs[0], layer_input) 2025-12-04T08:56:07.0907169Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-12-04T08:56:07.0907303Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T08:56:07.0907613Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:07.0907713Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:07.0907716Z 2025-12-04T08:56:07.0907827Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0908052Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0908123Z res = mod(**inputs) 2025-12-04T08:56:07.0908430Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0908512Z outputs = self.mobilebert( 2025-12-04T08:56:07.0908816Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0908899Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0909207Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0909282Z layer_outputs = layer_module( 2025-12-04T08:56:07.0909597Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.0909700Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.0910013Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T08:56:07.0910157Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T08:56:07.0910469Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T08:56:07.0910568Z hidden_states = self.dense(hidden_states) 2025-12-04T08:56:07.0910572Z 2025-12-04T08:56:07.0910680Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0910899Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0910968Z res = mod(**inputs) 2025-12-04T08:56:07.0911272Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0911366Z outputs = self.mobilebert( 2025-12-04T08:56:07.0911668Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0911765Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0912061Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0912134Z layer_outputs = layer_module( 2025-12-04T08:56:07.0912427Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.0912523Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.0912810Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T08:56:07.0912930Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T08:56:07.0913234Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T08:56:07.0913356Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:56:07.0913360Z 2025-12-04T08:56:07.0913465Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0913668Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0913741Z res = mod(**inputs) 2025-12-04T08:56:07.0914027Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0914098Z outputs = self.mobilebert( 2025-12-04T08:56:07.0914395Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0914467Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0914773Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0914849Z layer_outputs = layer_module( 2025-12-04T08:56:07.0915152Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.0915259Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.0915562Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T08:56:07.0915705Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T08:56:07.0916016Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T08:56:07.0916106Z layer_outputs = self.dense(hidden_states) 2025-12-04T08:56:07.0916110Z 2025-12-04T08:56:07.0916229Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0916442Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0916513Z res = mod(**inputs) 2025-12-04T08:56:07.0916842Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0916919Z outputs = self.mobilebert( 2025-12-04T08:56:07.0917227Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0917304Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0917613Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0917695Z layer_outputs = layer_module( 2025-12-04T08:56:07.0917994Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.0918191Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.0918532Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T08:56:07.0918672Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T08:56:07.0918998Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T08:56:07.0919132Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T08:56:07.0919466Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:07.0919567Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:07.0919571Z 2025-12-04T08:56:07.0919688Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0919944Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0920012Z res = mod(**inputs) 2025-12-04T08:56:07.0920314Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0920397Z outputs = self.mobilebert( 2025-12-04T08:56:07.0920862Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0920953Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0921242Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0921315Z layer_outputs = layer_module( 2025-12-04T08:56:07.0921615Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.0921719Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.0922029Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T08:56:07.0922153Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T08:56:07.0922457Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T08:56:07.0922558Z hidden_states = self.dense(hidden_states) 2025-12-04T08:56:07.0922563Z 2025-12-04T08:56:07.0922673Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0922887Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0922965Z res = mod(**inputs) 2025-12-04T08:56:07.0923279Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0923365Z outputs = self.mobilebert( 2025-12-04T08:56:07.0923669Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0923796Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0924093Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0924165Z layer_outputs = layer_module( 2025-12-04T08:56:07.0924459Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.0924555Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.0924848Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T08:56:07.0924970Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T08:56:07.0925296Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T08:56:07.0925434Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:56:07.0925447Z 2025-12-04T08:56:07.0925552Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0925752Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0925827Z res = mod(**inputs) 2025-12-04T08:56:07.0926112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0926185Z outputs = self.mobilebert( 2025-12-04T08:56:07.0926487Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0926585Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0926879Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0926955Z layer_outputs = layer_module( 2025-12-04T08:56:07.0927241Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.0927345Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.0927630Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T08:56:07.0927762Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T08:56:07.0928047Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T08:56:07.0928144Z layer_outputs = self.dense(hidden_states) 2025-12-04T08:56:07.0928148Z 2025-12-04T08:56:07.0928262Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0928465Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0928533Z res = mod(**inputs) 2025-12-04T08:56:07.0928825Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0928895Z outputs = self.mobilebert( 2025-12-04T08:56:07.0929188Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0929261Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0929546Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0929625Z layer_outputs = layer_module( 2025-12-04T08:56:07.0929909Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.0930012Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.0930317Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T08:56:07.0930446Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T08:56:07.0930742Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T08:56:07.0930867Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T08:56:07.0931156Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:07.0931259Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:07.0931263Z 2025-12-04T08:56:07.0931369Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0931595Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0931675Z res = mod(**inputs) 2025-12-04T08:56:07.0931961Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0932043Z outputs = self.mobilebert( 2025-12-04T08:56:07.0932330Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0932410Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0932725Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0932801Z layer_outputs = layer_module( 2025-12-04T08:56:07.0933114Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.0933233Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.0933546Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T08:56:07.0933676Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T08:56:07.0933985Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T08:56:07.0934082Z hidden_states = self.dense(hidden_states) 2025-12-04T08:56:07.0934085Z 2025-12-04T08:56:07.0934196Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0934409Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0934482Z res = mod(**inputs) 2025-12-04T08:56:07.0934768Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0934846Z outputs = self.mobilebert( 2025-12-04T08:56:07.0935135Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0935208Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0935507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0935583Z layer_outputs = layer_module( 2025-12-04T08:56:07.0935892Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.0935998Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.0936311Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T08:56:07.0936439Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T08:56:07.0936740Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T08:56:07.0936884Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:56:07.0936896Z 2025-12-04T08:56:07.0937010Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0937224Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0937301Z res = mod(**inputs) 2025-12-04T08:56:07.0937616Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0937692Z outputs = self.mobilebert( 2025-12-04T08:56:07.0938018Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0938097Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0938434Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0938527Z layer_outputs = layer_module( 2025-12-04T08:56:07.0938829Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.0938935Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.0939235Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T08:56:07.0939369Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T08:56:07.0939690Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T08:56:07.0939780Z layer_outputs = self.dense(hidden_states) 2025-12-04T08:56:07.0939801Z 2025-12-04T08:56:07.0939919Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0940134Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0940204Z res = mod(**inputs) 2025-12-04T08:56:07.0940515Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0940590Z outputs = self.mobilebert( 2025-12-04T08:56:07.0940897Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0940975Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0941289Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0941372Z layer_outputs = layer_module( 2025-12-04T08:56:07.0941678Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.0941781Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.0942091Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T08:56:07.0942224Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T08:56:07.0942535Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T08:56:07.0942667Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T08:56:07.0942979Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:07.0943084Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:07.0943089Z 2025-12-04T08:56:07.0943199Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0943422Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0943495Z res = mod(**inputs) 2025-12-04T08:56:07.0943828Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0943914Z outputs = self.mobilebert( 2025-12-04T08:56:07.0944219Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0944305Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0944625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0944701Z layer_outputs = layer_module( 2025-12-04T08:56:07.0945022Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-12-04T08:56:07.0945171Z intermediate_output = self.intermediate(attention_output) 2025-12-04T08:56:07.0945491Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T08:56:07.0945589Z hidden_states = self.dense(hidden_states) 2025-12-04T08:56:07.0945593Z 2025-12-04T08:56:07.0945701Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0945920Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0945990Z res = mod(**inputs) 2025-12-04T08:56:07.0946290Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0946374Z outputs = self.mobilebert( 2025-12-04T08:56:07.0946677Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0946780Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0947087Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0947163Z layer_outputs = layer_module( 2025-12-04T08:56:07.0947473Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-12-04T08:56:07.0947602Z intermediate_output = self.intermediate(attention_output) 2025-12-04T08:56:07.0947905Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T08:56:07.0948033Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:56:07.0948037Z 2025-12-04T08:56:07.0948144Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0948367Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0948436Z res = mod(**inputs) 2025-12-04T08:56:07.0948739Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0948822Z outputs = self.mobilebert( 2025-12-04T08:56:07.0949125Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0949208Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0949511Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0949587Z layer_outputs = layer_module( 2025-12-04T08:56:07.0949896Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T08:56:07.0950070Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T08:56:07.0950374Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-12-04T08:56:07.0950514Z layer_output = self.dense(intermediate_states) 2025-12-04T08:56:07.0950519Z 2025-12-04T08:56:07.0950629Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0950847Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0950916Z res = mod(**inputs) 2025-12-04T08:56:07.0951217Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0951300Z outputs = self.mobilebert( 2025-12-04T08:56:07.0951601Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0951686Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0952001Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0952095Z layer_outputs = layer_module( 2025-12-04T08:56:07.0952407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T08:56:07.0952577Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T08:56:07.0952885Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-12-04T08:56:07.0953017Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-12-04T08:56:07.0953317Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:07.0953441Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:07.0953445Z 2025-12-04T08:56:07.0953555Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0953771Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0953846Z res = mod(**inputs) 2025-12-04T08:56:07.0954146Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0954229Z outputs = self.mobilebert( 2025-12-04T08:56:07.0954528Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0954605Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0954912Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0954991Z layer_outputs = layer_module( 2025-12-04T08:56:07.0955301Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T08:56:07.0955473Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T08:56:07.0955773Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-12-04T08:56:07.0955912Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-12-04T08:56:07.0956232Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-12-04T08:56:07.0956324Z layer_outputs = self.dense(hidden_states) 2025-12-04T08:56:07.0956334Z 2025-12-04T08:56:07.0956443Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0956656Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0956734Z res = mod(**inputs) 2025-12-04T08:56:07.0957039Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0957132Z outputs = self.mobilebert( 2025-12-04T08:56:07.0957443Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0957521Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0957832Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0957910Z layer_outputs = layer_module( 2025-12-04T08:56:07.0958315Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T08:56:07.0958503Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T08:56:07.0958857Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-12-04T08:56:07.0959019Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-12-04T08:56:07.0959340Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-12-04T08:56:07.0959473Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T08:56:07.0959784Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:07.0959884Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:07.0959888Z 2025-12-04T08:56:07.0959999Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0960219Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0960307Z res = mod(**inputs) 2025-12-04T08:56:07.0960619Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0960697Z outputs = self.mobilebert( 2025-12-04T08:56:07.0961001Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0961083Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0961369Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0961447Z layer_outputs = layer_module( 2025-12-04T08:56:07.0961738Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T08:56:07.0961908Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T08:56:07.0962208Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-12-04T08:56:07.0962321Z shared_attention_input = self.attention(hidden_states) 2025-12-04T08:56:07.0962629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-12-04T08:56:07.0962713Z layer_input = self.dense(hidden_states) 2025-12-04T08:56:07.0962717Z 2025-12-04T08:56:07.0962821Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0963026Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0963093Z res = mod(**inputs) 2025-12-04T08:56:07.0963377Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0963459Z outputs = self.mobilebert( 2025-12-04T08:56:07.0963744Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0963826Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0964129Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0964202Z layer_outputs = layer_module( 2025-12-04T08:56:07.0964496Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T08:56:07.0964585Z self_attention_outputs = self.attention( 2025-12-04T08:56:07.0964881Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T08:56:07.0964955Z self_outputs = self.self( 2025-12-04T08:56:07.0965240Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-12-04T08:56:07.0965387Z self.value(value_tensor) 2025-12-04T08:56:07.0965405Z 2025-12-04T08:56:07.0965510Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0965713Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0965787Z res = mod(**inputs) 2025-12-04T08:56:07.0966076Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0966153Z outputs = self.mobilebert( 2025-12-04T08:56:07.0966443Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0966516Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0966811Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0966902Z layer_outputs = layer_module( 2025-12-04T08:56:07.0967192Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T08:56:07.0967367Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T08:56:07.0967656Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-12-04T08:56:07.0967779Z bottlenecked_hidden_states = self.input(hidden_states) 2025-12-04T08:56:07.0968069Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-12-04T08:56:07.0968152Z layer_input = self.dense(hidden_states) 2025-12-04T08:56:07.0968165Z 2025-12-04T08:56:07.0968271Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0968475Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0968550Z res = mod(**inputs) 2025-12-04T08:56:07.0968841Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0968912Z outputs = self.mobilebert( 2025-12-04T08:56:07.0969208Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0969281Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0969573Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0969644Z layer_outputs = layer_module( 2025-12-04T08:56:07.0969928Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T08:56:07.0970098Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T08:56:07.0970389Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-12-04T08:56:07.0970521Z shared_attention_input = self.attention(hidden_states) 2025-12-04T08:56:07.0970817Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-12-04T08:56:07.0970906Z layer_input = self.LayerNorm(layer_input) 2025-12-04T08:56:07.0971198Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:07.0971288Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:07.0971292Z 2025-12-04T08:56:07.0971394Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0971601Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0971669Z res = mod(**inputs) 2025-12-04T08:56:07.0971973Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0972065Z outputs = self.mobilebert( 2025-12-04T08:56:07.0972359Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0972440Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0972746Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0972823Z layer_outputs = layer_module( 2025-12-04T08:56:07.0973115Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T08:56:07.0973203Z self_attention_outputs = self.attention( 2025-12-04T08:56:07.0973517Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T08:56:07.0973593Z self_outputs = self.self( 2025-12-04T08:56:07.0973880Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-12-04T08:56:07.0973961Z self.query(query_tensor) 2025-12-04T08:56:07.0973964Z 2025-12-04T08:56:07.0974068Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0974275Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0974340Z res = mod(**inputs) 2025-12-04T08:56:07.0974623Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0974703Z outputs = self.mobilebert( 2025-12-04T08:56:07.0974993Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0975068Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0975362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0975435Z layer_outputs = layer_module( 2025-12-04T08:56:07.0975728Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T08:56:07.0975816Z self_attention_outputs = self.attention( 2025-12-04T08:56:07.0976099Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T08:56:07.0976179Z self_outputs = self.self( 2025-12-04T08:56:07.0976466Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-12-04T08:56:07.0976545Z self.key(key_tensor) 2025-12-04T08:56:07.0976548Z 2025-12-04T08:56:07.0976637Z cudagraph partition due to non gpu ops 2025-12-04T08:56:07.0976719Z cudagraph partition due to non gpu ops 2025-12-04T08:56:07.0976847Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0977054Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0977120Z res = mod(**inputs) 2025-12-04T08:56:07.0977412Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0977484Z outputs = self.mobilebert( 2025-12-04T08:56:07.0977797Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0977874Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0978175Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0978292Z layer_outputs = layer_module( 2025-12-04T08:56:07.0978598Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T08:56:07.0978695Z self_attention_outputs = self.attention( 2025-12-04T08:56:07.0978996Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-12-04T08:56:07.0979128Z attention_output = self.output(self_outputs[0], layer_input) 2025-12-04T08:56:07.0979439Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-12-04T08:56:07.0979536Z layer_outputs = self.dense(hidden_states) 2025-12-04T08:56:07.0979539Z 2025-12-04T08:56:07.0979643Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0979869Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0979935Z res = mod(**inputs) 2025-12-04T08:56:07.0980229Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0980300Z outputs = self.mobilebert( 2025-12-04T08:56:07.0980585Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0980666Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0980953Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0981032Z layer_outputs = layer_module( 2025-12-04T08:56:07.0981322Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T08:56:07.0981409Z self_attention_outputs = self.attention( 2025-12-04T08:56:07.0981701Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-12-04T08:56:07.0981828Z attention_output = self.output(self_outputs[0], layer_input) 2025-12-04T08:56:07.0982112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-12-04T08:56:07.0982247Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T08:56:07.0982538Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:07.0982637Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:07.0982640Z 2025-12-04T08:56:07.0982744Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0982946Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0983021Z res = mod(**inputs) 2025-12-04T08:56:07.0983309Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0983404Z outputs = self.mobilebert( 2025-12-04T08:56:07.0983691Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0983764Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0984058Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0984130Z layer_outputs = layer_module( 2025-12-04T08:56:07.0984418Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.0984525Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.0984828Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T08:56:07.0984968Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T08:56:07.0985253Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T08:56:07.0985340Z hidden_states = self.dense(hidden_states) 2025-12-04T08:56:07.0985344Z 2025-12-04T08:56:07.0985456Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0985656Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0985727Z res = mod(**inputs) 2025-12-04T08:56:07.0986014Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0986103Z outputs = self.mobilebert( 2025-12-04T08:56:07.0986394Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0986469Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0986760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0986830Z layer_outputs = layer_module( 2025-12-04T08:56:07.0987115Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.0987215Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.0987500Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T08:56:07.0987611Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T08:56:07.0987907Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T08:56:07.0988021Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:56:07.0988025Z 2025-12-04T08:56:07.0988137Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0988334Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0988398Z res = mod(**inputs) 2025-12-04T08:56:07.0988687Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0988759Z outputs = self.mobilebert( 2025-12-04T08:56:07.0989056Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0989128Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0989414Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0989493Z layer_outputs = layer_module( 2025-12-04T08:56:07.0989791Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.0989888Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.0990192Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T08:56:07.0990329Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T08:56:07.0990642Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T08:56:07.0990732Z layer_outputs = self.dense(hidden_states) 2025-12-04T08:56:07.0990736Z 2025-12-04T08:56:07.0990847Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0991087Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0991175Z res = mod(**inputs) 2025-12-04T08:56:07.0991498Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0991570Z outputs = self.mobilebert( 2025-12-04T08:56:07.0991857Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0991940Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0992244Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0992319Z layer_outputs = layer_module( 2025-12-04T08:56:07.0992628Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.0992749Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.0993060Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T08:56:07.0993198Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T08:56:07.0993501Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T08:56:07.0993642Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T08:56:07.0993945Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:07.0994050Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:07.0994053Z 2025-12-04T08:56:07.0994162Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0994376Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0994455Z res = mod(**inputs) 2025-12-04T08:56:07.0994760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0994845Z outputs = self.mobilebert( 2025-12-04T08:56:07.0995161Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0995238Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0995549Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0995626Z layer_outputs = layer_module( 2025-12-04T08:56:07.0995929Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.0996039Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.0996344Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T08:56:07.0996492Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T08:56:07.0996794Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T08:56:07.0996891Z hidden_states = self.dense(hidden_states) 2025-12-04T08:56:07.0996894Z 2025-12-04T08:56:07.0997010Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.0997222Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.0997298Z res = mod(**inputs) 2025-12-04T08:56:07.0997607Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.0997686Z outputs = self.mobilebert( 2025-12-04T08:56:07.0998011Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.0998202Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.0998512Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.0998601Z layer_outputs = layer_module( 2025-12-04T08:56:07.0998914Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.0999026Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.0999353Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T08:56:07.0999477Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T08:56:07.0999823Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T08:56:07.0999960Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:56:07.0999964Z 2025-12-04T08:56:07.1000082Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1000295Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1000365Z res = mod(**inputs) 2025-12-04T08:56:07.1000678Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1000754Z outputs = self.mobilebert( 2025-12-04T08:56:07.1001064Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1001151Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1001458Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1001541Z layer_outputs = layer_module( 2025-12-04T08:56:07.1001844Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.1001944Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.1002251Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T08:56:07.1002384Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T08:56:07.1002701Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T08:56:07.1002791Z layer_outputs = self.dense(hidden_states) 2025-12-04T08:56:07.1002796Z 2025-12-04T08:56:07.1002905Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1003127Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1003195Z res = mod(**inputs) 2025-12-04T08:56:07.1003531Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1003608Z outputs = self.mobilebert( 2025-12-04T08:56:07.1003910Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1003994Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1004306Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1004380Z layer_outputs = layer_module( 2025-12-04T08:56:07.1004702Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.1004821Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.1005149Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T08:56:07.1005282Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T08:56:07.1005585Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T08:56:07.1005725Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T08:56:07.1006030Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:07.1006135Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:07.1006139Z 2025-12-04T08:56:07.1006265Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1006483Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1006561Z res = mod(**inputs) 2025-12-04T08:56:07.1006867Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1006943Z outputs = self.mobilebert( 2025-12-04T08:56:07.1007253Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1007330Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1007644Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1007719Z layer_outputs = layer_module( 2025-12-04T08:56:07.1008023Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.1008131Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.1008445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T08:56:07.1008574Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T08:56:07.1008879Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T08:56:07.1008969Z hidden_states = self.dense(hidden_states) 2025-12-04T08:56:07.1008972Z 2025-12-04T08:56:07.1009089Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1009303Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1009378Z res = mod(**inputs) 2025-12-04T08:56:07.1009681Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1009760Z outputs = self.mobilebert( 2025-12-04T08:56:07.1010073Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1010170Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1010475Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1010559Z layer_outputs = layer_module( 2025-12-04T08:56:07.1010862Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.1010968Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.1011275Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T08:56:07.1011396Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T08:56:07.1011723Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T08:56:07.1011864Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:56:07.1011868Z 2025-12-04T08:56:07.1011986Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1012202Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1012270Z res = mod(**inputs) 2025-12-04T08:56:07.1012586Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1012663Z outputs = self.mobilebert( 2025-12-04T08:56:07.1012974Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1013080Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1013387Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1013474Z layer_outputs = layer_module( 2025-12-04T08:56:07.1013780Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.1013882Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.1014198Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T08:56:07.1014329Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T08:56:07.1014642Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T08:56:07.1014733Z layer_outputs = self.dense(hidden_states) 2025-12-04T08:56:07.1014738Z 2025-12-04T08:56:07.1014854Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1015061Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1015127Z res = mod(**inputs) 2025-12-04T08:56:07.1015413Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1015494Z outputs = self.mobilebert( 2025-12-04T08:56:07.1015779Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1015858Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1016143Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1016215Z layer_outputs = layer_module( 2025-12-04T08:56:07.1016520Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.1016619Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.1016928Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T08:56:07.1017056Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T08:56:07.1017340Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T08:56:07.1017471Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T08:56:07.1017756Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:07.1017856Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:07.1017861Z 2025-12-04T08:56:07.1017965Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1018185Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1018280Z res = mod(**inputs) 2025-12-04T08:56:07.1018574Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1018646Z outputs = self.mobilebert( 2025-12-04T08:56:07.1018954Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1019027Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1019320Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1019390Z layer_outputs = layer_module( 2025-12-04T08:56:07.1019678Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-12-04T08:56:07.1019826Z intermediate_output = self.intermediate(attention_output) 2025-12-04T08:56:07.1020118Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T08:56:07.1020211Z hidden_states = self.dense(hidden_states) 2025-12-04T08:56:07.1020214Z 2025-12-04T08:56:07.1020318Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1020519Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1020590Z res = mod(**inputs) 2025-12-04T08:56:07.1021041Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1021119Z outputs = self.mobilebert( 2025-12-04T08:56:07.1021414Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1021492Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1021788Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1021861Z layer_outputs = layer_module( 2025-12-04T08:56:07.1022145Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-12-04T08:56:07.1022274Z intermediate_output = self.intermediate(attention_output) 2025-12-04T08:56:07.1022558Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T08:56:07.1022677Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:56:07.1022681Z 2025-12-04T08:56:07.1022785Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1022986Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1023060Z res = mod(**inputs) 2025-12-04T08:56:07.1023387Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1023459Z outputs = self.mobilebert( 2025-12-04T08:56:07.1023753Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1023826Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1024118Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1024190Z layer_outputs = layer_module( 2025-12-04T08:56:07.1024473Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T08:56:07.1024646Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T08:56:07.1024973Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-12-04T08:56:07.1025104Z layer_output = self.dense(intermediate_states) 2025-12-04T08:56:07.1025107Z 2025-12-04T08:56:07.1025216Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1025417Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1025490Z res = mod(**inputs) 2025-12-04T08:56:07.1025776Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1025848Z outputs = self.mobilebert( 2025-12-04T08:56:07.1026149Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1026246Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1026541Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1026613Z layer_outputs = layer_module( 2025-12-04T08:56:07.1026901Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T08:56:07.1027069Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T08:56:07.1027358Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-12-04T08:56:07.1027491Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-12-04T08:56:07.1027778Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:07.1027874Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:07.1027877Z 2025-12-04T08:56:07.1027993Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1028195Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1028269Z res = mod(**inputs) 2025-12-04T08:56:07.1028558Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1028629Z outputs = self.mobilebert( 2025-12-04T08:56:07.1028924Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1028998Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1029281Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1029363Z layer_outputs = layer_module( 2025-12-04T08:56:07.1029804Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T08:56:07.1030062Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T08:56:07.1030543Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-12-04T08:56:07.1030677Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-12-04T08:56:07.1031066Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-12-04T08:56:07.1031156Z layer_outputs = self.dense(hidden_states) 2025-12-04T08:56:07.1031160Z 2025-12-04T08:56:07.1031315Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1031522Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1031624Z res = mod(**inputs) 2025-12-04T08:56:07.1031944Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1032120Z outputs = self.mobilebert( 2025-12-04T08:56:07.1032435Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1032552Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1032889Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1033021Z layer_outputs = layer_module( 2025-12-04T08:56:07.1033370Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T08:56:07.1033546Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T08:56:07.1033875Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-12-04T08:56:07.1034016Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-12-04T08:56:07.1034319Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-12-04T08:56:07.1034451Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T08:56:07.1034765Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:07.1034864Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:07.1034868Z 2025-12-04T08:56:07.1034987Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1035204Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1035277Z res = mod(**inputs) 2025-12-04T08:56:07.1035588Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1035666Z outputs = self.mobilebert( 2025-12-04T08:56:07.1035969Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1036053Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1036355Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1036438Z layer_outputs = layer_module( 2025-12-04T08:56:07.1036741Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T08:56:07.1036915Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T08:56:07.1037227Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-12-04T08:56:07.1037347Z shared_attention_input = self.attention(hidden_states) 2025-12-04T08:56:07.1037674Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-12-04T08:56:07.1037768Z layer_input = self.dense(hidden_states) 2025-12-04T08:56:07.1037772Z 2025-12-04T08:56:07.1037892Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1038236Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1038343Z res = mod(**inputs) 2025-12-04T08:56:07.1038672Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1038763Z outputs = self.mobilebert( 2025-12-04T08:56:07.1039100Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1039207Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1039522Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1039601Z layer_outputs = layer_module( 2025-12-04T08:56:07.1039921Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T08:56:07.1040027Z self_attention_outputs = self.attention( 2025-12-04T08:56:07.1040337Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T08:56:07.1040414Z self_outputs = self.self( 2025-12-04T08:56:07.1040721Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-12-04T08:56:07.1040826Z self.value(value_tensor) 2025-12-04T08:56:07.1040832Z 2025-12-04T08:56:07.1040944Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1041157Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1041232Z res = mod(**inputs) 2025-12-04T08:56:07.1041532Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1041613Z outputs = self.mobilebert( 2025-12-04T08:56:07.1041923Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1042001Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1042318Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1042398Z layer_outputs = layer_module( 2025-12-04T08:56:07.1042707Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T08:56:07.1042884Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T08:56:07.1043187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-12-04T08:56:07.1043315Z bottlenecked_hidden_states = self.input(hidden_states) 2025-12-04T08:56:07.1043628Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-12-04T08:56:07.1043715Z layer_input = self.dense(hidden_states) 2025-12-04T08:56:07.1043727Z 2025-12-04T08:56:07.1043838Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1044050Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1044129Z res = mod(**inputs) 2025-12-04T08:56:07.1044433Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1044527Z outputs = self.mobilebert( 2025-12-04T08:56:07.1044839Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1044918Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1045235Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1045311Z layer_outputs = layer_module( 2025-12-04T08:56:07.1045618Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T08:56:07.1045797Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T08:56:07.1046119Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-12-04T08:56:07.1046264Z shared_attention_input = self.attention(hidden_states) 2025-12-04T08:56:07.1046568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-12-04T08:56:07.1046663Z layer_input = self.LayerNorm(layer_input) 2025-12-04T08:56:07.1046980Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:07.1047079Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:07.1047083Z 2025-12-04T08:56:07.1047195Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1047413Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1047500Z res = mod(**inputs) 2025-12-04T08:56:07.1047821Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1047902Z outputs = self.mobilebert( 2025-12-04T08:56:07.1048208Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1048295Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1048597Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1048682Z layer_outputs = layer_module( 2025-12-04T08:56:07.1049051Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T08:56:07.1049152Z self_attention_outputs = self.attention( 2025-12-04T08:56:07.1049449Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T08:56:07.1049524Z self_outputs = self.self( 2025-12-04T08:56:07.1049811Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-12-04T08:56:07.1049890Z self.query(query_tensor) 2025-12-04T08:56:07.1049894Z 2025-12-04T08:56:07.1049999Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1050205Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1050272Z res = mod(**inputs) 2025-12-04T08:56:07.1050559Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1050638Z outputs = self.mobilebert( 2025-12-04T08:56:07.1050923Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1051007Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1051317Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1051392Z layer_outputs = layer_module( 2025-12-04T08:56:07.1051687Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T08:56:07.1051774Z self_attention_outputs = self.attention( 2025-12-04T08:56:07.1052070Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T08:56:07.1052151Z self_outputs = self.self( 2025-12-04T08:56:07.1052439Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-12-04T08:56:07.1052517Z self.key(key_tensor) 2025-12-04T08:56:07.1052520Z 2025-12-04T08:56:07.1052620Z cudagraph partition due to non gpu ops 2025-12-04T08:56:07.1052722Z cudagraph partition due to non gpu ops 2025-12-04T08:56:07.1052837Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1053043Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1053109Z res = mod(**inputs) 2025-12-04T08:56:07.1053404Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1053477Z outputs = self.mobilebert( 2025-12-04T08:56:07.1053773Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1053852Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1054162Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1054244Z layer_outputs = layer_module( 2025-12-04T08:56:07.1054534Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T08:56:07.1054626Z self_attention_outputs = self.attention( 2025-12-04T08:56:07.1054921Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-12-04T08:56:07.1055046Z attention_output = self.output(self_outputs[0], layer_input) 2025-12-04T08:56:07.1055340Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-12-04T08:56:07.1055426Z layer_outputs = self.dense(hidden_states) 2025-12-04T08:56:07.1055429Z 2025-12-04T08:56:07.1055534Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1055744Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1055810Z res = mod(**inputs) 2025-12-04T08:56:07.1056103Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1056176Z outputs = self.mobilebert( 2025-12-04T08:56:07.1056460Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1056541Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1056831Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1056909Z layer_outputs = layer_module( 2025-12-04T08:56:07.1057192Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T08:56:07.1057281Z self_attention_outputs = self.attention( 2025-12-04T08:56:07.1057574Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-12-04T08:56:07.1057715Z attention_output = self.output(self_outputs[0], layer_input) 2025-12-04T08:56:07.1058001Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-12-04T08:56:07.1058137Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T08:56:07.1058430Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:07.1058527Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:07.1058530Z 2025-12-04T08:56:07.1058635Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1058836Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1058926Z res = mod(**inputs) 2025-12-04T08:56:07.1059228Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1059306Z outputs = self.mobilebert( 2025-12-04T08:56:07.1059593Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1059667Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1059958Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1060028Z layer_outputs = layer_module( 2025-12-04T08:56:07.1060320Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.1060440Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.1060740Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T08:56:07.1060865Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T08:56:07.1061154Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T08:56:07.1061240Z hidden_states = self.dense(hidden_states) 2025-12-04T08:56:07.1061251Z 2025-12-04T08:56:07.1061355Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1061554Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1061626Z res = mod(**inputs) 2025-12-04T08:56:07.1061916Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1061991Z outputs = self.mobilebert( 2025-12-04T08:56:07.1062304Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1062381Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1062677Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1062755Z layer_outputs = layer_module( 2025-12-04T08:56:07.1063042Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.1063146Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.1063431Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T08:56:07.1063544Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T08:56:07.1063842Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T08:56:07.1063957Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:56:07.1063961Z 2025-12-04T08:56:07.1064094Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1064297Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1064361Z res = mod(**inputs) 2025-12-04T08:56:07.1064663Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1064738Z outputs = self.mobilebert( 2025-12-04T08:56:07.1065029Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1065102Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1065409Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1065508Z layer_outputs = layer_module( 2025-12-04T08:56:07.1065803Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.1065899Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.1066211Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T08:56:07.1066337Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T08:56:07.1066621Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T08:56:07.1066703Z layer_outputs = self.dense(hidden_states) 2025-12-04T08:56:07.1066708Z 2025-12-04T08:56:07.1066829Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1067039Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1067106Z res = mod(**inputs) 2025-12-04T08:56:07.1067397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1067467Z outputs = self.mobilebert( 2025-12-04T08:56:07.1067747Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1067829Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1068109Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1068178Z layer_outputs = layer_module( 2025-12-04T08:56:07.1068464Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.1068557Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.1068849Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T08:56:07.1068975Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T08:56:07.1069251Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T08:56:07.1069380Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T08:56:07.1069668Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:07.1069767Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:07.1069771Z 2025-12-04T08:56:07.1069874Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1070081Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1070157Z res = mod(**inputs) 2025-12-04T08:56:07.1070521Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1070603Z outputs = self.mobilebert( 2025-12-04T08:56:07.1070896Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1070968Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1071255Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1071326Z layer_outputs = layer_module( 2025-12-04T08:56:07.1071615Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.1071718Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.1072027Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T08:56:07.1072169Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T08:56:07.1072457Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T08:56:07.1072543Z hidden_states = self.dense(hidden_states) 2025-12-04T08:56:07.1072547Z 2025-12-04T08:56:07.1072662Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1072866Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1072939Z res = mod(**inputs) 2025-12-04T08:56:07.1073228Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1073331Z outputs = self.mobilebert( 2025-12-04T08:56:07.1073632Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1073713Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1074026Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1074110Z layer_outputs = layer_module( 2025-12-04T08:56:07.1074416Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.1074522Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.1074828Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T08:56:07.1074948Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T08:56:07.1075266Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T08:56:07.1075389Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:56:07.1075393Z 2025-12-04T08:56:07.1075510Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1075729Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1075800Z res = mod(**inputs) 2025-12-04T08:56:07.1076122Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1076200Z outputs = self.mobilebert( 2025-12-04T08:56:07.1076526Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1076604Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1076916Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1077002Z layer_outputs = layer_module( 2025-12-04T08:56:07.1077330Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.1077435Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.1077749Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T08:56:07.1077887Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T08:56:07.1078371Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T08:56:07.1078480Z layer_outputs = self.dense(hidden_states) 2025-12-04T08:56:07.1078491Z 2025-12-04T08:56:07.1078612Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1078872Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1078962Z res = mod(**inputs) 2025-12-04T08:56:07.1079287Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1079364Z outputs = self.mobilebert( 2025-12-04T08:56:07.1079686Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1079771Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1080086Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1080169Z layer_outputs = layer_module( 2025-12-04T08:56:07.1080494Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.1080614Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.1080914Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T08:56:07.1081048Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T08:56:07.1081351Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T08:56:07.1081495Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T08:56:07.1081803Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:07.1081911Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:07.1081915Z 2025-12-04T08:56:07.1082026Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1082242Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1082320Z res = mod(**inputs) 2025-12-04T08:56:07.1082639Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1082717Z outputs = self.mobilebert( 2025-12-04T08:56:07.1083025Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1083102Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1083418Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1083494Z layer_outputs = layer_module( 2025-12-04T08:56:07.1083807Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.1083914Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.1084218Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T08:56:07.1084370Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T08:56:07.1084674Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T08:56:07.1084764Z hidden_states = self.dense(hidden_states) 2025-12-04T08:56:07.1084767Z 2025-12-04T08:56:07.1084888Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1085105Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1085186Z res = mod(**inputs) 2025-12-04T08:56:07.1085502Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1085582Z outputs = self.mobilebert( 2025-12-04T08:56:07.1085910Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1086005Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1086306Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1086389Z layer_outputs = layer_module( 2025-12-04T08:56:07.1086696Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.1086805Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.1087114Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T08:56:07.1087254Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T08:56:07.1087585Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T08:56:07.1087712Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:56:07.1087717Z 2025-12-04T08:56:07.1087835Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1088048Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1088117Z res = mod(**inputs) 2025-12-04T08:56:07.1088437Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1088514Z outputs = self.mobilebert( 2025-12-04T08:56:07.1088824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1088911Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1089218Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1089301Z layer_outputs = layer_module( 2025-12-04T08:56:07.1089609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.1089709Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.1090033Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T08:56:07.1090169Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T08:56:07.1090495Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T08:56:07.1090585Z layer_outputs = self.dense(hidden_states) 2025-12-04T08:56:07.1090591Z 2025-12-04T08:56:07.1090700Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1090923Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1090996Z res = mod(**inputs) 2025-12-04T08:56:07.1091324Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1091409Z outputs = self.mobilebert( 2025-12-04T08:56:07.1091736Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1091823Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1092126Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1092201Z layer_outputs = layer_module( 2025-12-04T08:56:07.1092514Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.1092652Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.1092961Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T08:56:07.1093095Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T08:56:07.1093397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T08:56:07.1093541Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T08:56:07.1093842Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:07.1093946Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:07.1093969Z 2025-12-04T08:56:07.1094082Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1094296Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1094374Z res = mod(**inputs) 2025-12-04T08:56:07.1094676Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1094753Z outputs = self.mobilebert( 2025-12-04T08:56:07.1095067Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1095148Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1095474Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1095551Z layer_outputs = layer_module( 2025-12-04T08:56:07.1095856Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-12-04T08:56:07.1096003Z intermediate_output = self.intermediate(attention_output) 2025-12-04T08:56:07.1096309Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T08:56:07.1096408Z hidden_states = self.dense(hidden_states) 2025-12-04T08:56:07.1096412Z 2025-12-04T08:56:07.1096525Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1096741Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1096820Z res = mod(**inputs) 2025-12-04T08:56:07.1097122Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1097198Z outputs = self.mobilebert( 2025-12-04T08:56:07.1097505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1097588Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1097902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1098003Z layer_outputs = layer_module( 2025-12-04T08:56:07.1098311Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-12-04T08:56:07.1098448Z intermediate_output = self.intermediate(attention_output) 2025-12-04T08:56:07.1098753Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T08:56:07.1098880Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:56:07.1098884Z 2025-12-04T08:56:07.1098994Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1099208Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1099303Z res = mod(**inputs) 2025-12-04T08:56:07.1099634Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1099714Z outputs = self.mobilebert( 2025-12-04T08:56:07.1100026Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1100103Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1100410Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1100486Z layer_outputs = layer_module( 2025-12-04T08:56:07.1100787Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T08:56:07.1100985Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T08:56:07.1101288Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-12-04T08:56:07.1101402Z layer_output = self.dense(intermediate_states) 2025-12-04T08:56:07.1101406Z 2025-12-04T08:56:07.1101518Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1101728Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1101808Z res = mod(**inputs) 2025-12-04T08:56:07.1102120Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1102204Z outputs = self.mobilebert( 2025-12-04T08:56:07.1102503Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1102585Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1102895Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1102972Z layer_outputs = layer_module( 2025-12-04T08:56:07.1103279Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T08:56:07.1103459Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T08:56:07.1103851Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-12-04T08:56:07.1104069Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-12-04T08:56:07.1104563Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:07.1104704Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:07.1104707Z 2025-12-04T08:56:07.1104830Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1105101Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1105236Z res = mod(**inputs) 2025-12-04T08:56:07.1105549Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1105627Z outputs = self.mobilebert( 2025-12-04T08:56:07.1105938Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1106016Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1106340Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1106429Z layer_outputs = layer_module( 2025-12-04T08:56:07.1106759Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T08:56:07.1106957Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T08:56:07.1107270Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-12-04T08:56:07.1107408Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-12-04T08:56:07.1107732Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-12-04T08:56:07.1107824Z layer_outputs = self.dense(hidden_states) 2025-12-04T08:56:07.1107828Z 2025-12-04T08:56:07.1107949Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1108165Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1108258Z res = mod(**inputs) 2025-12-04T08:56:07.1108592Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1108673Z outputs = self.mobilebert( 2025-12-04T08:56:07.1109007Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1109087Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1109407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1109492Z layer_outputs = layer_module( 2025-12-04T08:56:07.1109816Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T08:56:07.1109991Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T08:56:07.1110326Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-12-04T08:56:07.1110467Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-12-04T08:56:07.1110791Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-12-04T08:56:07.1110927Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T08:56:07.1111244Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:07.1111354Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:07.1111358Z 2025-12-04T08:56:07.1111472Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1111702Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1111776Z res = mod(**inputs) 2025-12-04T08:56:07.1112098Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1112186Z outputs = self.mobilebert( 2025-12-04T08:56:07.1112542Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1112622Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1112961Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1113069Z layer_outputs = layer_module( 2025-12-04T08:56:07.1113392Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T08:56:07.1113572Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T08:56:07.1113910Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-12-04T08:56:07.1114058Z shared_attention_input = self.attention(hidden_states) 2025-12-04T08:56:07.1114383Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-12-04T08:56:07.1114480Z layer_input = self.dense(hidden_states) 2025-12-04T08:56:07.1114484Z 2025-12-04T08:56:07.1114598Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1114817Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1114896Z res = mod(**inputs) 2025-12-04T08:56:07.1115211Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1115300Z outputs = self.mobilebert( 2025-12-04T08:56:07.1115629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1115711Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1116028Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1116108Z layer_outputs = layer_module( 2025-12-04T08:56:07.1116417Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T08:56:07.1116521Z self_attention_outputs = self.attention( 2025-12-04T08:56:07.1116828Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T08:56:07.1116917Z self_outputs = self.self( 2025-12-04T08:56:07.1117225Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-12-04T08:56:07.1117308Z self.value(value_tensor) 2025-12-04T08:56:07.1117313Z 2025-12-04T08:56:07.1117435Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1117654Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1117733Z res = mod(**inputs) 2025-12-04T08:56:07.1118113Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1118208Z outputs = self.mobilebert( 2025-12-04T08:56:07.1118534Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1118614Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1118927Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1119018Z layer_outputs = layer_module( 2025-12-04T08:56:07.1119331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T08:56:07.1119547Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T08:56:07.1119866Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-12-04T08:56:07.1119989Z bottlenecked_hidden_states = self.input(hidden_states) 2025-12-04T08:56:07.1120314Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-12-04T08:56:07.1120411Z layer_input = self.dense(hidden_states) 2025-12-04T08:56:07.1120415Z 2025-12-04T08:56:07.1120538Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1121017Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1121209Z res = mod(**inputs) 2025-12-04T08:56:07.1121560Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1121642Z outputs = self.mobilebert( 2025-12-04T08:56:07.1121954Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1122044Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1122354Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1122443Z layer_outputs = layer_module( 2025-12-04T08:56:07.1122756Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T08:56:07.1122962Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T08:56:07.1123290Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-12-04T08:56:07.1123416Z shared_attention_input = self.attention(hidden_states) 2025-12-04T08:56:07.1123740Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-12-04T08:56:07.1123838Z layer_input = self.LayerNorm(layer_input) 2025-12-04T08:56:07.1124157Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:07.1124263Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:07.1124268Z 2025-12-04T08:56:07.1124378Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1124599Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1124669Z res = mod(**inputs) 2025-12-04T08:56:07.1124978Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1125068Z outputs = self.mobilebert( 2025-12-04T08:56:07.1125375Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1125452Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1125766Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1125841Z layer_outputs = layer_module( 2025-12-04T08:56:07.1126157Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T08:56:07.1126252Z self_attention_outputs = self.attention( 2025-12-04T08:56:07.1126561Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T08:56:07.1126647Z self_outputs = self.self( 2025-12-04T08:56:07.1126983Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-12-04T08:56:07.1127069Z self.query(query_tensor) 2025-12-04T08:56:07.1127072Z 2025-12-04T08:56:07.1127184Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1127406Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1127481Z res = mod(**inputs) 2025-12-04T08:56:07.1127789Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1127864Z outputs = self.mobilebert( 2025-12-04T08:56:07.1128180Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1128733Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1129056Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1129134Z layer_outputs = layer_module( 2025-12-04T08:56:07.1129434Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T08:56:07.1129533Z self_attention_outputs = self.attention( 2025-12-04T08:56:07.1129837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T08:56:07.1129920Z self_outputs = self.self( 2025-12-04T08:56:07.1130225Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-12-04T08:56:07.1130321Z self.key(key_tensor) 2025-12-04T08:56:07.1130324Z 2025-12-04T08:56:07.1130420Z cudagraph partition due to non gpu ops 2025-12-04T08:56:07.1130508Z cudagraph partition due to non gpu ops 2025-12-04T08:56:07.1130623Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1130839Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1130908Z res = mod(**inputs) 2025-12-04T08:56:07.1131217Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1131293Z outputs = self.mobilebert( 2025-12-04T08:56:07.1131592Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1131680Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1131982Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1132060Z layer_outputs = layer_module( 2025-12-04T08:56:07.1132367Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T08:56:07.1132459Z self_attention_outputs = self.attention( 2025-12-04T08:56:07.1132767Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-12-04T08:56:07.1132899Z attention_output = self.output(self_outputs[0], layer_input) 2025-12-04T08:56:07.1133205Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-12-04T08:56:07.1133303Z layer_outputs = self.dense(hidden_states) 2025-12-04T08:56:07.1133306Z 2025-12-04T08:56:07.1133415Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1133636Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1133708Z res = mod(**inputs) 2025-12-04T08:56:07.1134028Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1134114Z outputs = self.mobilebert( 2025-12-04T08:56:07.1134424Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1134503Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1134816Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1134892Z layer_outputs = layer_module( 2025-12-04T08:56:07.1135203Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T08:56:07.1135295Z self_attention_outputs = self.attention( 2025-12-04T08:56:07.1135617Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-12-04T08:56:07.1135776Z attention_output = self.output(self_outputs[0], layer_input) 2025-12-04T08:56:07.1136084Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-12-04T08:56:07.1136231Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T08:56:07.1136532Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:07.1136628Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:07.1136632Z 2025-12-04T08:56:07.1136750Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1136983Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1137059Z res = mod(**inputs) 2025-12-04T08:56:07.1137366Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1137443Z outputs = self.mobilebert( 2025-12-04T08:56:07.1137754Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1137831Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1138142Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1138229Z layer_outputs = layer_module( 2025-12-04T08:56:07.1138529Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.1138640Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.1138954Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T08:56:07.1139077Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T08:56:07.1139389Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T08:56:07.1139479Z hidden_states = self.dense(hidden_states) 2025-12-04T08:56:07.1139483Z 2025-12-04T08:56:07.1139599Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1139812Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1139883Z res = mod(**inputs) 2025-12-04T08:56:07.1140191Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1140269Z outputs = self.mobilebert( 2025-12-04T08:56:07.1140572Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1140657Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1140974Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1141058Z layer_outputs = layer_module( 2025-12-04T08:56:07.1141358Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.1141458Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.1141769Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T08:56:07.1141887Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T08:56:07.1142214Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T08:56:07.1142354Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:56:07.1142358Z 2025-12-04T08:56:07.1142469Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1142690Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1142760Z res = mod(**inputs) 2025-12-04T08:56:07.1143064Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1143151Z outputs = self.mobilebert( 2025-12-04T08:56:07.1143454Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1143538Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1143859Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1143938Z layer_outputs = layer_module( 2025-12-04T08:56:07.1144248Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.1144351Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.1144656Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T08:56:07.1144799Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T08:56:07.1145084Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T08:56:07.1145177Z layer_outputs = self.dense(hidden_states) 2025-12-04T08:56:07.1145180Z 2025-12-04T08:56:07.1145284Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1145497Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1145567Z res = mod(**inputs) 2025-12-04T08:56:07.1145871Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1145952Z outputs = self.mobilebert( 2025-12-04T08:56:07.1146260Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1146339Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1146647Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1146724Z layer_outputs = layer_module( 2025-12-04T08:56:07.1147037Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.1147139Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.1147441Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T08:56:07.1147607Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T08:56:07.1147913Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T08:56:07.1148053Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T08:56:07.1148377Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:07.1148474Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:07.1148477Z 2025-12-04T08:56:07.1148597Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1148812Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1148899Z res = mod(**inputs) 2025-12-04T08:56:07.1149237Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1149312Z outputs = self.mobilebert( 2025-12-04T08:56:07.1149623Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1149702Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1150028Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1150112Z layer_outputs = layer_module( 2025-12-04T08:56:07.1150427Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.1150554Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.1150870Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T08:56:07.1150992Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T08:56:07.1151305Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T08:56:07.1151389Z hidden_states = self.dense(hidden_states) 2025-12-04T08:56:07.1151393Z 2025-12-04T08:56:07.1151495Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1151710Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1151776Z res = mod(**inputs) 2025-12-04T08:56:07.1152094Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1152170Z outputs = self.mobilebert( 2025-12-04T08:56:07.1152473Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1152560Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1152863Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1152945Z layer_outputs = layer_module( 2025-12-04T08:56:07.1153246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.1153346Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.1153660Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T08:56:07.1153780Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T08:56:07.1154092Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T08:56:07.1154218Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:56:07.1154221Z 2025-12-04T08:56:07.1154360Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1154580Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1154650Z res = mod(**inputs) 2025-12-04T08:56:07.1154954Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1155038Z outputs = self.mobilebert( 2025-12-04T08:56:07.1155341Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1155426Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1155760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1155851Z layer_outputs = layer_module( 2025-12-04T08:56:07.1156162Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.1156265Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.1156571Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T08:56:07.1156702Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T08:56:07.1157003Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T08:56:07.1157100Z layer_outputs = self.dense(hidden_states) 2025-12-04T08:56:07.1157122Z 2025-12-04T08:56:07.1157231Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1157445Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1157521Z res = mod(**inputs) 2025-12-04T08:56:07.1157828Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1157913Z outputs = self.mobilebert( 2025-12-04T08:56:07.1158350Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1158438Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1158755Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1158836Z layer_outputs = layer_module( 2025-12-04T08:56:07.1159165Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.1159270Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.1159576Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T08:56:07.1159716Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T08:56:07.1160016Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T08:56:07.1160148Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T08:56:07.1160457Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:07.1160556Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:07.1160560Z 2025-12-04T08:56:07.1160680Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1160896Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1160967Z res = mod(**inputs) 2025-12-04T08:56:07.1161304Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1161381Z outputs = self.mobilebert( 2025-12-04T08:56:07.1161689Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1161765Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1162062Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1162146Z layer_outputs = layer_module( 2025-12-04T08:56:07.1162449Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.1162550Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.1162876Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T08:56:07.1163017Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T08:56:07.1163330Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T08:56:07.1163417Z hidden_states = self.dense(hidden_states) 2025-12-04T08:56:07.1163420Z 2025-12-04T08:56:07.1163528Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1163748Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1163816Z res = mod(**inputs) 2025-12-04T08:56:07.1164120Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1164215Z outputs = self.mobilebert( 2025-12-04T08:56:07.1164518Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1164604Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1164905Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1164986Z layer_outputs = layer_module( 2025-12-04T08:56:07.1165295Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.1165395Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.1165707Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T08:56:07.1165826Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T08:56:07.1166127Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T08:56:07.1166257Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:56:07.1166262Z 2025-12-04T08:56:07.1166373Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1166591Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1166660Z res = mod(**inputs) 2025-12-04T08:56:07.1166976Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1167063Z outputs = self.mobilebert( 2025-12-04T08:56:07.1167375Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1167463Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1167766Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1167851Z layer_outputs = layer_module( 2025-12-04T08:56:07.1168325Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.1168473Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.1168778Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T08:56:07.1168921Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T08:56:07.1169229Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T08:56:07.1169325Z layer_outputs = self.dense(hidden_states) 2025-12-04T08:56:07.1169334Z 2025-12-04T08:56:07.1169445Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1169675Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1169770Z res = mod(**inputs) 2025-12-04T08:56:07.1170075Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1170157Z outputs = self.mobilebert( 2025-12-04T08:56:07.1170460Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1170539Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1170853Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1170928Z layer_outputs = layer_module( 2025-12-04T08:56:07.1171253Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.1171379Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.1171685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T08:56:07.1171830Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T08:56:07.1172150Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T08:56:07.1172348Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T08:56:07.1172911Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:07.1173053Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:07.1173057Z 2025-12-04T08:56:07.1173247Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1173583Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1173690Z res = mod(**inputs) 2025-12-04T08:56:07.1174150Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1174228Z outputs = self.mobilebert( 2025-12-04T08:56:07.1174523Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1174599Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1174898Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1174978Z layer_outputs = layer_module( 2025-12-04T08:56:07.1175262Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-12-04T08:56:07.1175388Z intermediate_output = self.intermediate(attention_output) 2025-12-04T08:56:07.1175705Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T08:56:07.1175792Z hidden_states = self.dense(hidden_states) 2025-12-04T08:56:07.1175796Z 2025-12-04T08:56:07.1175904Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1176105Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1176170Z res = mod(**inputs) 2025-12-04T08:56:07.1176461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1176534Z outputs = self.mobilebert( 2025-12-04T08:56:07.1176826Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1176902Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1177225Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1177308Z layer_outputs = layer_module( 2025-12-04T08:56:07.1177594Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-12-04T08:56:07.1177714Z intermediate_output = self.intermediate(attention_output) 2025-12-04T08:56:07.1178005Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T08:56:07.1178118Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:56:07.1178122Z 2025-12-04T08:56:07.1178232Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1178515Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1178581Z res = mod(**inputs) 2025-12-04T08:56:07.1178879Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1178953Z outputs = self.mobilebert( 2025-12-04T08:56:07.1179246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1179320Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1179607Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1179688Z layer_outputs = layer_module( 2025-12-04T08:56:07.1179986Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T08:56:07.1180152Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T08:56:07.1180449Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-12-04T08:56:07.1180550Z layer_output = self.dense(intermediate_states) 2025-12-04T08:56:07.1180553Z 2025-12-04T08:56:07.1180665Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1180866Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1180931Z res = mod(**inputs) 2025-12-04T08:56:07.1181224Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1181297Z outputs = self.mobilebert( 2025-12-04T08:56:07.1181593Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1181671Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1181957Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1182040Z layer_outputs = layer_module( 2025-12-04T08:56:07.1182341Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T08:56:07.1182512Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T08:56:07.1182799Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-12-04T08:56:07.1182922Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-12-04T08:56:07.1183222Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:07.1183315Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:07.1183319Z 2025-12-04T08:56:07.1183434Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1183655Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1183723Z res = mod(**inputs) 2025-12-04T08:56:07.1184017Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1184089Z outputs = self.mobilebert( 2025-12-04T08:56:07.1184375Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1184453Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1184745Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1184845Z layer_outputs = layer_module( 2025-12-04T08:56:07.1185146Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T08:56:07.1185321Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T08:56:07.1185623Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-12-04T08:56:07.1185748Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-12-04T08:56:07.1186045Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-12-04T08:56:07.1186142Z layer_outputs = self.dense(hidden_states) 2025-12-04T08:56:07.1186146Z 2025-12-04T08:56:07.1186254Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1186485Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1186552Z res = mod(**inputs) 2025-12-04T08:56:07.1186839Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1186920Z outputs = self.mobilebert( 2025-12-04T08:56:07.1187204Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1187286Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1187573Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1187645Z layer_outputs = layer_module( 2025-12-04T08:56:07.1187938Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T08:56:07.1188097Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T08:56:07.1188390Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-12-04T08:56:07.1188516Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-12-04T08:56:07.1188816Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-12-04T08:56:07.1188948Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T08:56:07.1189235Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:07.1189330Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:07.1189342Z 2025-12-04T08:56:07.1189446Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1189650Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1189725Z res = mod(**inputs) 2025-12-04T08:56:07.1190026Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1190143Z outputs = self.mobilebert( 2025-12-04T08:56:07.1190643Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1190734Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1191030Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1191102Z layer_outputs = layer_module( 2025-12-04T08:56:07.1191388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T08:56:07.1191562Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T08:56:07.1191872Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-12-04T08:56:07.1191992Z shared_attention_input = self.attention(hidden_states) 2025-12-04T08:56:07.1192278Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-12-04T08:56:07.1192364Z layer_input = self.dense(hidden_states) 2025-12-04T08:56:07.1192367Z 2025-12-04T08:56:07.1192480Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1192677Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1192740Z res = mod(**inputs) 2025-12-04T08:56:07.1193033Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1193103Z outputs = self.mobilebert( 2025-12-04T08:56:07.1193397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1193471Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1193761Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1193845Z layer_outputs = layer_module( 2025-12-04T08:56:07.1194145Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T08:56:07.1194244Z self_attention_outputs = self.attention( 2025-12-04T08:56:07.1194544Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T08:56:07.1194621Z self_outputs = self.self( 2025-12-04T08:56:07.1194934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-12-04T08:56:07.1195013Z self.value(value_tensor) 2025-12-04T08:56:07.1195019Z 2025-12-04T08:56:07.1195129Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1195379Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1195450Z res = mod(**inputs) 2025-12-04T08:56:07.1195763Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1195837Z outputs = self.mobilebert( 2025-12-04T08:56:07.1196150Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1196237Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1196549Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1196635Z layer_outputs = layer_module( 2025-12-04T08:56:07.1196980Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T08:56:07.1197178Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T08:56:07.1197507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-12-04T08:56:07.1197629Z bottlenecked_hidden_states = self.input(hidden_states) 2025-12-04T08:56:07.1197950Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-12-04T08:56:07.1198129Z layer_input = self.dense(hidden_states) 2025-12-04T08:56:07.1198137Z 2025-12-04T08:56:07.1198258Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1198515Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1198590Z res = mod(**inputs) 2025-12-04T08:56:07.1198915Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1199004Z outputs = self.mobilebert( 2025-12-04T08:56:07.1199327Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1199417Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1199738Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1199815Z layer_outputs = layer_module( 2025-12-04T08:56:07.1200128Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T08:56:07.1200300Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T08:56:07.1200614Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-12-04T08:56:07.1200743Z shared_attention_input = self.attention(hidden_states) 2025-12-04T08:56:07.1201044Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-12-04T08:56:07.1201146Z layer_input = self.LayerNorm(layer_input) 2025-12-04T08:56:07.1201465Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:07.1201562Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:07.1201565Z 2025-12-04T08:56:07.1201684Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1201909Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1201987Z res = mod(**inputs) 2025-12-04T08:56:07.1202301Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1202398Z outputs = self.mobilebert( 2025-12-04T08:56:07.1202706Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1202785Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1203110Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1203188Z layer_outputs = layer_module( 2025-12-04T08:56:07.1203504Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T08:56:07.1203605Z self_attention_outputs = self.attention( 2025-12-04T08:56:07.1203946Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T08:56:07.1204063Z self_outputs = self.self( 2025-12-04T08:56:07.1204396Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-12-04T08:56:07.1204475Z self.query(query_tensor) 2025-12-04T08:56:07.1204479Z 2025-12-04T08:56:07.1204595Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1204818Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1204886Z res = mod(**inputs) 2025-12-04T08:56:07.1205196Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1205271Z outputs = self.mobilebert( 2025-12-04T08:56:07.1205586Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1205690Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1205995Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1206077Z layer_outputs = layer_module( 2025-12-04T08:56:07.1206378Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T08:56:07.1206470Z self_attention_outputs = self.attention( 2025-12-04T08:56:07.1206788Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T08:56:07.1206864Z self_outputs = self.self( 2025-12-04T08:56:07.1207180Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-12-04T08:56:07.1207256Z self.key(key_tensor) 2025-12-04T08:56:07.1207260Z 2025-12-04T08:56:07.1207349Z cudagraph partition due to non gpu ops 2025-12-04T08:56:07.1207447Z cudagraph partition due to non gpu ops 2025-12-04T08:56:07.1207558Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1207766Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1207842Z res = mod(**inputs) 2025-12-04T08:56:07.1208143Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1208225Z outputs = self.mobilebert( 2025-12-04T08:56:07.1208536Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1208613Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1208920Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1208998Z layer_outputs = layer_module( 2025-12-04T08:56:07.1209324Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T08:56:07.1209416Z self_attention_outputs = self.attention( 2025-12-04T08:56:07.1209742Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-12-04T08:56:07.1209888Z attention_output = self.output(self_outputs[0], layer_input) 2025-12-04T08:56:07.1210245Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-12-04T08:56:07.1210348Z layer_outputs = self.dense(hidden_states) 2025-12-04T08:56:07.1210359Z 2025-12-04T08:56:07.1210471Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1210690Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1210803Z res = mod(**inputs) 2025-12-04T08:56:07.1211107Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1211185Z outputs = self.mobilebert( 2025-12-04T08:56:07.1211497Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1211574Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1211883Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1211957Z layer_outputs = layer_module( 2025-12-04T08:56:07.1212259Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T08:56:07.1212374Z self_attention_outputs = self.attention( 2025-12-04T08:56:07.1212677Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-12-04T08:56:07.1212810Z attention_output = self.output(self_outputs[0], layer_input) 2025-12-04T08:56:07.1213115Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-12-04T08:56:07.1213250Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T08:56:07.1213559Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:07.1213651Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:07.1213654Z 2025-12-04T08:56:07.1213757Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1213963Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1214029Z res = mod(**inputs) 2025-12-04T08:56:07.1214325Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1214399Z outputs = self.mobilebert( 2025-12-04T08:56:07.1214685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1214767Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1215052Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1215123Z layer_outputs = layer_module( 2025-12-04T08:56:07.1215418Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.1215516Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.1215809Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T08:56:07.1215925Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T08:56:07.1216226Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T08:56:07.1216318Z hidden_states = self.dense(hidden_states) 2025-12-04T08:56:07.1216322Z 2025-12-04T08:56:07.1216424Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1216633Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1216699Z res = mod(**inputs) 2025-12-04T08:56:07.1216987Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1217069Z outputs = self.mobilebert( 2025-12-04T08:56:07.1217508Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1217610Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1217905Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1217979Z layer_outputs = layer_module( 2025-12-04T08:56:07.1218290Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.1218429Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.1218868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T08:56:07.1219051Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T08:56:07.1219523Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T08:56:07.1219718Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:56:07.1219724Z 2025-12-04T08:56:07.1219886Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1220214Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1220316Z res = mod(**inputs) 2025-12-04T08:56:07.1220953Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1221080Z outputs = self.mobilebert( 2025-12-04T08:56:07.1221602Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1221716Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1222236Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1222349Z layer_outputs = layer_module( 2025-12-04T08:56:07.1222854Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.1222999Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.1223306Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T08:56:07.1223450Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T08:56:07.1223753Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T08:56:07.1223844Z layer_outputs = self.dense(hidden_states) 2025-12-04T08:56:07.1223850Z 2025-12-04T08:56:07.1223978Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1224183Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1224258Z res = mod(**inputs) 2025-12-04T08:56:07.1224645Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1224720Z outputs = self.mobilebert( 2025-12-04T08:56:07.1225016Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1225090Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1225392Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1225477Z layer_outputs = layer_module( 2025-12-04T08:56:07.1225776Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.1225895Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.1226214Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T08:56:07.1226373Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T08:56:07.1226683Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T08:56:07.1226812Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T08:56:07.1227104Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:07.1227196Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:07.1227200Z 2025-12-04T08:56:07.1227305Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1227541Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1227609Z res = mod(**inputs) 2025-12-04T08:56:07.1227995Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1228070Z outputs = self.mobilebert( 2025-12-04T08:56:07.1228356Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1228437Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1228724Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1228795Z layer_outputs = layer_module( 2025-12-04T08:56:07.1229092Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.1229190Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.1229480Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T08:56:07.1229595Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T08:56:07.1229882Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T08:56:07.1229973Z hidden_states = self.dense(hidden_states) 2025-12-04T08:56:07.1229977Z 2025-12-04T08:56:07.1230079Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1230285Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1230350Z res = mod(**inputs) 2025-12-04T08:56:07.1230637Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1230719Z outputs = self.mobilebert( 2025-12-04T08:56:07.1231003Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1231079Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1231388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1231462Z layer_outputs = layer_module( 2025-12-04T08:56:07.1231753Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.1231849Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.1232132Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T08:56:07.1232252Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T08:56:07.1232569Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T08:56:07.1232707Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:56:07.1232711Z 2025-12-04T08:56:07.1232815Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1233016Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1233089Z res = mod(**inputs) 2025-12-04T08:56:07.1233373Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1233446Z outputs = self.mobilebert( 2025-12-04T08:56:07.1233869Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1233968Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1234324Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1234398Z layer_outputs = layer_module( 2025-12-04T08:56:07.1234684Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.1234786Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.1235088Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T08:56:07.1235229Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T08:56:07.1235539Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T08:56:07.1235629Z layer_outputs = self.dense(hidden_states) 2025-12-04T08:56:07.1235633Z 2025-12-04T08:56:07.1235751Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1235982Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1236384Z res = mod(**inputs) 2025-12-04T08:56:07.1236823Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1237290Z outputs = self.mobilebert( 2025-12-04T08:56:07.1237747Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1238342Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1238811Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1239291Z layer_outputs = layer_module( 2025-12-04T08:56:07.1239740Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.1240259Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.1240764Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T08:56:07.1241281Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T08:56:07.1241803Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T08:56:07.1242314Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T08:56:07.1242828Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:07.1243352Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:07.1243599Z 2025-12-04T08:56:07.1243717Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1244161Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1244532Z res = mod(**inputs) 2025-12-04T08:56:07.1245020Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1245481Z outputs = self.mobilebert( 2025-12-04T08:56:07.1245915Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1246369Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1246823Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1247274Z layer_outputs = layer_module( 2025-12-04T08:56:07.1247718Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.1248222Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.1248765Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T08:56:07.1249268Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T08:56:07.1249759Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T08:56:07.1250228Z hidden_states = self.dense(hidden_states) 2025-12-04T08:56:07.1250384Z 2025-12-04T08:56:07.1250507Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1250905Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1251262Z res = mod(**inputs) 2025-12-04T08:56:07.1251696Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1252146Z outputs = self.mobilebert( 2025-12-04T08:56:07.1252586Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1253023Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1253432Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1253854Z layer_outputs = layer_module( 2025-12-04T08:56:07.1254251Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.1254687Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.1255121Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T08:56:07.1255577Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T08:56:07.1256023Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T08:56:07.1256483Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:56:07.1256670Z 2025-12-04T08:56:07.1256783Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1257143Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1257468Z res = mod(**inputs) 2025-12-04T08:56:07.1257854Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1258270Z outputs = self.mobilebert( 2025-12-04T08:56:07.1258668Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1259088Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1259520Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1259949Z layer_outputs = layer_module( 2025-12-04T08:56:07.1260369Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.1260825Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.1261261Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T08:56:07.1261730Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T08:56:07.1262200Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T08:56:07.1262650Z layer_outputs = self.dense(hidden_states) 2025-12-04T08:56:07.1262812Z 2025-12-04T08:56:07.1262923Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1263277Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1263613Z res = mod(**inputs) 2025-12-04T08:56:07.1264023Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1264487Z outputs = self.mobilebert( 2025-12-04T08:56:07.1264897Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1265329Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1265750Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1266171Z layer_outputs = layer_module( 2025-12-04T08:56:07.1266605Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.1267082Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.1267559Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T08:56:07.1268060Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T08:56:07.1268568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T08:56:07.1269079Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T08:56:07.1269585Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:07.1270057Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:07.1270225Z 2025-12-04T08:56:07.1270346Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1270749Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1271101Z res = mod(**inputs) 2025-12-04T08:56:07.1271554Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1272012Z outputs = self.mobilebert( 2025-12-04T08:56:07.1272464Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1272916Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1273366Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1273820Z layer_outputs = layer_module( 2025-12-04T08:56:07.1274267Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-12-04T08:56:07.1274789Z intermediate_output = self.intermediate(attention_output) 2025-12-04T08:56:07.1275312Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T08:56:07.1275782Z hidden_states = self.dense(hidden_states) 2025-12-04T08:56:07.1275942Z 2025-12-04T08:56:07.1276056Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1276454Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1276813Z res = mod(**inputs) 2025-12-04T08:56:07.1277247Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1277694Z outputs = self.mobilebert( 2025-12-04T08:56:07.1278237Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1278748Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1279211Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1279663Z layer_outputs = layer_module( 2025-12-04T08:56:07.1280119Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-12-04T08:56:07.1280632Z intermediate_output = self.intermediate(attention_output) 2025-12-04T08:56:07.1281141Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T08:56:07.1281672Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:56:07.1281866Z 2025-12-04T08:56:07.1281986Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1282385Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1282747Z res = mod(**inputs) 2025-12-04T08:56:07.1283174Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1283644Z outputs = self.mobilebert( 2025-12-04T08:56:07.1284073Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1284530Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1284977Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1285436Z layer_outputs = layer_module( 2025-12-04T08:56:07.1285854Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T08:56:07.1286406Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T08:56:07.1286947Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-12-04T08:56:07.1287450Z layer_output = self.dense(intermediate_states) 2025-12-04T08:56:07.1287610Z 2025-12-04T08:56:07.1287719Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1288112Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1288467Z res = mod(**inputs) 2025-12-04T08:56:07.1288883Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1289337Z outputs = self.mobilebert( 2025-12-04T08:56:07.1289778Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1290210Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1290661Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1291108Z layer_outputs = layer_module( 2025-12-04T08:56:07.1291527Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T08:56:07.1303679Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T08:56:07.1304297Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-12-04T08:56:07.1304805Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-12-04T08:56:07.1305304Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:07.1305879Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:07.1306049Z 2025-12-04T08:56:07.1306168Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1306550Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1306891Z res = mod(**inputs) 2025-12-04T08:56:07.1307286Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1307717Z outputs = self.mobilebert( 2025-12-04T08:56:07.1308131Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1308550Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1308959Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1309376Z layer_outputs = layer_module( 2025-12-04T08:56:07.1309790Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T08:56:07.1310293Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T08:56:07.1310790Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-12-04T08:56:07.1311257Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-12-04T08:56:07.1311724Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-12-04T08:56:07.1312159Z layer_outputs = self.dense(hidden_states) 2025-12-04T08:56:07.1312303Z 2025-12-04T08:56:07.1312413Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1312785Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1313125Z res = mod(**inputs) 2025-12-04T08:56:07.1313538Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1313995Z outputs = self.mobilebert( 2025-12-04T08:56:07.1314463Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1314923Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1315365Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1315819Z layer_outputs = layer_module( 2025-12-04T08:56:07.1316265Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T08:56:07.1316809Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T08:56:07.1317367Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-12-04T08:56:07.1317896Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-12-04T08:56:07.1318546Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-12-04T08:56:07.1319081Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T08:56:07.1319590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:07.1320072Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:07.1320237Z 2025-12-04T08:56:07.1320374Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1320961Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1321409Z res = mod(**inputs) 2025-12-04T08:56:07.1321844Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1322307Z outputs = self.mobilebert( 2025-12-04T08:56:07.1322750Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1323208Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1323658Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1324101Z layer_outputs = layer_module( 2025-12-04T08:56:07.1324549Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T08:56:07.1325102Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T08:56:07.1325650Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-12-04T08:56:07.1326143Z shared_attention_input = self.attention(hidden_states) 2025-12-04T08:56:07.1326631Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-12-04T08:56:07.1327096Z layer_input = self.dense(hidden_states) 2025-12-04T08:56:07.1327249Z 2025-12-04T08:56:07.1327372Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1327760Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1328120Z res = mod(**inputs) 2025-12-04T08:56:07.1328544Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1328996Z outputs = self.mobilebert( 2025-12-04T08:56:07.1329430Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1329882Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1330353Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1330775Z layer_outputs = layer_module( 2025-12-04T08:56:07.1331197Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T08:56:07.1331637Z self_attention_outputs = self.attention( 2025-12-04T08:56:07.1332077Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T08:56:07.1332496Z self_outputs = self.self( 2025-12-04T08:56:07.1332908Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-12-04T08:56:07.1333333Z self.value(value_tensor) 2025-12-04T08:56:07.1333506Z 2025-12-04T08:56:07.1333623Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1333998Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1334333Z res = mod(**inputs) 2025-12-04T08:56:07.1334734Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1335154Z outputs = self.mobilebert( 2025-12-04T08:56:07.1335568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1335996Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1336420Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1336860Z layer_outputs = layer_module( 2025-12-04T08:56:07.1337284Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T08:56:07.1337803Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T08:56:07.1338315Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-12-04T08:56:07.1338786Z bottlenecked_hidden_states = self.input(hidden_states) 2025-12-04T08:56:07.1339250Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-12-04T08:56:07.1339685Z layer_input = self.dense(hidden_states) 2025-12-04T08:56:07.1339825Z 2025-12-04T08:56:07.1339933Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1340306Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1340646Z res = mod(**inputs) 2025-12-04T08:56:07.1341048Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1341469Z outputs = self.mobilebert( 2025-12-04T08:56:07.1341883Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1342317Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1342734Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1343163Z layer_outputs = layer_module( 2025-12-04T08:56:07.1343582Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T08:56:07.1344103Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T08:56:07.1344618Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-12-04T08:56:07.1345106Z shared_attention_input = self.attention(hidden_states) 2025-12-04T08:56:07.1345584Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-12-04T08:56:07.1346023Z layer_input = self.LayerNorm(layer_input) 2025-12-04T08:56:07.1346462Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:07.1346897Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:07.1347045Z 2025-12-04T08:56:07.1347155Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1347511Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1347837Z res = mod(**inputs) 2025-12-04T08:56:07.1348240Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1348668Z outputs = self.mobilebert( 2025-12-04T08:56:07.1349064Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1349476Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1349880Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1350284Z layer_outputs = layer_module( 2025-12-04T08:56:07.1350690Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T08:56:07.1351120Z self_attention_outputs = self.attention( 2025-12-04T08:56:07.1351581Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T08:56:07.1351987Z self_outputs = self.self( 2025-12-04T08:56:07.1352386Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-12-04T08:56:07.1352800Z self.query(query_tensor) 2025-12-04T08:56:07.1352915Z 2025-12-04T08:56:07.1353025Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1353380Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1353714Z res = mod(**inputs) 2025-12-04T08:56:07.1354109Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1354528Z outputs = self.mobilebert( 2025-12-04T08:56:07.1354961Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1355430Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1355881Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1356322Z layer_outputs = layer_module( 2025-12-04T08:56:07.1356768Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T08:56:07.1357232Z self_attention_outputs = self.attention( 2025-12-04T08:56:07.1357692Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T08:56:07.1358244Z self_outputs = self.self( 2025-12-04T08:56:07.1358706Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-12-04T08:56:07.1359236Z self.key(key_tensor) 2025-12-04T08:56:07.1359357Z 2025-12-04T08:56:07.1359448Z cudagraph partition due to non gpu ops 2025-12-04T08:56:07.1359691Z cudagraph partition due to non gpu ops 2025-12-04T08:56:07.1359967Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1360356Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1360693Z res = mod(**inputs) 2025-12-04T08:56:07.1361093Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1361519Z outputs = self.mobilebert( 2025-12-04T08:56:07.1361922Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1362352Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1362778Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1363232Z layer_outputs = layer_module( 2025-12-04T08:56:07.1363750Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T08:56:07.1364227Z self_attention_outputs = self.attention( 2025-12-04T08:56:07.1364690Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-12-04T08:56:07.1365167Z attention_output = self.output(self_outputs[0], layer_input) 2025-12-04T08:56:07.1365654Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-12-04T08:56:07.1366105Z layer_outputs = self.dense(hidden_states) 2025-12-04T08:56:07.1366254Z 2025-12-04T08:56:07.1366375Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1366755Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1367091Z res = mod(**inputs) 2025-12-04T08:56:07.1367493Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1367916Z outputs = self.mobilebert( 2025-12-04T08:56:07.1368321Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1368749Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1369167Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1369605Z layer_outputs = layer_module( 2025-12-04T08:56:07.1370052Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T08:56:07.1370528Z self_attention_outputs = self.attention( 2025-12-04T08:56:07.1370980Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-12-04T08:56:07.1371463Z attention_output = self.output(self_outputs[0], layer_input) 2025-12-04T08:56:07.1371935Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-12-04T08:56:07.1372417Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T08:56:07.1372897Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:07.1373337Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:07.1373497Z 2025-12-04T08:56:07.1373603Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1373998Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1374343Z res = mod(**inputs) 2025-12-04T08:56:07.1374775Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1375245Z outputs = self.mobilebert( 2025-12-04T08:56:07.1375682Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1376122Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1376541Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1376966Z layer_outputs = layer_module( 2025-12-04T08:56:07.1377381Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.1377822Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.1378287Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T08:56:07.1378770Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T08:56:07.1379226Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T08:56:07.1379662Z hidden_states = self.dense(hidden_states) 2025-12-04T08:56:07.1379812Z 2025-12-04T08:56:07.1379918Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1380284Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1380607Z res = mod(**inputs) 2025-12-04T08:56:07.1381000Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1381442Z outputs = self.mobilebert( 2025-12-04T08:56:07.1381857Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1382283Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1382705Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1383144Z layer_outputs = layer_module( 2025-12-04T08:56:07.1383553Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.1384020Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.1384485Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T08:56:07.1384954Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T08:56:07.1385411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T08:56:07.1385886Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:56:07.1386063Z 2025-12-04T08:56:07.1386173Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1386539Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1386876Z res = mod(**inputs) 2025-12-04T08:56:07.1387261Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1387674Z outputs = self.mobilebert( 2025-12-04T08:56:07.1388068Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1388478Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1388883Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1389317Z layer_outputs = layer_module( 2025-12-04T08:56:07.1389736Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.1390174Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.1390611Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T08:56:07.1391074Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T08:56:07.1391534Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T08:56:07.1391958Z layer_outputs = self.dense(hidden_states) 2025-12-04T08:56:07.1392096Z 2025-12-04T08:56:07.1392209Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1392578Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1392938Z res = mod(**inputs) 2025-12-04T08:56:07.1393328Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1393744Z outputs = self.mobilebert( 2025-12-04T08:56:07.1394143Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1394565Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1394979Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1395404Z layer_outputs = layer_module( 2025-12-04T08:56:07.1395819Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.1396290Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.1396743Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T08:56:07.1397232Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T08:56:07.1397735Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T08:56:07.1398479Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T08:56:07.1399022Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:07.1399506Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:07.1399684Z 2025-12-04T08:56:07.1399793Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1400183Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1400519Z res = mod(**inputs) 2025-12-04T08:56:07.1400906Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1401325Z outputs = self.mobilebert( 2025-12-04T08:56:07.1401731Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1402164Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1402569Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1402989Z layer_outputs = layer_module( 2025-12-04T08:56:07.1403403Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.1403841Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.1404294Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T08:56:07.1404798Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T08:56:07.1405262Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T08:56:07.1405701Z hidden_states = self.dense(hidden_states) 2025-12-04T08:56:07.1405847Z 2025-12-04T08:56:07.1405952Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1406319Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1406654Z res = mod(**inputs) 2025-12-04T08:56:07.1407041Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1407473Z outputs = self.mobilebert( 2025-12-04T08:56:07.1407902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1408353Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1408778Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1409205Z layer_outputs = layer_module( 2025-12-04T08:56:07.1409622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.1410066Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.1410513Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T08:56:07.1410987Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T08:56:07.1411471Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T08:56:07.1411934Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:56:07.1412111Z 2025-12-04T08:56:07.1412221Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1412595Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1412918Z res = mod(**inputs) 2025-12-04T08:56:07.1413319Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1413741Z outputs = self.mobilebert( 2025-12-04T08:56:07.1414152Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1414576Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1415002Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1415435Z layer_outputs = layer_module( 2025-12-04T08:56:07.1415850Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.1416305Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.1416759Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T08:56:07.1417229Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T08:56:07.1417690Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T08:56:07.1418122Z layer_outputs = self.dense(hidden_states) 2025-12-04T08:56:07.1418269Z 2025-12-04T08:56:07.1418375Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1418736Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1419052Z res = mod(**inputs) 2025-12-04T08:56:07.1419462Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1419885Z outputs = self.mobilebert( 2025-12-04T08:56:07.1420299Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1420958Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1421504Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1421934Z layer_outputs = layer_module( 2025-12-04T08:56:07.1422366Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.1422975Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.1423451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T08:56:07.1423931Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T08:56:07.1424404Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T08:56:07.1424883Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T08:56:07.1425357Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:07.1425808Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:07.1425964Z 2025-12-04T08:56:07.1426072Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1426476Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1426811Z res = mod(**inputs) 2025-12-04T08:56:07.1427210Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1427630Z outputs = self.mobilebert( 2025-12-04T08:56:07.1428064Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1428520Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1428930Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1429360Z layer_outputs = layer_module( 2025-12-04T08:56:07.1429775Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.1430227Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.1430669Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T08:56:07.1431144Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T08:56:07.1431597Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T08:56:07.1432041Z hidden_states = self.dense(hidden_states) 2025-12-04T08:56:07.1432187Z 2025-12-04T08:56:07.1432294Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1432663Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1433000Z res = mod(**inputs) 2025-12-04T08:56:07.1433392Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1433823Z outputs = self.mobilebert( 2025-12-04T08:56:07.1434263Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1434744Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1435184Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1435637Z layer_outputs = layer_module( 2025-12-04T08:56:07.1436085Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.1436556Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.1437028Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T08:56:07.1437526Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T08:56:07.1438082Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T08:56:07.1438610Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:56:07.1438802Z 2025-12-04T08:56:07.1438921Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1439324Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1439686Z res = mod(**inputs) 2025-12-04T08:56:07.1440123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1440579Z outputs = self.mobilebert( 2025-12-04T08:56:07.1441004Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1441450Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1441854Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1442277Z layer_outputs = layer_module( 2025-12-04T08:56:07.1442698Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.1443143Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.1443591Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T08:56:07.1444071Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T08:56:07.1444550Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T08:56:07.1444981Z layer_outputs = self.dense(hidden_states) 2025-12-04T08:56:07.1445134Z 2025-12-04T08:56:07.1445240Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1445619Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1445947Z res = mod(**inputs) 2025-12-04T08:56:07.1446336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1446759Z outputs = self.mobilebert( 2025-12-04T08:56:07.1447177Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1447594Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1448015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1448440Z layer_outputs = layer_module( 2025-12-04T08:56:07.1448857Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.1449308Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.1449784Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T08:56:07.1450263Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T08:56:07.1450739Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T08:56:07.1451209Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T08:56:07.1451687Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:07.1452152Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:07.1452311Z 2025-12-04T08:56:07.1452434Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1452840Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1453219Z res = mod(**inputs) 2025-12-04T08:56:07.1453642Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1454085Z outputs = self.mobilebert( 2025-12-04T08:56:07.1454499Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1454928Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1455345Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1455760Z layer_outputs = layer_module( 2025-12-04T08:56:07.1456179Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-12-04T08:56:07.1456679Z intermediate_output = self.intermediate(attention_output) 2025-12-04T08:56:07.1457147Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T08:56:07.1457600Z hidden_states = self.dense(hidden_states) 2025-12-04T08:56:07.1457757Z 2025-12-04T08:56:07.1457868Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1458259Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1458604Z res = mod(**inputs) 2025-12-04T08:56:07.1459032Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1459490Z outputs = self.mobilebert( 2025-12-04T08:56:07.1459935Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1460389Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1460840Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1461291Z layer_outputs = layer_module( 2025-12-04T08:56:07.1461728Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-12-04T08:56:07.1462196Z intermediate_output = self.intermediate(attention_output) 2025-12-04T08:56:07.1462677Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T08:56:07.1463143Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:56:07.1463147Z 2025-12-04T08:56:07.1463259Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1463479Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1463551Z res = mod(**inputs) 2025-12-04T08:56:07.1463862Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1463986Z outputs = self.mobilebert( 2025-12-04T08:56:07.1464295Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1464383Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1464691Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1464767Z layer_outputs = layer_module( 2025-12-04T08:56:07.1465090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T08:56:07.1465266Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T08:56:07.1465611Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-12-04T08:56:07.1465734Z layer_output = self.dense(intermediate_states) 2025-12-04T08:56:07.1465738Z 2025-12-04T08:56:07.1465851Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1466074Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1466145Z res = mod(**inputs) 2025-12-04T08:56:07.1466446Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1466532Z outputs = self.mobilebert( 2025-12-04T08:56:07.1466834Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1466939Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1467255Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1467337Z layer_outputs = layer_module( 2025-12-04T08:56:07.1467660Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T08:56:07.1467834Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T08:56:07.1468152Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-12-04T08:56:07.1468286Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-12-04T08:56:07.1468597Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:07.1468709Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:07.1468713Z 2025-12-04T08:56:07.1468828Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1469054Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1469124Z res = mod(**inputs) 2025-12-04T08:56:07.1469436Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1469519Z outputs = self.mobilebert( 2025-12-04T08:56:07.1469828Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1469908Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1470223Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1470298Z layer_outputs = layer_module( 2025-12-04T08:56:07.1470620Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T08:56:07.1470791Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T08:56:07.1471122Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-12-04T08:56:07.1471263Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-12-04T08:56:07.1471565Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-12-04T08:56:07.1471664Z layer_outputs = self.dense(hidden_states) 2025-12-04T08:56:07.1471669Z 2025-12-04T08:56:07.1471779Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1471993Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1472073Z res = mod(**inputs) 2025-12-04T08:56:07.1472394Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1472489Z outputs = self.mobilebert( 2025-12-04T08:56:07.1472801Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1472879Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1473190Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1473267Z layer_outputs = layer_module( 2025-12-04T08:56:07.1473567Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T08:56:07.1473743Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T08:56:07.1474066Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-12-04T08:56:07.1474206Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-12-04T08:56:07.1474514Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-12-04T08:56:07.1474645Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T08:56:07.1474971Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:07.1475073Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:07.1475077Z 2025-12-04T08:56:07.1475195Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1475412Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1475485Z res = mod(**inputs) 2025-12-04T08:56:07.1475798Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1475877Z outputs = self.mobilebert( 2025-12-04T08:56:07.1476184Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1476271Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1476572Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1476656Z layer_outputs = layer_module( 2025-12-04T08:56:07.1476984Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T08:56:07.1477165Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T08:56:07.1477483Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-12-04T08:56:07.1477607Z shared_attention_input = self.attention(hidden_states) 2025-12-04T08:56:07.1477934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-12-04T08:56:07.1478087Z layer_input = self.dense(hidden_states) 2025-12-04T08:56:07.1478094Z 2025-12-04T08:56:07.1478215Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1478449Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1478520Z res = mod(**inputs) 2025-12-04T08:56:07.1478835Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1478927Z outputs = self.mobilebert( 2025-12-04T08:56:07.1479273Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1479385Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1479712Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1479791Z layer_outputs = layer_module( 2025-12-04T08:56:07.1480105Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T08:56:07.1480201Z self_attention_outputs = self.attention( 2025-12-04T08:56:07.1480511Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T08:56:07.1480591Z self_outputs = self.self( 2025-12-04T08:56:07.1480904Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-12-04T08:56:07.1481013Z self.value(value_tensor) 2025-12-04T08:56:07.1481019Z 2025-12-04T08:56:07.1481131Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1481345Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1481422Z res = mod(**inputs) 2025-12-04T08:56:07.1481720Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1481803Z outputs = self.mobilebert( 2025-12-04T08:56:07.1482115Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1482194Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1482502Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1482581Z layer_outputs = layer_module( 2025-12-04T08:56:07.1482893Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T08:56:07.1483060Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T08:56:07.1483346Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-12-04T08:56:07.1483467Z bottlenecked_hidden_states = self.input(hidden_states) 2025-12-04T08:56:07.1483756Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-12-04T08:56:07.1483840Z layer_input = self.dense(hidden_states) 2025-12-04T08:56:07.1483850Z 2025-12-04T08:56:07.1483954Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1484152Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1484227Z res = mod(**inputs) 2025-12-04T08:56:07.1484514Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1484606Z outputs = self.mobilebert( 2025-12-04T08:56:07.1484900Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1484974Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1485266Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1485338Z layer_outputs = layer_module( 2025-12-04T08:56:07.1485625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T08:56:07.1485797Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T08:56:07.1486098Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-12-04T08:56:07.1486238Z shared_attention_input = self.attention(hidden_states) 2025-12-04T08:56:07.1486527Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-12-04T08:56:07.1486617Z layer_input = self.LayerNorm(layer_input) 2025-12-04T08:56:07.1486906Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:07.1486998Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:07.1487002Z 2025-12-04T08:56:07.1487112Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1487331Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1487426Z res = mod(**inputs) 2025-12-04T08:56:07.1487742Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1487821Z outputs = self.mobilebert( 2025-12-04T08:56:07.1488130Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1488216Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1488520Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1488601Z layer_outputs = layer_module( 2025-12-04T08:56:07.1488907Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T08:56:07.1489000Z self_attention_outputs = self.attention( 2025-12-04T08:56:07.1489312Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T08:56:07.1489388Z self_outputs = self.self( 2025-12-04T08:56:07.1489677Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-12-04T08:56:07.1489758Z self.query(query_tensor) 2025-12-04T08:56:07.1489761Z 2025-12-04T08:56:07.1489866Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1490072Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1490139Z res = mod(**inputs) 2025-12-04T08:56:07.1490431Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1490510Z outputs = self.mobilebert( 2025-12-04T08:56:07.1490797Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1490882Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1491199Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1491273Z layer_outputs = layer_module( 2025-12-04T08:56:07.1491570Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T08:56:07.1491660Z self_attention_outputs = self.attention( 2025-12-04T08:56:07.1491949Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T08:56:07.1492028Z self_outputs = self.self( 2025-12-04T08:56:07.1492322Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-12-04T08:56:07.1492402Z self.key(key_tensor) 2025-12-04T08:56:07.1492405Z 2025-12-04T08:56:07.1492508Z cudagraph partition due to non gpu ops 2025-12-04T08:56:07.1492611Z cudagraph partition due to non gpu ops 2025-12-04T08:56:07.1492727Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1492937Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1493004Z res = mod(**inputs) 2025-12-04T08:56:07.1493302Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1493374Z outputs = self.mobilebert( 2025-12-04T08:56:07.1493665Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1493740Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1494025Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1494127Z layer_outputs = layer_module( 2025-12-04T08:56:07.1494419Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T08:56:07.1494513Z self_attention_outputs = self.attention( 2025-12-04T08:56:07.1494799Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-12-04T08:56:07.1494927Z attention_output = self.output(self_outputs[0], layer_input) 2025-12-04T08:56:07.1495220Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-12-04T08:56:07.1495306Z layer_outputs = self.dense(hidden_states) 2025-12-04T08:56:07.1495310Z 2025-12-04T08:56:07.1495423Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1495624Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1495692Z res = mod(**inputs) 2025-12-04T08:56:07.1495987Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1496060Z outputs = self.mobilebert( 2025-12-04T08:56:07.1496342Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1496423Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1496708Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1496789Z layer_outputs = layer_module( 2025-12-04T08:56:07.1497078Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T08:56:07.1497161Z self_attention_outputs = self.attention( 2025-12-04T08:56:07.1497445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-12-04T08:56:07.1497594Z attention_output = self.output(self_outputs[0], layer_input) 2025-12-04T08:56:07.1497879Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-12-04T08:56:07.1498014Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T08:56:07.1498306Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:07.1498403Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:07.1498407Z 2025-12-04T08:56:07.1498509Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1498707Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1498781Z res = mod(**inputs) 2025-12-04T08:56:07.1499081Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1499183Z outputs = self.mobilebert( 2025-12-04T08:56:07.1499479Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1499552Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1499844Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1499916Z layer_outputs = layer_module( 2025-12-04T08:56:07.1500226Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.1500323Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.1500628Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T08:56:07.1500752Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T08:56:07.1501040Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T08:56:07.1501124Z hidden_states = self.dense(hidden_states) 2025-12-04T08:56:07.1501136Z 2025-12-04T08:56:07.1501242Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1501445Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1501518Z res = mod(**inputs) 2025-12-04T08:56:07.1501805Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1501880Z outputs = self.mobilebert( 2025-12-04T08:56:07.1502176Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1502251Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1502545Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1502617Z layer_outputs = layer_module( 2025-12-04T08:56:07.1502900Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.1503002Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.1503292Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T08:56:07.1503404Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T08:56:07.1503699Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T08:56:07.1503818Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:56:07.1503821Z 2025-12-04T08:56:07.1503956Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1504159Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1504224Z res = mod(**inputs) 2025-12-04T08:56:07.1504516Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1504587Z outputs = self.mobilebert( 2025-12-04T08:56:07.1504879Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1504952Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1505239Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1505336Z layer_outputs = layer_module( 2025-12-04T08:56:07.1505638Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.1505735Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.1506027Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T08:56:07.1506153Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T08:56:07.1506448Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T08:56:07.1506531Z layer_outputs = self.dense(hidden_states) 2025-12-04T08:56:07.1506535Z 2025-12-04T08:56:07.1506637Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1506866Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1506934Z res = mod(**inputs) 2025-12-04T08:56:07.1507225Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1507297Z outputs = self.mobilebert( 2025-12-04T08:56:07.1507581Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1507663Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1507948Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1508027Z layer_outputs = layer_module( 2025-12-04T08:56:07.1508309Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.1508407Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.1508699Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T08:56:07.1508828Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T08:56:07.1509113Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T08:56:07.1509243Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T08:56:07.1509533Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:07.1509632Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:07.1509635Z 2025-12-04T08:56:07.1509739Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1509940Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1510015Z res = mod(**inputs) 2025-12-04T08:56:07.1510302Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1510407Z outputs = self.mobilebert( 2025-12-04T08:56:07.1510692Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1510767Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1511060Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1511132Z layer_outputs = layer_module( 2025-12-04T08:56:07.1511413Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.1511516Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.1511817Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T08:56:07.1511956Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T08:56:07.1512251Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T08:56:07.1512340Z hidden_states = self.dense(hidden_states) 2025-12-04T08:56:07.1512343Z 2025-12-04T08:56:07.1512460Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1512671Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1512746Z res = mod(**inputs) 2025-12-04T08:56:07.1513046Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1513151Z outputs = self.mobilebert( 2025-12-04T08:56:07.1513463Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1513541Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1513843Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1513927Z layer_outputs = layer_module( 2025-12-04T08:56:07.1514230Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.1514336Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.1514637Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T08:56:07.1514756Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T08:56:07.1515070Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T08:56:07.1515192Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:56:07.1515196Z 2025-12-04T08:56:07.1515314Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1515527Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1515596Z res = mod(**inputs) 2025-12-04T08:56:07.1515907Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1515983Z outputs = self.mobilebert( 2025-12-04T08:56:07.1516292Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1516370Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1516674Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1516763Z layer_outputs = layer_module( 2025-12-04T08:56:07.1517082Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.1517183Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.1517497Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T08:56:07.1517628Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T08:56:07.1517950Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T08:56:07.1518116Z layer_outputs = self.dense(hidden_states) 2025-12-04T08:56:07.1518122Z 2025-12-04T08:56:07.1518243Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1518492Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1518584Z res = mod(**inputs) 2025-12-04T08:56:07.1518918Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1518996Z outputs = self.mobilebert( 2025-12-04T08:56:07.1519319Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1519407Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1519728Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1519807Z layer_outputs = layer_module( 2025-12-04T08:56:07.1520135Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.1520264Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.1520589Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T08:56:07.1520888Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T08:56:07.1521265Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T08:56:07.1521405Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T08:56:07.1521718Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:07.1521822Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:07.1521826Z 2025-12-04T08:56:07.1521938Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1522156Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1522239Z res = mod(**inputs) 2025-12-04T08:56:07.1522542Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1522627Z outputs = self.mobilebert( 2025-12-04T08:56:07.1522935Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1523012Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1523330Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1523406Z layer_outputs = layer_module( 2025-12-04T08:56:07.1523719Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.1523828Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.1524132Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T08:56:07.1524316Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T08:56:07.1524618Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T08:56:07.1524707Z hidden_states = self.dense(hidden_states) 2025-12-04T08:56:07.1524710Z 2025-12-04T08:56:07.1524829Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1525039Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1525116Z res = mod(**inputs) 2025-12-04T08:56:07.1525431Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1525510Z outputs = self.mobilebert( 2025-12-04T08:56:07.1525897Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1525999Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1526301Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1526385Z layer_outputs = layer_module( 2025-12-04T08:56:07.1526697Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.1526803Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.1527114Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T08:56:07.1527238Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T08:56:07.1527570Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T08:56:07.1527687Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:56:07.1527690Z 2025-12-04T08:56:07.1527804Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1528008Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1528074Z res = mod(**inputs) 2025-12-04T08:56:07.1528367Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1528441Z outputs = self.mobilebert( 2025-12-04T08:56:07.1528730Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1528812Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1529102Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1529186Z layer_outputs = layer_module( 2025-12-04T08:56:07.1529474Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.1529569Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.1529864Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T08:56:07.1529990Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T08:56:07.1530285Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T08:56:07.1530368Z layer_outputs = self.dense(hidden_states) 2025-12-04T08:56:07.1530373Z 2025-12-04T08:56:07.1530475Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1530686Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1530752Z res = mod(**inputs) 2025-12-04T08:56:07.1531060Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1531142Z outputs = self.mobilebert( 2025-12-04T08:56:07.1531427Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1531516Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1531814Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1531886Z layer_outputs = layer_module( 2025-12-04T08:56:07.1532179Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.1532297Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.1532623Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T08:56:07.1532755Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T08:56:07.1533061Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T08:56:07.1533200Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T08:56:07.1533500Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:07.1533603Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:07.1533607Z 2025-12-04T08:56:07.1533737Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1533951Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1534030Z res = mod(**inputs) 2025-12-04T08:56:07.1534336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1534412Z outputs = self.mobilebert( 2025-12-04T08:56:07.1534723Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1534800Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1535119Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1535195Z layer_outputs = layer_module( 2025-12-04T08:56:07.1535496Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-12-04T08:56:07.1535638Z intermediate_output = self.intermediate(attention_output) 2025-12-04T08:56:07.1535942Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T08:56:07.1536041Z hidden_states = self.dense(hidden_states) 2025-12-04T08:56:07.1536045Z 2025-12-04T08:56:07.1536154Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1536365Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1536443Z res = mod(**inputs) 2025-12-04T08:56:07.1536748Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1536823Z outputs = self.mobilebert( 2025-12-04T08:56:07.1537132Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1537213Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1537521Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1537617Z layer_outputs = layer_module( 2025-12-04T08:56:07.1537923Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-12-04T08:56:07.1538060Z intermediate_output = self.intermediate(attention_output) 2025-12-04T08:56:07.1538372Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T08:56:07.1538521Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:56:07.1538525Z 2025-12-04T08:56:07.1538634Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1538848Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1538925Z res = mod(**inputs) 2025-12-04T08:56:07.1539273Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1539351Z outputs = self.mobilebert( 2025-12-04T08:56:07.1539660Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1539737Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1540058Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1540134Z layer_outputs = layer_module( 2025-12-04T08:56:07.1540443Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T08:56:07.1540643Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T08:56:07.1540959Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-12-04T08:56:07.1541068Z layer_output = self.dense(intermediate_states) 2025-12-04T08:56:07.1541073Z 2025-12-04T08:56:07.1541186Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1541399Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1541476Z res = mod(**inputs) 2025-12-04T08:56:07.1541847Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1541930Z outputs = self.mobilebert( 2025-12-04T08:56:07.1542247Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1542327Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1542641Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1542718Z layer_outputs = layer_module( 2025-12-04T08:56:07.1543028Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T08:56:07.1543208Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T08:56:07.1543525Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-12-04T08:56:07.1543662Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-12-04T08:56:07.1543979Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:07.1544080Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:07.1544084Z 2025-12-04T08:56:07.1544204Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1544418Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1544513Z res = mod(**inputs) 2025-12-04T08:56:07.1544816Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1544891Z outputs = self.mobilebert( 2025-12-04T08:56:07.1545204Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1545282Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1545595Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1545678Z layer_outputs = layer_module( 2025-12-04T08:56:07.1546001Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T08:56:07.1546200Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T08:56:07.1546520Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-12-04T08:56:07.1546652Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-12-04T08:56:07.1546965Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-12-04T08:56:07.1547056Z layer_outputs = self.dense(hidden_states) 2025-12-04T08:56:07.1547060Z 2025-12-04T08:56:07.1547178Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1547391Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1547480Z res = mod(**inputs) 2025-12-04T08:56:07.1547793Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1547869Z outputs = self.mobilebert( 2025-12-04T08:56:07.1548180Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1548259Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1548561Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1548645Z layer_outputs = layer_module( 2025-12-04T08:56:07.1548945Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T08:56:07.1549112Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T08:56:07.1549423Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-12-04T08:56:07.1549555Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-12-04T08:56:07.1549862Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-12-04T08:56:07.1549991Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T08:56:07.1550307Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:07.1550414Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:07.1550418Z 2025-12-04T08:56:07.1550526Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1550743Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1550815Z res = mod(**inputs) 2025-12-04T08:56:07.1551118Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1551203Z outputs = self.mobilebert( 2025-12-04T08:56:07.1551525Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1551603Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1551912Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1551988Z layer_outputs = layer_module( 2025-12-04T08:56:07.1552296Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T08:56:07.1552470Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T08:56:07.1552791Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-12-04T08:56:07.1552938Z shared_attention_input = self.attention(hidden_states) 2025-12-04T08:56:07.1553243Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-12-04T08:56:07.1553338Z layer_input = self.dense(hidden_states) 2025-12-04T08:56:07.1553342Z 2025-12-04T08:56:07.1553452Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1553661Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1553738Z res = mod(**inputs) 2025-12-04T08:56:07.1554041Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1554123Z outputs = self.mobilebert( 2025-12-04T08:56:07.1554444Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1554524Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1554834Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1554910Z layer_outputs = layer_module( 2025-12-04T08:56:07.1555209Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T08:56:07.1555308Z self_attention_outputs = self.attention( 2025-12-04T08:56:07.1555611Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T08:56:07.1555696Z self_outputs = self.self( 2025-12-04T08:56:07.1556007Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-12-04T08:56:07.1556090Z self.value(value_tensor) 2025-12-04T08:56:07.1556095Z 2025-12-04T08:56:07.1556217Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1556443Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1556518Z res = mod(**inputs) 2025-12-04T08:56:07.1556818Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1556894Z outputs = self.mobilebert( 2025-12-04T08:56:07.1557210Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1557286Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1557585Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1557669Z layer_outputs = layer_module( 2025-12-04T08:56:07.1557973Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T08:56:07.1558251Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T08:56:07.1558579Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-12-04T08:56:07.1558705Z bottlenecked_hidden_states = self.input(hidden_states) 2025-12-04T08:56:07.1559031Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-12-04T08:56:07.1559124Z layer_input = self.dense(hidden_states) 2025-12-04T08:56:07.1559128Z 2025-12-04T08:56:07.1559251Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1559473Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1559547Z res = mod(**inputs) 2025-12-04T08:56:07.1559904Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1559999Z outputs = self.mobilebert( 2025-12-04T08:56:07.1560305Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1560392Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1560697Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1560780Z layer_outputs = layer_module( 2025-12-04T08:56:07.1561086Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-12-04T08:56:07.1561256Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-12-04T08:56:07.1561601Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-12-04T08:56:07.1561716Z shared_attention_input = self.attention(hidden_states) 2025-12-04T08:56:07.1562009Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-12-04T08:56:07.1562097Z layer_input = self.LayerNorm(layer_input) 2025-12-04T08:56:07.1562382Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:07.1562483Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:07.1562486Z 2025-12-04T08:56:07.1562589Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1562794Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1562864Z res = mod(**inputs) 2025-12-04T08:56:07.1563148Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1563229Z outputs = self.mobilebert( 2025-12-04T08:56:07.1563514Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1563588Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1563881Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1563953Z layer_outputs = layer_module( 2025-12-04T08:56:07.1564244Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T08:56:07.1564332Z self_attention_outputs = self.attention( 2025-12-04T08:56:07.1564622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T08:56:07.1564705Z self_outputs = self.self( 2025-12-04T08:56:07.1565012Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-12-04T08:56:07.1565093Z self.query(query_tensor) 2025-12-04T08:56:07.1565097Z 2025-12-04T08:56:07.1565202Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1565405Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1565476Z res = mod(**inputs) 2025-12-04T08:56:07.1565766Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1565840Z outputs = self.mobilebert( 2025-12-04T08:56:07.1566140Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1566251Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1566575Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1566647Z layer_outputs = layer_module( 2025-12-04T08:56:07.1566932Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T08:56:07.1567025Z self_attention_outputs = self.attention( 2025-12-04T08:56:07.1567315Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-12-04T08:56:07.1567393Z self_outputs = self.self( 2025-12-04T08:56:07.1567679Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-12-04T08:56:07.1567769Z self.key(key_tensor) 2025-12-04T08:56:07.1567773Z 2025-12-04T08:56:07.1567866Z cudagraph partition due to non gpu ops 2025-12-04T08:56:07.1567950Z cudagraph partition due to non gpu ops 2025-12-04T08:56:07.1568056Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1568265Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1568332Z res = mod(**inputs) 2025-12-04T08:56:07.1568620Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1568692Z outputs = self.mobilebert( 2025-12-04T08:56:07.1568976Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1569057Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1569341Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1569417Z layer_outputs = layer_module( 2025-12-04T08:56:07.1569709Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T08:56:07.1569798Z self_attention_outputs = self.attention( 2025-12-04T08:56:07.1570089Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-12-04T08:56:07.1570214Z attention_output = self.output(self_outputs[0], layer_input) 2025-12-04T08:56:07.1570496Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-12-04T08:56:07.1570590Z layer_outputs = self.dense(hidden_states) 2025-12-04T08:56:07.1570594Z 2025-12-04T08:56:07.1570699Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1570908Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1570974Z res = mod(**inputs) 2025-12-04T08:56:07.1571293Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1571376Z outputs = self.mobilebert( 2025-12-04T08:56:07.1571665Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1571740Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1572047Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1572123Z layer_outputs = layer_module( 2025-12-04T08:56:07.1572432Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-12-04T08:56:07.1572534Z self_attention_outputs = self.attention( 2025-12-04T08:56:07.1572837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-12-04T08:56:07.1572990Z attention_output = self.output(self_outputs[0], layer_input) 2025-12-04T08:56:07.1573274Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-12-04T08:56:07.1573408Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T08:56:07.1573692Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:07.1573784Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:07.1573788Z 2025-12-04T08:56:07.1573899Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1574098Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1574189Z res = mod(**inputs) 2025-12-04T08:56:07.1574472Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1574546Z outputs = self.mobilebert( 2025-12-04T08:56:07.1574837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1574910Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1575195Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1575274Z layer_outputs = layer_module( 2025-12-04T08:56:07.1575558Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.1575660Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.1575949Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T08:56:07.1576065Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T08:56:07.1576363Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T08:56:07.1576448Z hidden_states = self.dense(hidden_states) 2025-12-04T08:56:07.1576451Z 2025-12-04T08:56:07.1576561Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1576761Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1576830Z res = mod(**inputs) 2025-12-04T08:56:07.1577133Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1577210Z outputs = self.mobilebert( 2025-12-04T08:56:07.1577519Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1577601Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1577907Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1577987Z layer_outputs = layer_module( 2025-12-04T08:56:07.1578272Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.1578367Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.1578659Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T08:56:07.1578773Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T08:56:07.1579065Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T08:56:07.1579200Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:56:07.1579221Z 2025-12-04T08:56:07.1579333Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1579558Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1579627Z res = mod(**inputs) 2025-12-04T08:56:07.1579928Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1580011Z outputs = self.mobilebert( 2025-12-04T08:56:07.1580324Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1580410Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1580721Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1580817Z layer_outputs = layer_module( 2025-12-04T08:56:07.1581133Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.1581233Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.1581540Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T08:56:07.1581675Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T08:56:07.1581987Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T08:56:07.1582086Z layer_outputs = self.dense(hidden_states) 2025-12-04T08:56:07.1582090Z 2025-12-04T08:56:07.1582199Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1582421Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1582495Z res = mod(**inputs) 2025-12-04T08:56:07.1582801Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1582885Z outputs = self.mobilebert( 2025-12-04T08:56:07.1583187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1583265Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1583588Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1583664Z layer_outputs = layer_module( 2025-12-04T08:56:07.1583986Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.1584087Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.1584396Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T08:56:07.1584561Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T08:56:07.1584864Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T08:56:07.1585002Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T08:56:07.1585314Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:07.1585412Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:07.1585416Z 2025-12-04T08:56:07.1585534Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1585749Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1585843Z res = mod(**inputs) 2025-12-04T08:56:07.1586167Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1586245Z outputs = self.mobilebert( 2025-12-04T08:56:07.1586553Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1586637Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1586922Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1587001Z layer_outputs = layer_module( 2025-12-04T08:56:07.1587290Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.1587412Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.1587700Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T08:56:07.1587816Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T08:56:07.1588110Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T08:56:07.1588196Z hidden_states = self.dense(hidden_states) 2025-12-04T08:56:07.1588199Z 2025-12-04T08:56:07.1588310Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1588512Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1588579Z res = mod(**inputs) 2025-12-04T08:56:07.1588875Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1588949Z outputs = self.mobilebert( 2025-12-04T08:56:07.1589237Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1589321Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1589612Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1589690Z layer_outputs = layer_module( 2025-12-04T08:56:07.1589975Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.1590068Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.1590359Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T08:56:07.1590471Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T08:56:07.1590759Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T08:56:07.1590881Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:56:07.1590884Z 2025-12-04T08:56:07.1591018Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1591227Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1591292Z res = mod(**inputs) 2025-12-04T08:56:07.1591579Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1591659Z outputs = self.mobilebert( 2025-12-04T08:56:07.1591943Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1592023Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1592338Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1592425Z layer_outputs = layer_module( 2025-12-04T08:56:07.1592720Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.1592815Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.1593105Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T08:56:07.1593230Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T08:56:07.1593514Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T08:56:07.1593605Z layer_outputs = self.dense(hidden_states) 2025-12-04T08:56:07.1593608Z 2025-12-04T08:56:07.1593734Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1593934Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1594007Z res = mod(**inputs) 2025-12-04T08:56:07.1594295Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1594374Z outputs = self.mobilebert( 2025-12-04T08:56:07.1594668Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1594744Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1595055Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1595130Z layer_outputs = layer_module( 2025-12-04T08:56:07.1595437Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.1595539Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.1595842Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T08:56:07.1595982Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T08:56:07.1596284Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T08:56:07.1596413Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T08:56:07.1596722Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:07.1596819Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:07.1596822Z 2025-12-04T08:56:07.1596938Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1597150Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1597223Z res = mod(**inputs) 2025-12-04T08:56:07.1597554Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1597631Z outputs = self.mobilebert( 2025-12-04T08:56:07.1597942Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1598019Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1598388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1598475Z layer_outputs = layer_module( 2025-12-04T08:56:07.1598777Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.1598881Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.1599213Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T08:56:07.1599354Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T08:56:07.1599664Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T08:56:07.1599753Z hidden_states = self.dense(hidden_states) 2025-12-04T08:56:07.1599757Z 2025-12-04T08:56:07.1599867Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1600096Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1600166Z res = mod(**inputs) 2025-12-04T08:56:07.1600481Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1600572Z outputs = self.mobilebert( 2025-12-04T08:56:07.1600864Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1600958Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1601259Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1601338Z layer_outputs = layer_module( 2025-12-04T08:56:07.1601630Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.1601724Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.1602020Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-12-04T08:56:07.1602134Z intermediate_output = self.intermediate(hidden_states) 2025-12-04T08:56:07.1602427Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T08:56:07.1602550Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:56:07.1602554Z 2025-12-04T08:56:07.1602659Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1602868Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1602933Z res = mod(**inputs) 2025-12-04T08:56:07.1603225Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1603305Z outputs = self.mobilebert( 2025-12-04T08:56:07.1603596Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1603675Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1603968Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1604040Z layer_outputs = layer_module( 2025-12-04T08:56:07.1604356Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.1604457Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.1604760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T08:56:07.1604900Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T08:56:07.1605209Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-12-04T08:56:07.1605300Z layer_outputs = self.dense(hidden_states) 2025-12-04T08:56:07.1605306Z 2025-12-04T08:56:07.1605409Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1605630Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1605726Z res = mod(**inputs) 2025-12-04T08:56:07.1606028Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1606111Z outputs = self.mobilebert( 2025-12-04T08:56:07.1606410Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1606486Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1606800Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1606876Z layer_outputs = layer_module( 2025-12-04T08:56:07.1607188Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-12-04T08:56:07.1607317Z attention_output = ffn_module(attention_output) 2025-12-04T08:56:07.1607626Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-12-04T08:56:07.1607765Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-12-04T08:56:07.1608065Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-12-04T08:56:07.1608195Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T08:56:07.1608517Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:07.1608614Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:07.1608618Z 2025-12-04T08:56:07.1608733Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1608949Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1609020Z res = mod(**inputs) 2025-12-04T08:56:07.1609336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1609410Z outputs = self.mobilebert( 2025-12-04T08:56:07.1609721Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1609797Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1610110Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1610191Z layer_outputs = layer_module( 2025-12-04T08:56:07.1610505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-12-04T08:56:07.1610637Z intermediate_output = self.intermediate(attention_output) 2025-12-04T08:56:07.1610950Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-12-04T08:56:07.1611059Z hidden_states = self.dense(hidden_states) 2025-12-04T08:56:07.1611063Z 2025-12-04T08:56:07.1611179Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1611391Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1611459Z res = mod(**inputs) 2025-12-04T08:56:07.1611771Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1611846Z outputs = self.mobilebert( 2025-12-04T08:56:07.1612155Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1612235Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1612557Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1612659Z layer_outputs = layer_module( 2025-12-04T08:56:07.1612962Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-12-04T08:56:07.1613090Z intermediate_output = self.intermediate(attention_output) 2025-12-04T08:56:07.1613399Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-12-04T08:56:07.1613517Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:56:07.1613521Z 2025-12-04T08:56:07.1613638Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1613848Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1613945Z res = mod(**inputs) 2025-12-04T08:56:07.1614266Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1614344Z outputs = self.mobilebert( 2025-12-04T08:56:07.1614662Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1614748Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1615035Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1615112Z layer_outputs = layer_module( 2025-12-04T08:56:07.1615398Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T08:56:07.1615556Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T08:56:07.1615850Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-12-04T08:56:07.1615946Z layer_output = self.dense(intermediate_states) 2025-12-04T08:56:07.1615951Z 2025-12-04T08:56:07.1616063Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1616260Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1616324Z res = mod(**inputs) 2025-12-04T08:56:07.1616617Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1616688Z outputs = self.mobilebert( 2025-12-04T08:56:07.1616985Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1617059Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1617346Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1617426Z layer_outputs = layer_module( 2025-12-04T08:56:07.1617729Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T08:56:07.1617890Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T08:56:07.1618164Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-12-04T08:56:07.1618282Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-12-04T08:56:07.1618563Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:07.1618655Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:07.1618658Z 2025-12-04T08:56:07.1618775Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1618991Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1619057Z res = mod(**inputs) 2025-12-04T08:56:07.1619341Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1619410Z outputs = self.mobilebert( 2025-12-04T08:56:07.1619687Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1619765Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1620039Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1620114Z layer_outputs = layer_module( 2025-12-04T08:56:07.1620409Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T08:56:07.1620565Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T08:56:07.1621110Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-12-04T08:56:07.1621247Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-12-04T08:56:07.1621533Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-12-04T08:56:07.1621619Z layer_outputs = self.dense(hidden_states) 2025-12-04T08:56:07.1621622Z 2025-12-04T08:56:07.1621725Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1621930Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1621999Z res = mod(**inputs) 2025-12-04T08:56:07.1622277Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-12-04T08:56:07.1622359Z outputs = self.mobilebert( 2025-12-04T08:56:07.1622637Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-12-04T08:56:07.1622717Z encoder_outputs = self.encoder( 2025-12-04T08:56:07.1622993Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-12-04T08:56:07.1623065Z layer_outputs = layer_module( 2025-12-04T08:56:07.1623353Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-12-04T08:56:07.1623507Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-12-04T08:56:07.1623796Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-12-04T08:56:07.1623919Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-12-04T08:56:07.1624249Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-12-04T08:56:07.1624379Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-12-04T08:56:07.1624657Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-12-04T08:56:07.1624746Z return input_tensor * self.weight + self.bias 2025-12-04T08:56:07.1624756Z 2025-12-04T08:56:07.1624856Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1625049Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1625121Z res = mod(**inputs) 2025-12-04T08:56:07.1625424Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 989, in forward 2025-12-04T08:56:07.1625541Z prediction_scores = self.cls(sequence_output) 2025-12-04T08:56:07.1625830Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 643, in forward 2025-12-04T08:56:07.1625945Z prediction_scores = self.predictions(sequence_output) 2025-12-04T08:56:07.1626241Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 631, in forward 2025-12-04T08:56:07.1626336Z hidden_states = self.transform(hidden_states) 2025-12-04T08:56:07.1626613Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 609, in forward 2025-12-04T08:56:07.1626705Z hidden_states = self.dense(hidden_states) 2025-12-04T08:56:07.1626736Z 2025-12-04T08:56:07.1626837Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1627038Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1627102Z res = mod(**inputs) 2025-12-04T08:56:07.1627384Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 989, in forward 2025-12-04T08:56:07.1627480Z prediction_scores = self.cls(sequence_output) 2025-12-04T08:56:07.1627758Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 643, in forward 2025-12-04T08:56:07.1627868Z prediction_scores = self.predictions(sequence_output) 2025-12-04T08:56:07.1628161Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 632, in forward 2025-12-04T08:56:07.1628373Z hidden_states = hidden_states.matmul(torch.cat([self.decoder.weight.t(), self.dense.weight], dim=0)) 2025-12-04T08:56:07.1628379Z 2025-12-04T08:56:07.1628492Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1628697Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1628764Z res = mod(**inputs) 2025-12-04T08:56:07.1629062Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 989, in forward 2025-12-04T08:56:07.1629155Z prediction_scores = self.cls(sequence_output) 2025-12-04T08:56:07.1629448Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 643, in forward 2025-12-04T08:56:07.1629559Z prediction_scores = self.predictions(sequence_output) 2025-12-04T08:56:07.1629848Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 633, in forward 2025-12-04T08:56:07.1629939Z hidden_states += self.decoder.bias 2025-12-04T08:56:07.1629943Z 2025-12-04T08:56:07.1630047Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:07.1630260Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:07.1630395Z res = mod(**inputs) 2025-12-04T08:56:07.1630681Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 994, in forward 2025-12-04T08:56:07.1630882Z masked_lm_loss = loss_fct(prediction_scores.view(-1, self.config.vocab_size), labels.view(-1)) 2025-12-04T08:56:07.1630886Z 2025-12-04T08:56:21.4655720Z Compilation time (from dynamo_timed): 41.286213893 2025-12-04T08:56:21.4656196Z pass 2025-12-04T08:56:21.4656609Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T08:56:21.4657901Z TIMING: _recursive_pre_grad_passes:0.02387 _recursive_joint_graph_passes:1.45289 _recursive_post_grad_passes:0.20579 async_compile.wait:1.27633 code_gen:12.61917 inductor_compile:17.31558 backend_compile:29.56195 gc:0.0004 entire_frame_compile:41.28621 total_wall_time:41.28621 2025-12-04T08:56:21.4658964Z STATS: call_* op count: 1449 | FakeTensorMode.__torch_dispatch__:30412 | FakeTensor.__torch_dispatch__:15340 | ProxyTorchDispatchMode.__torch_dispatch__:8821 2025-12-04T08:56:21.4659513Z Dynamo produced 1 graphs covering 1449 ops with 0 graph breaks (0 unique) 2025-12-04T08:56:25.0537203Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/__init__.py:65: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-12-04T08:56:25.0538094Z import pynvml # type: ignore[import] 2025-12-04T08:56:28.7429494Z 2025-12-04T08:56:31.4305589Z loading model: 0it [00:00, ?it/s] 2025-12-04T08:56:31.4306446Z loading model: 0it [00:02, ?it/s] 2025-12-04T08:56:31.4312409Z cpu eval OPTForCausalLM 2025-12-04T08:56:37.1549312Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T08:56:39.8276710Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T08:56:42.7367104Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T08:56:50.2860544Z cudagraph partition due to non gpu ops 2025-12-04T08:56:50.2861061Z cudagraph partition due to non gpu ops 2025-12-04T08:56:50.2861423Z cudagraph partition due to non gpu ops 2025-12-04T08:56:50.2861763Z cudagraph partition due to non gpu ops 2025-12-04T08:56:50.2862131Z cudagraph partition due to non gpu ops 2025-12-04T08:56:50.2862447Z cudagraph partition due to non gpu ops 2025-12-04T08:56:50.2862791Z cudagraph partition due to non gpu ops 2025-12-04T08:56:50.2863121Z cudagraph partition due to non gpu ops 2025-12-04T08:56:50.2863490Z cudagraph partition due to non gpu ops 2025-12-04T08:56:50.2863830Z cudagraph partition due to non gpu ops 2025-12-04T08:56:50.2864175Z cudagraph partition due to non gpu ops 2025-12-04T08:56:50.2864501Z cudagraph partition due to non gpu ops 2025-12-04T08:56:50.2864900Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:50.2865548Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:50.2866087Z res = mod(**inputs) 2025-12-04T08:56:50.2866704Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.2867368Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.2868071Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T08:56:50.2868765Z outputs = self.model.decoder( 2025-12-04T08:56:50.2869382Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.2870003Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.2870638Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T08:56:50.2871730Z layer_outputs = decoder_layer( 2025-12-04T08:56:50.2872371Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:56:50.2873026Z return super().__call__(*args, **kwargs) 2025-12-04T08:56:50.2873730Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.2874422Z return func(*args, **kwargs) 2025-12-04T08:56:50.2875067Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T08:56:50.2875794Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:56:50.2876637Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.2877404Z return func(*args, **kwargs) 2025-12-04T08:56:50.2878236Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 160, in forward 2025-12-04T08:56:50.2879049Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T08:56:50.2879365Z 2025-12-04T08:56:50.2879559Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:50.2880198Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:50.2880807Z res = mod(**inputs) 2025-12-04T08:56:50.2881443Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.2882145Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.2883031Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T08:56:50.2883763Z outputs = self.model.decoder( 2025-12-04T08:56:50.2884362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.2884988Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.2885684Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T08:56:50.2886362Z layer_outputs = decoder_layer( 2025-12-04T08:56:50.2886990Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:56:50.2887654Z return super().__call__(*args, **kwargs) 2025-12-04T08:56:50.2888323Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.2888967Z return func(*args, **kwargs) 2025-12-04T08:56:50.2889610Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T08:56:50.2890346Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:56:50.2891049Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.2891682Z return func(*args, **kwargs) 2025-12-04T08:56:50.2892346Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 163, in forward 2025-12-04T08:56:50.2893030Z key_states = self.k_proj(hidden_states) 2025-12-04T08:56:50.2893279Z 2025-12-04T08:56:50.2893443Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:50.2894089Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:50.2894656Z res = mod(**inputs) 2025-12-04T08:56:50.2895230Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.2895868Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.2896585Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T08:56:50.2897260Z outputs = self.model.decoder( 2025-12-04T08:56:50.2897916Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.2898554Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.2899242Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T08:56:50.2899903Z layer_outputs = decoder_layer( 2025-12-04T08:56:50.2900509Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:56:50.2901162Z return super().__call__(*args, **kwargs) 2025-12-04T08:56:50.2901865Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.2902563Z return func(*args, **kwargs) 2025-12-04T08:56:50.2903193Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T08:56:50.2903915Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:56:50.2904620Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.2905262Z return func(*args, **kwargs) 2025-12-04T08:56:50.2905887Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 164, in forward 2025-12-04T08:56:50.2906578Z value_states = self.v_proj(hidden_states) 2025-12-04T08:56:50.2906825Z 2025-12-04T08:56:50.2906973Z cudagraph partition due to non gpu ops 2025-12-04T08:56:50.2907379Z cudagraph partition due to non gpu ops 2025-12-04T08:56:50.2907765Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:50.2908453Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:50.2909030Z res = mod(**inputs) 2025-12-04T08:56:50.2909673Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.2910297Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.2910999Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T08:56:50.2911692Z outputs = self.model.decoder( 2025-12-04T08:56:50.2912340Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.2912998Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.2913696Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T08:56:50.2914393Z layer_outputs = decoder_layer( 2025-12-04T08:56:50.2915051Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:56:50.2915735Z return super().__call__(*args, **kwargs) 2025-12-04T08:56:50.2916450Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.2917179Z return func(*args, **kwargs) 2025-12-04T08:56:50.2917853Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T08:56:50.2918734Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:56:50.2919497Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.2920163Z return func(*args, **kwargs) 2025-12-04T08:56:50.2921032Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 179, in forward 2025-12-04T08:56:50.2921725Z attn_output, attn_weights = attention_interface( 2025-12-04T08:56:50.2922646Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T08:56:50.2923664Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T08:56:50.2924033Z 2025-12-04T08:56:50.2924239Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:50.2924941Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:50.2925485Z res = mod(**inputs) 2025-12-04T08:56:50.2926006Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.2926628Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.2927359Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T08:56:50.2928108Z outputs = self.model.decoder( 2025-12-04T08:56:50.2928737Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.2929403Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.2930118Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T08:56:50.2930830Z layer_outputs = decoder_layer( 2025-12-04T08:56:50.2931453Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:56:50.2932064Z return super().__call__(*args, **kwargs) 2025-12-04T08:56:50.2932746Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.2933507Z return func(*args, **kwargs) 2025-12-04T08:56:50.2934170Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T08:56:50.2934926Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:56:50.2935691Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.2936372Z return func(*args, **kwargs) 2025-12-04T08:56:50.2937020Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 191, in forward 2025-12-04T08:56:50.2937740Z attn_output = self.out_proj(attn_output) 2025-12-04T08:56:50.2937978Z 2025-12-04T08:56:50.2938157Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:50.2938806Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:50.2939380Z res = mod(**inputs) 2025-12-04T08:56:50.2939950Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.2940576Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.2941221Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T08:56:50.2941921Z outputs = self.model.decoder( 2025-12-04T08:56:50.2942528Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.2943193Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.2943908Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T08:56:50.2944580Z layer_outputs = decoder_layer( 2025-12-04T08:56:50.2945227Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:56:50.2945825Z return super().__call__(*args, **kwargs) 2025-12-04T08:56:50.2946472Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.2947170Z return func(*args, **kwargs) 2025-12-04T08:56:50.2947911Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 281, in forward 2025-12-04T08:56:50.2948648Z hidden_states = self.fc1(hidden_states) 2025-12-04T08:56:50.2948902Z 2025-12-04T08:56:50.2949073Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:50.2949727Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:50.2950307Z res = mod(**inputs) 2025-12-04T08:56:50.2950939Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.2951618Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.2952323Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T08:56:50.2953113Z outputs = self.model.decoder( 2025-12-04T08:56:50.2953821Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.2954494Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.2955207Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T08:56:50.2955942Z layer_outputs = decoder_layer( 2025-12-04T08:56:50.2956627Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:56:50.2957332Z return super().__call__(*args, **kwargs) 2025-12-04T08:56:50.2958221Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.2959036Z return func(*args, **kwargs) 2025-12-04T08:56:50.2959797Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 282, in forward 2025-12-04T08:56:50.2960604Z hidden_states = self.activation_fn(hidden_states) 2025-12-04T08:56:50.2960898Z 2025-12-04T08:56:50.2961492Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:50.2962191Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:50.2962795Z res = mod(**inputs) 2025-12-04T08:56:50.2963412Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.2964083Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.2964800Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T08:56:50.2965550Z outputs = self.model.decoder( 2025-12-04T08:56:50.2966226Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.2966934Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.2967676Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T08:56:50.2968403Z layer_outputs = decoder_layer( 2025-12-04T08:56:50.2969078Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:56:50.2969791Z return super().__call__(*args, **kwargs) 2025-12-04T08:56:50.2970551Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.2971301Z return func(*args, **kwargs) 2025-12-04T08:56:50.2972028Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 284, in forward 2025-12-04T08:56:50.2972793Z hidden_states = self.fc2(hidden_states) 2025-12-04T08:56:50.2973063Z 2025-12-04T08:56:50.2973268Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:50.2973987Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:50.2974614Z res = mod(**inputs) 2025-12-04T08:56:50.2975329Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.2976019Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.2976736Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T08:56:50.2977454Z outputs = self.model.decoder( 2025-12-04T08:56:50.2978109Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.2978761Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.2979461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T08:56:50.2980210Z layer_outputs = decoder_layer( 2025-12-04T08:56:50.2980905Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:56:50.2981596Z return super().__call__(*args, **kwargs) 2025-12-04T08:56:50.2982285Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.2982968Z return func(*args, **kwargs) 2025-12-04T08:56:50.2983669Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T08:56:50.2984467Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:56:50.2985270Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.2986044Z return func(*args, **kwargs) 2025-12-04T08:56:50.2986804Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 160, in forward 2025-12-04T08:56:50.2987626Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T08:56:50.2987938Z 2025-12-04T08:56:50.2988125Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:50.2988904Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:50.2989576Z res = mod(**inputs) 2025-12-04T08:56:50.2990105Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.2990786Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.2991545Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T08:56:50.2992240Z outputs = self.model.decoder( 2025-12-04T08:56:50.2992904Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.2993604Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.2994335Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T08:56:50.2995078Z layer_outputs = decoder_layer( 2025-12-04T08:56:50.2995767Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:56:50.2996469Z return super().__call__(*args, **kwargs) 2025-12-04T08:56:50.2997197Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.2997908Z return func(*args, **kwargs) 2025-12-04T08:56:50.2998741Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T08:56:50.2999547Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:56:50.3000331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.3001081Z return func(*args, **kwargs) 2025-12-04T08:56:50.3001912Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 163, in forward 2025-12-04T08:56:50.3002674Z key_states = self.k_proj(hidden_states) 2025-12-04T08:56:50.3002932Z 2025-12-04T08:56:50.3003121Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:50.3003806Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:50.3004418Z res = mod(**inputs) 2025-12-04T08:56:50.3005031Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3005739Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3006465Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T08:56:50.3007274Z outputs = self.model.decoder( 2025-12-04T08:56:50.3007975Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3008679Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3009422Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T08:56:50.3010179Z layer_outputs = decoder_layer( 2025-12-04T08:56:50.3010877Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:56:50.3011580Z return super().__call__(*args, **kwargs) 2025-12-04T08:56:50.3012345Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.3013080Z return func(*args, **kwargs) 2025-12-04T08:56:50.3013860Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T08:56:50.3014650Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:56:50.3015467Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.3016216Z return func(*args, **kwargs) 2025-12-04T08:56:50.3016925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 164, in forward 2025-12-04T08:56:50.3017706Z value_states = self.v_proj(hidden_states) 2025-12-04T08:56:50.3017989Z 2025-12-04T08:56:50.3018137Z cudagraph partition due to non gpu ops 2025-12-04T08:56:50.3018534Z cudagraph partition due to non gpu ops 2025-12-04T08:56:50.3018975Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:50.3019672Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:50.3020275Z res = mod(**inputs) 2025-12-04T08:56:50.3021186Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3021895Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3022652Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T08:56:50.3023428Z outputs = self.model.decoder( 2025-12-04T08:56:50.3024136Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3024861Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3025633Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T08:56:50.3026403Z layer_outputs = decoder_layer( 2025-12-04T08:56:50.3027119Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:56:50.3027875Z return super().__call__(*args, **kwargs) 2025-12-04T08:56:50.3028673Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.3029645Z return func(*args, **kwargs) 2025-12-04T08:56:50.3030374Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T08:56:50.3031165Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:56:50.3031964Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.3032698Z return func(*args, **kwargs) 2025-12-04T08:56:50.3033457Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 179, in forward 2025-12-04T08:56:50.3034356Z attn_output, attn_weights = attention_interface( 2025-12-04T08:56:50.3035404Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T08:56:50.3036442Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T08:56:50.3036813Z 2025-12-04T08:56:50.3037011Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:50.3037750Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:50.3038514Z res = mod(**inputs) 2025-12-04T08:56:50.3039168Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3039891Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3040649Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T08:56:50.3041398Z outputs = self.model.decoder( 2025-12-04T08:56:50.3042159Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3042871Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3043624Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T08:56:50.3044386Z layer_outputs = decoder_layer( 2025-12-04T08:56:50.3045085Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:56:50.3045792Z return super().__call__(*args, **kwargs) 2025-12-04T08:56:50.3046546Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.3047282Z return func(*args, **kwargs) 2025-12-04T08:56:50.3048015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T08:56:50.3048845Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:56:50.3049649Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.3050406Z return func(*args, **kwargs) 2025-12-04T08:56:50.3051145Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 191, in forward 2025-12-04T08:56:50.3051901Z attn_output = self.out_proj(attn_output) 2025-12-04T08:56:50.3052172Z 2025-12-04T08:56:50.3052363Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:50.3053061Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:50.3053690Z res = mod(**inputs) 2025-12-04T08:56:50.3054323Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3055012Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3055730Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T08:56:50.3056459Z outputs = self.model.decoder( 2025-12-04T08:56:50.3057185Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3057911Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3058645Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T08:56:50.3059343Z layer_outputs = decoder_layer( 2025-12-04T08:56:50.3060036Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:56:50.3060757Z return super().__call__(*args, **kwargs) 2025-12-04T08:56:50.3061516Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.3062258Z return func(*args, **kwargs) 2025-12-04T08:56:50.3063003Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 281, in forward 2025-12-04T08:56:50.3063769Z hidden_states = self.fc1(hidden_states) 2025-12-04T08:56:50.3064014Z 2025-12-04T08:56:50.3064201Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:50.3064919Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:50.3065523Z res = mod(**inputs) 2025-12-04T08:56:50.3066136Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3066798Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3067524Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T08:56:50.3068271Z outputs = self.model.decoder( 2025-12-04T08:56:50.3068918Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3069673Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3070419Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T08:56:50.3071173Z layer_outputs = decoder_layer( 2025-12-04T08:56:50.3071868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:56:50.3072598Z return super().__call__(*args, **kwargs) 2025-12-04T08:56:50.3073368Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.3074109Z return func(*args, **kwargs) 2025-12-04T08:56:50.3074840Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 282, in forward 2025-12-04T08:56:50.3075660Z hidden_states = self.activation_fn(hidden_states) 2025-12-04T08:56:50.3075964Z 2025-12-04T08:56:50.3076172Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:50.3076890Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:50.3077527Z res = mod(**inputs) 2025-12-04T08:56:50.3078288Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3079023Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3079780Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T08:56:50.3080568Z outputs = self.model.decoder( 2025-12-04T08:56:50.3081276Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3081974Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3082740Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T08:56:50.3083517Z layer_outputs = decoder_layer( 2025-12-04T08:56:50.3084195Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:56:50.3084976Z return super().__call__(*args, **kwargs) 2025-12-04T08:56:50.3085736Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.3086480Z return func(*args, **kwargs) 2025-12-04T08:56:50.3087196Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 284, in forward 2025-12-04T08:56:50.3087971Z hidden_states = self.fc2(hidden_states) 2025-12-04T08:56:50.3088242Z 2025-12-04T08:56:50.3088441Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:50.3089152Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:50.3089779Z res = mod(**inputs) 2025-12-04T08:56:50.3090505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3091267Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3092018Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T08:56:50.3092783Z outputs = self.model.decoder( 2025-12-04T08:56:50.3093450Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3094147Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3094881Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T08:56:50.3095641Z layer_outputs = decoder_layer( 2025-12-04T08:56:50.3096347Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:56:50.3097112Z return super().__call__(*args, **kwargs) 2025-12-04T08:56:50.3097878Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.3098635Z return func(*args, **kwargs) 2025-12-04T08:56:50.3099363Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T08:56:50.3100133Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:56:50.3100926Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.3101663Z return func(*args, **kwargs) 2025-12-04T08:56:50.3102355Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 160, in forward 2025-12-04T08:56:50.3103186Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T08:56:50.3103526Z 2025-12-04T08:56:50.3103719Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:50.3104437Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:50.3105055Z res = mod(**inputs) 2025-12-04T08:56:50.3105705Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3106410Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3107155Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T08:56:50.3107915Z outputs = self.model.decoder( 2025-12-04T08:56:50.3108598Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3109289Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3110031Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T08:56:50.3110772Z layer_outputs = decoder_layer( 2025-12-04T08:56:50.3111474Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:56:50.3112256Z return super().__call__(*args, **kwargs) 2025-12-04T08:56:50.3113015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.3113763Z return func(*args, **kwargs) 2025-12-04T08:56:50.3114489Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T08:56:50.3115299Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:56:50.3116121Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.3116881Z return func(*args, **kwargs) 2025-12-04T08:56:50.3117688Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 163, in forward 2025-12-04T08:56:50.3118623Z key_states = self.k_proj(hidden_states) 2025-12-04T08:56:50.3118915Z 2025-12-04T08:56:50.3119124Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:50.3119860Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:50.3120499Z res = mod(**inputs) 2025-12-04T08:56:50.3121445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3122226Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3122991Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T08:56:50.3123767Z outputs = self.model.decoder( 2025-12-04T08:56:50.3124468Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3125337Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3126128Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T08:56:50.3126876Z layer_outputs = decoder_layer( 2025-12-04T08:56:50.3127582Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:56:50.3128316Z return super().__call__(*args, **kwargs) 2025-12-04T08:56:50.3129110Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.3129895Z return func(*args, **kwargs) 2025-12-04T08:56:50.3130624Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T08:56:50.3131448Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:56:50.3132250Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.3132989Z return func(*args, **kwargs) 2025-12-04T08:56:50.3133695Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 164, in forward 2025-12-04T08:56:50.3134473Z value_states = self.v_proj(hidden_states) 2025-12-04T08:56:50.3134746Z 2025-12-04T08:56:50.3134892Z cudagraph partition due to non gpu ops 2025-12-04T08:56:50.3135291Z cudagraph partition due to non gpu ops 2025-12-04T08:56:50.3135737Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:50.3136445Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:50.3137085Z res = mod(**inputs) 2025-12-04T08:56:50.3137735Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3138460Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3139218Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T08:56:50.3139998Z outputs = self.model.decoder( 2025-12-04T08:56:50.3140807Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3141514Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3142248Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T08:56:50.3143004Z layer_outputs = decoder_layer( 2025-12-04T08:56:50.3143707Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:56:50.3144431Z return super().__call__(*args, **kwargs) 2025-12-04T08:56:50.3145203Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.3145956Z return func(*args, **kwargs) 2025-12-04T08:56:50.3146766Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T08:56:50.3147641Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:56:50.3148383Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.3149068Z return func(*args, **kwargs) 2025-12-04T08:56:50.3149757Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 179, in forward 2025-12-04T08:56:50.3150569Z attn_output, attn_weights = attention_interface( 2025-12-04T08:56:50.3151471Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T08:56:50.3152465Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T08:56:50.3152868Z 2025-12-04T08:56:50.3153074Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:50.3153794Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:50.3154441Z res = mod(**inputs) 2025-12-04T08:56:50.3155086Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3155800Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3156569Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T08:56:50.3157317Z outputs = self.model.decoder( 2025-12-04T08:56:50.3157981Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3158784Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3159558Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T08:56:50.3160326Z layer_outputs = decoder_layer( 2025-12-04T08:56:50.3161025Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:56:50.3161752Z return super().__call__(*args, **kwargs) 2025-12-04T08:56:50.3162525Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.3163274Z return func(*args, **kwargs) 2025-12-04T08:56:50.3163982Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T08:56:50.3164759Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:56:50.3165587Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.3166323Z return func(*args, **kwargs) 2025-12-04T08:56:50.3167050Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 191, in forward 2025-12-04T08:56:50.3167805Z attn_output = self.out_proj(attn_output) 2025-12-04T08:56:50.3168071Z 2025-12-04T08:56:50.3168328Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:50.3169045Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:50.3169670Z res = mod(**inputs) 2025-12-04T08:56:50.3170315Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3171014Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3171771Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T08:56:50.3172522Z outputs = self.model.decoder( 2025-12-04T08:56:50.3173187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3173909Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3174684Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T08:56:50.3175432Z layer_outputs = decoder_layer( 2025-12-04T08:56:50.3176121Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:56:50.3176834Z return super().__call__(*args, **kwargs) 2025-12-04T08:56:50.3177596Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.3178352Z return func(*args, **kwargs) 2025-12-04T08:56:50.3179051Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 281, in forward 2025-12-04T08:56:50.3179774Z hidden_states = self.fc1(hidden_states) 2025-12-04T08:56:50.3180073Z 2025-12-04T08:56:50.3180275Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:50.3180966Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:50.3181605Z res = mod(**inputs) 2025-12-04T08:56:50.3182251Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3182961Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3183702Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T08:56:50.3184466Z outputs = self.model.decoder( 2025-12-04T08:56:50.3185160Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3185887Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3186629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T08:56:50.3187384Z layer_outputs = decoder_layer( 2025-12-04T08:56:50.3188052Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:56:50.3188737Z return super().__call__(*args, **kwargs) 2025-12-04T08:56:50.3189454Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.3190136Z return func(*args, **kwargs) 2025-12-04T08:56:50.3190772Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 282, in forward 2025-12-04T08:56:50.3191510Z hidden_states = self.activation_fn(hidden_states) 2025-12-04T08:56:50.3191801Z 2025-12-04T08:56:50.3191975Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:50.3192644Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:50.3193268Z res = mod(**inputs) 2025-12-04T08:56:50.3193875Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3194500Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3195300Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T08:56:50.3196038Z outputs = self.model.decoder( 2025-12-04T08:56:50.3196719Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3197411Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3198242Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T08:56:50.3198975Z layer_outputs = decoder_layer( 2025-12-04T08:56:50.3199641Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:56:50.3200468Z return super().__call__(*args, **kwargs) 2025-12-04T08:56:50.3201238Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.3201979Z return func(*args, **kwargs) 2025-12-04T08:56:50.3202708Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 284, in forward 2025-12-04T08:56:50.3203458Z hidden_states = self.fc2(hidden_states) 2025-12-04T08:56:50.3203710Z 2025-12-04T08:56:50.3203901Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:50.3204603Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:50.3205229Z res = mod(**inputs) 2025-12-04T08:56:50.3205845Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3206596Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3207348Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T08:56:50.3208090Z outputs = self.model.decoder( 2025-12-04T08:56:50.3208741Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3209416Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3210158Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T08:56:50.3210892Z layer_outputs = decoder_layer( 2025-12-04T08:56:50.3211573Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:56:50.3212282Z return super().__call__(*args, **kwargs) 2025-12-04T08:56:50.3213022Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.3213750Z return func(*args, **kwargs) 2025-12-04T08:56:50.3214462Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 287, in forward 2025-12-04T08:56:50.3215321Z hidden_states = (residual + hidden_states).view(hidden_states_shape) 2025-12-04T08:56:50.3215679Z 2025-12-04T08:56:50.3215887Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:50.3216578Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:50.3217215Z res = mod(**inputs) 2025-12-04T08:56:50.3217839Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3218529Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3219271Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T08:56:50.3220026Z outputs = self.model.decoder( 2025-12-04T08:56:50.3220920Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3221601Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3222496Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T08:56:50.3223180Z layer_outputs = decoder_layer( 2025-12-04T08:56:50.3223800Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:56:50.3224463Z return super().__call__(*args, **kwargs) 2025-12-04T08:56:50.3225187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.3225901Z return func(*args, **kwargs) 2025-12-04T08:56:50.3226590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T08:56:50.3227473Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:56:50.3228312Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.3229043Z return func(*args, **kwargs) 2025-12-04T08:56:50.3229752Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 160, in forward 2025-12-04T08:56:50.3230560Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T08:56:50.3230876Z 2025-12-04T08:56:50.3231079Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:50.3231781Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:50.3232409Z res = mod(**inputs) 2025-12-04T08:56:50.3233058Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3233848Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3234596Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T08:56:50.3235349Z outputs = self.model.decoder( 2025-12-04T08:56:50.3236048Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3236766Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3237526Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T08:56:50.3238435Z layer_outputs = decoder_layer( 2025-12-04T08:56:50.3239148Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:56:50.3239882Z return super().__call__(*args, **kwargs) 2025-12-04T08:56:50.3240677Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.3241448Z return func(*args, **kwargs) 2025-12-04T08:56:50.3242166Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T08:56:50.3242995Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:56:50.3243817Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.3244567Z return func(*args, **kwargs) 2025-12-04T08:56:50.3245306Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 163, in forward 2025-12-04T08:56:50.3246102Z key_states = self.k_proj(hidden_states) 2025-12-04T08:56:50.3246371Z 2025-12-04T08:56:50.3246582Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:50.3247311Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:50.3247958Z res = mod(**inputs) 2025-12-04T08:56:50.3248609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3249341Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3250157Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T08:56:50.3250889Z outputs = self.model.decoder( 2025-12-04T08:56:50.3251532Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3252242Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3252984Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T08:56:50.3253734Z layer_outputs = decoder_layer( 2025-12-04T08:56:50.3254412Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:56:50.3255195Z return super().__call__(*args, **kwargs) 2025-12-04T08:56:50.3255964Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.3256713Z return func(*args, **kwargs) 2025-12-04T08:56:50.3257428Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T08:56:50.3258216Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:56:50.3259006Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.3259765Z return func(*args, **kwargs) 2025-12-04T08:56:50.3260478Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 164, in forward 2025-12-04T08:56:50.3261253Z value_states = self.v_proj(hidden_states) 2025-12-04T08:56:50.3261587Z 2025-12-04T08:56:50.3261733Z cudagraph partition due to non gpu ops 2025-12-04T08:56:50.3262130Z cudagraph partition due to non gpu ops 2025-12-04T08:56:50.3262565Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:50.3263281Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:50.3263912Z res = mod(**inputs) 2025-12-04T08:56:50.3264512Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3265200Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3265923Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T08:56:50.3266673Z outputs = self.model.decoder( 2025-12-04T08:56:50.3267322Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3268040Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3268792Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T08:56:50.3269522Z layer_outputs = decoder_layer( 2025-12-04T08:56:50.3270200Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:56:50.3270908Z return super().__call__(*args, **kwargs) 2025-12-04T08:56:50.3271658Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.3272384Z return func(*args, **kwargs) 2025-12-04T08:56:50.3273100Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T08:56:50.3273892Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:56:50.3274678Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.3275413Z return func(*args, **kwargs) 2025-12-04T08:56:50.3276145Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 179, in forward 2025-12-04T08:56:50.3277016Z attn_output, attn_weights = attention_interface( 2025-12-04T08:56:50.3277930Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T08:56:50.3279026Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T08:56:50.3279407Z 2025-12-04T08:56:50.3279606Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:50.3280333Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:50.3280972Z res = mod(**inputs) 2025-12-04T08:56:50.3281631Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3282361Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3283210Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T08:56:50.3284098Z outputs = self.model.decoder( 2025-12-04T08:56:50.3284909Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3285666Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3286498Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T08:56:50.3287357Z layer_outputs = decoder_layer( 2025-12-04T08:56:50.3288180Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:56:50.3288908Z return super().__call__(*args, **kwargs) 2025-12-04T08:56:50.3289790Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.3290558Z return func(*args, **kwargs) 2025-12-04T08:56:50.3291298Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T08:56:50.3292107Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:56:50.3292898Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.3293636Z return func(*args, **kwargs) 2025-12-04T08:56:50.3294356Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 191, in forward 2025-12-04T08:56:50.3295138Z attn_output = self.out_proj(attn_output) 2025-12-04T08:56:50.3295419Z 2025-12-04T08:56:50.3295611Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:50.3296340Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:50.3297011Z res = mod(**inputs) 2025-12-04T08:56:50.3297653Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3298350Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3299102Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T08:56:50.3299831Z outputs = self.model.decoder( 2025-12-04T08:56:50.3300497Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3301191Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3301932Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T08:56:50.3302666Z layer_outputs = decoder_layer( 2025-12-04T08:56:50.3303350Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:56:50.3304071Z return super().__call__(*args, **kwargs) 2025-12-04T08:56:50.3304893Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.3305636Z return func(*args, **kwargs) 2025-12-04T08:56:50.3306352Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 281, in forward 2025-12-04T08:56:50.3307115Z hidden_states = self.fc1(hidden_states) 2025-12-04T08:56:50.3307374Z 2025-12-04T08:56:50.3307571Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:50.3308273Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:50.3308895Z res = mod(**inputs) 2025-12-04T08:56:50.3309511Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3310205Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3310993Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T08:56:50.3311772Z outputs = self.model.decoder( 2025-12-04T08:56:50.3312448Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3313151Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3313891Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T08:56:50.3314612Z layer_outputs = decoder_layer( 2025-12-04T08:56:50.3315279Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:56:50.3316000Z return super().__call__(*args, **kwargs) 2025-12-04T08:56:50.3316764Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.3317543Z return func(*args, **kwargs) 2025-12-04T08:56:50.3318379Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 282, in forward 2025-12-04T08:56:50.3319213Z hidden_states = self.activation_fn(hidden_states) 2025-12-04T08:56:50.3319525Z 2025-12-04T08:56:50.3319730Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:50.3320434Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:50.3321199Z res = mod(**inputs) 2025-12-04T08:56:50.3321716Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3322123Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3322545Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T08:56:50.3322975Z outputs = self.model.decoder( 2025-12-04T08:56:50.3323363Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3323748Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3324168Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T08:56:50.3324583Z layer_outputs = decoder_layer( 2025-12-04T08:56:50.3324964Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:56:50.3325369Z return super().__call__(*args, **kwargs) 2025-12-04T08:56:50.3325792Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.3326218Z return func(*args, **kwargs) 2025-12-04T08:56:50.3326612Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 284, in forward 2025-12-04T08:56:50.3327042Z hidden_states = self.fc2(hidden_states) 2025-12-04T08:56:50.3327196Z 2025-12-04T08:56:50.3327323Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:50.3327828Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:50.3328194Z res = mod(**inputs) 2025-12-04T08:56:50.3328557Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3328951Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3329355Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T08:56:50.3329774Z outputs = self.model.decoder( 2025-12-04T08:56:50.3330159Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3330548Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3330998Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T08:56:50.3331440Z layer_outputs = decoder_layer( 2025-12-04T08:56:50.3331824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:56:50.3332220Z return super().__call__(*args, **kwargs) 2025-12-04T08:56:50.3332649Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.3333068Z return func(*args, **kwargs) 2025-12-04T08:56:50.3333486Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T08:56:50.3333921Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:56:50.3334356Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.3334790Z return func(*args, **kwargs) 2025-12-04T08:56:50.3335190Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 160, in forward 2025-12-04T08:56:50.3335639Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T08:56:50.3335830Z 2025-12-04T08:56:50.3335945Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:50.3336342Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:50.3336686Z res = mod(**inputs) 2025-12-04T08:56:50.3337044Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3337443Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3337854Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T08:56:50.3338263Z outputs = self.model.decoder( 2025-12-04T08:56:50.3338641Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3339030Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3339433Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T08:56:50.3339848Z layer_outputs = decoder_layer( 2025-12-04T08:56:50.3340231Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:56:50.3340631Z return super().__call__(*args, **kwargs) 2025-12-04T08:56:50.3341038Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.3341446Z return func(*args, **kwargs) 2025-12-04T08:56:50.3341839Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T08:56:50.3342270Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:56:50.3342706Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.3343141Z return func(*args, **kwargs) 2025-12-04T08:56:50.3343541Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 163, in forward 2025-12-04T08:56:50.3343953Z key_states = self.k_proj(hidden_states) 2025-12-04T08:56:50.3344114Z 2025-12-04T08:56:50.3344229Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:50.3344622Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:50.3344976Z res = mod(**inputs) 2025-12-04T08:56:50.3345328Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3345719Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3346154Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T08:56:50.3346591Z outputs = self.model.decoder( 2025-12-04T08:56:50.3346994Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3347381Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3347788Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T08:56:50.3348204Z layer_outputs = decoder_layer( 2025-12-04T08:56:50.3348600Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:56:50.3349010Z return super().__call__(*args, **kwargs) 2025-12-04T08:56:50.3349438Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.3349882Z return func(*args, **kwargs) 2025-12-04T08:56:50.3350294Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T08:56:50.3350746Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:56:50.3351201Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.3351631Z return func(*args, **kwargs) 2025-12-04T08:56:50.3352039Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 164, in forward 2025-12-04T08:56:50.3352475Z value_states = self.v_proj(hidden_states) 2025-12-04T08:56:50.3352633Z 2025-12-04T08:56:50.3352729Z cudagraph partition due to non gpu ops 2025-12-04T08:56:50.3352972Z cudagraph partition due to non gpu ops 2025-12-04T08:56:50.3353242Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:50.3353646Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:50.3354016Z res = mod(**inputs) 2025-12-04T08:56:50.3354395Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3354804Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3355224Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T08:56:50.3355649Z outputs = self.model.decoder( 2025-12-04T08:56:50.3356039Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3356429Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3356856Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T08:56:50.3357283Z layer_outputs = decoder_layer( 2025-12-04T08:56:50.3357681Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:56:50.3358236Z return super().__call__(*args, **kwargs) 2025-12-04T08:56:50.3358734Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.3359169Z return func(*args, **kwargs) 2025-12-04T08:56:50.3359574Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T08:56:50.3360032Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:56:50.3360480Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.3360948Z return func(*args, **kwargs) 2025-12-04T08:56:50.3361349Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 179, in forward 2025-12-04T08:56:50.3361830Z attn_output, attn_weights = attention_interface( 2025-12-04T08:56:50.3362380Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T08:56:50.3362944Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T08:56:50.3363154Z 2025-12-04T08:56:50.3363277Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:50.3363699Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:50.3364065Z res = mod(**inputs) 2025-12-04T08:56:50.3364437Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3364859Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3365281Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T08:56:50.3365722Z outputs = self.model.decoder( 2025-12-04T08:56:50.3366104Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3366502Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3366925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T08:56:50.3367337Z layer_outputs = decoder_layer( 2025-12-04T08:56:50.3367727Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:56:50.3368131Z return super().__call__(*args, **kwargs) 2025-12-04T08:56:50.3368553Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.3368960Z return func(*args, **kwargs) 2025-12-04T08:56:50.3369363Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T08:56:50.3369815Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:56:50.3370267Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.3370683Z return func(*args, **kwargs) 2025-12-04T08:56:50.3371098Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 191, in forward 2025-12-04T08:56:50.3371543Z attn_output = self.out_proj(attn_output) 2025-12-04T08:56:50.3371696Z 2025-12-04T08:56:50.3371815Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:50.3372215Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:50.3372577Z res = mod(**inputs) 2025-12-04T08:56:50.3372939Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3373327Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3373743Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T08:56:50.3374167Z outputs = self.model.decoder( 2025-12-04T08:56:50.3374568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3374959Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3375371Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T08:56:50.3375784Z layer_outputs = decoder_layer( 2025-12-04T08:56:50.3376163Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:56:50.3376563Z return super().__call__(*args, **kwargs) 2025-12-04T08:56:50.3376984Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.3377406Z return func(*args, **kwargs) 2025-12-04T08:56:50.3377822Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 281, in forward 2025-12-04T08:56:50.3378244Z hidden_states = self.fc1(hidden_states) 2025-12-04T08:56:50.3378395Z 2025-12-04T08:56:50.3378518Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:50.3378906Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:50.3379260Z res = mod(**inputs) 2025-12-04T08:56:50.3379624Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3380019Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3380424Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T08:56:50.3380868Z outputs = self.model.decoder( 2025-12-04T08:56:50.3381262Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3381661Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3382101Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T08:56:50.3382529Z layer_outputs = decoder_layer( 2025-12-04T08:56:50.3382930Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:56:50.3383340Z return super().__call__(*args, **kwargs) 2025-12-04T08:56:50.3383772Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.3384206Z return func(*args, **kwargs) 2025-12-04T08:56:50.3384613Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 282, in forward 2025-12-04T08:56:50.3385077Z hidden_states = self.activation_fn(hidden_states) 2025-12-04T08:56:50.3385260Z 2025-12-04T08:56:50.3385379Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:50.3385788Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:50.3386150Z res = mod(**inputs) 2025-12-04T08:56:50.3386522Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3386937Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3387355Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T08:56:50.3387785Z outputs = self.model.decoder( 2025-12-04T08:56:50.3388180Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3388585Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3389006Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T08:56:50.3389438Z layer_outputs = decoder_layer( 2025-12-04T08:56:50.3389864Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:56:50.3390276Z return super().__call__(*args, **kwargs) 2025-12-04T08:56:50.3390697Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.3391118Z return func(*args, **kwargs) 2025-12-04T08:56:50.3391528Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 284, in forward 2025-12-04T08:56:50.3391953Z hidden_states = self.fc2(hidden_states) 2025-12-04T08:56:50.3392116Z 2025-12-04T08:56:50.3392232Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:50.3392661Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:50.3393045Z res = mod(**inputs) 2025-12-04T08:56:50.3393410Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3393821Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3394250Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T08:56:50.3394677Z outputs = self.model.decoder( 2025-12-04T08:56:50.3395082Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3395482Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3395913Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T08:56:50.3396348Z layer_outputs = decoder_layer( 2025-12-04T08:56:50.3396746Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:56:50.3397161Z return super().__call__(*args, **kwargs) 2025-12-04T08:56:50.3397594Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.3398007Z return func(*args, **kwargs) 2025-12-04T08:56:50.3398536Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 287, in forward 2025-12-04T08:56:50.3399037Z hidden_states = (residual + hidden_states).view(hidden_states_shape) 2025-12-04T08:56:50.3399253Z 2025-12-04T08:56:50.3399373Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:50.3399783Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:50.3400155Z res = mod(**inputs) 2025-12-04T08:56:50.3400530Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3400938Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3401368Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T08:56:50.3401796Z outputs = self.model.decoder( 2025-12-04T08:56:50.3402186Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3402581Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3402993Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T08:56:50.3403403Z layer_outputs = decoder_layer( 2025-12-04T08:56:50.3403781Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:56:50.3404184Z return super().__call__(*args, **kwargs) 2025-12-04T08:56:50.3404604Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.3405007Z return func(*args, **kwargs) 2025-12-04T08:56:50.3405506Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T08:56:50.3405950Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:56:50.3406390Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.3406802Z return func(*args, **kwargs) 2025-12-04T08:56:50.3407196Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 160, in forward 2025-12-04T08:56:50.3407653Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T08:56:50.3407839Z 2025-12-04T08:56:50.3407964Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:50.3408385Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:50.3408780Z res = mod(**inputs) 2025-12-04T08:56:50.3409160Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3409547Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3409960Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T08:56:50.3410377Z outputs = self.model.decoder( 2025-12-04T08:56:50.3410757Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3411149Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3411562Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T08:56:50.3412000Z layer_outputs = decoder_layer( 2025-12-04T08:56:50.3412377Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:56:50.3412778Z return super().__call__(*args, **kwargs) 2025-12-04T08:56:50.3413209Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.3413621Z return func(*args, **kwargs) 2025-12-04T08:56:50.3414012Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T08:56:50.3414456Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:56:50.3414899Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.3415314Z return func(*args, **kwargs) 2025-12-04T08:56:50.3415717Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 163, in forward 2025-12-04T08:56:50.3416147Z key_states = self.k_proj(hidden_states) 2025-12-04T08:56:50.3416299Z 2025-12-04T08:56:50.3416423Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:50.3416808Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:50.3417162Z res = mod(**inputs) 2025-12-04T08:56:50.3417527Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3417922Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3418325Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T08:56:50.3418740Z outputs = self.model.decoder( 2025-12-04T08:56:50.3419116Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3419498Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3419909Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T08:56:50.3420318Z layer_outputs = decoder_layer( 2025-12-04T08:56:50.3420993Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:56:50.3421401Z return super().__call__(*args, **kwargs) 2025-12-04T08:56:50.3421822Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.3422245Z return func(*args, **kwargs) 2025-12-04T08:56:50.3422635Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T08:56:50.3423080Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:56:50.3423526Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.3423940Z return func(*args, **kwargs) 2025-12-04T08:56:50.3424405Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 164, in forward 2025-12-04T08:56:50.3424833Z value_states = self.v_proj(hidden_states) 2025-12-04T08:56:50.3424996Z 2025-12-04T08:56:50.3425088Z cudagraph partition due to non gpu ops 2025-12-04T08:56:50.3425324Z cudagraph partition due to non gpu ops 2025-12-04T08:56:50.3425581Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:50.3425984Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:50.3426339Z res = mod(**inputs) 2025-12-04T08:56:50.3426695Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3427087Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3427537Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T08:56:50.3427961Z outputs = self.model.decoder( 2025-12-04T08:56:50.3428340Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3428734Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3429149Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T08:56:50.3429561Z layer_outputs = decoder_layer( 2025-12-04T08:56:50.3429957Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:56:50.3430362Z return super().__call__(*args, **kwargs) 2025-12-04T08:56:50.3430786Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.3431202Z return func(*args, **kwargs) 2025-12-04T08:56:50.3431613Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T08:56:50.3432066Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:56:50.3432506Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.3432918Z return func(*args, **kwargs) 2025-12-04T08:56:50.3433323Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 179, in forward 2025-12-04T08:56:50.3433773Z attn_output, attn_weights = attention_interface( 2025-12-04T08:56:50.3434266Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T08:56:50.3434812Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T08:56:50.3435026Z 2025-12-04T08:56:50.3435144Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:50.3435546Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:50.3435903Z res = mod(**inputs) 2025-12-04T08:56:50.3436286Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3436689Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3437104Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T08:56:50.3437547Z outputs = self.model.decoder( 2025-12-04T08:56:50.3437934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3438434Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3438859Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T08:56:50.3439294Z layer_outputs = decoder_layer( 2025-12-04T08:56:50.3439720Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:56:50.3440147Z return super().__call__(*args, **kwargs) 2025-12-04T08:56:50.3440574Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.3440985Z return func(*args, **kwargs) 2025-12-04T08:56:50.3441381Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T08:56:50.3441826Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:56:50.3442276Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.3442704Z return func(*args, **kwargs) 2025-12-04T08:56:50.3443113Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 191, in forward 2025-12-04T08:56:50.3443567Z attn_output = self.out_proj(attn_output) 2025-12-04T08:56:50.3443734Z 2025-12-04T08:56:50.3443850Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:50.3444256Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:50.3444626Z res = mod(**inputs) 2025-12-04T08:56:50.3444997Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3445406Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3445827Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T08:56:50.3446245Z outputs = self.model.decoder( 2025-12-04T08:56:50.3446633Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3447031Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3447447Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T08:56:50.3447875Z layer_outputs = decoder_layer( 2025-12-04T08:56:50.3448275Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:56:50.3448650Z return super().__call__(*args, **kwargs) 2025-12-04T08:56:50.3449043Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.3449421Z return func(*args, **kwargs) 2025-12-04T08:56:50.3449787Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 281, in forward 2025-12-04T08:56:50.3450173Z hidden_states = self.fc1(hidden_states) 2025-12-04T08:56:50.3450313Z 2025-12-04T08:56:50.3450421Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:50.3450786Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:50.3451114Z res = mod(**inputs) 2025-12-04T08:56:50.3451464Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3451835Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3452229Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T08:56:50.3452623Z outputs = self.model.decoder( 2025-12-04T08:56:50.3452995Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3453392Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3453784Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T08:56:50.3454170Z layer_outputs = decoder_layer( 2025-12-04T08:56:50.3454566Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:56:50.3454961Z return super().__call__(*args, **kwargs) 2025-12-04T08:56:50.3455345Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.3455714Z return func(*args, **kwargs) 2025-12-04T08:56:50.3456079Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 282, in forward 2025-12-04T08:56:50.3456484Z hidden_states = self.activation_fn(hidden_states) 2025-12-04T08:56:50.3456637Z 2025-12-04T08:56:50.3456748Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:50.3457101Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:50.3457435Z res = mod(**inputs) 2025-12-04T08:56:50.3457794Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3458156Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3458553Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T08:56:50.3458938Z outputs = self.model.decoder( 2025-12-04T08:56:50.3459288Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3459641Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3460029Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T08:56:50.3460426Z layer_outputs = decoder_layer( 2025-12-04T08:56:50.3460773Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:56:50.3461144Z return super().__call__(*args, **kwargs) 2025-12-04T08:56:50.3461553Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.3461964Z return func(*args, **kwargs) 2025-12-04T08:56:50.3462358Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 284, in forward 2025-12-04T08:56:50.3462784Z hidden_states = self.fc2(hidden_states) 2025-12-04T08:56:50.3462925Z 2025-12-04T08:56:50.3463039Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:50.3463404Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:50.3463734Z res = mod(**inputs) 2025-12-04T08:56:50.3464070Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3464450Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3464827Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T08:56:50.3465220Z outputs = self.model.decoder( 2025-12-04T08:56:50.3465595Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3465960Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3466365Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T08:56:50.3466777Z layer_outputs = decoder_layer( 2025-12-04T08:56:50.3467165Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:56:50.3467560Z return super().__call__(*args, **kwargs) 2025-12-04T08:56:50.3467959Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.3468352Z return func(*args, **kwargs) 2025-12-04T08:56:50.3468752Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T08:56:50.3469176Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:56:50.3469609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.3469997Z return func(*args, **kwargs) 2025-12-04T08:56:50.3470249Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 160, in forward 2025-12-04T08:56:50.3470367Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T08:56:50.3470382Z 2025-12-04T08:56:50.3470490Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:50.3470703Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:50.3470778Z res = mod(**inputs) 2025-12-04T08:56:50.3471023Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3471103Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3471360Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T08:56:50.3471436Z outputs = self.model.decoder( 2025-12-04T08:56:50.3471667Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3471743Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3471990Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T08:56:50.3472074Z layer_outputs = decoder_layer( 2025-12-04T08:56:50.3472305Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:56:50.3472388Z return super().__call__(*args, **kwargs) 2025-12-04T08:56:50.3472647Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.3472722Z return func(*args, **kwargs) 2025-12-04T08:56:50.3472976Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T08:56:50.3473078Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:56:50.3473327Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.3473406Z return func(*args, **kwargs) 2025-12-04T08:56:50.3473653Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 163, in forward 2025-12-04T08:56:50.3473737Z key_states = self.k_proj(hidden_states) 2025-12-04T08:56:50.3473747Z 2025-12-04T08:56:50.3473854Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:50.3474059Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:50.3474135Z res = mod(**inputs) 2025-12-04T08:56:50.3474394Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3474471Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3474722Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T08:56:50.3474798Z outputs = self.model.decoder( 2025-12-04T08:56:50.3475030Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3475105Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3475350Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T08:56:50.3475437Z layer_outputs = decoder_layer( 2025-12-04T08:56:50.3475684Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:56:50.3475782Z return super().__call__(*args, **kwargs) 2025-12-04T08:56:50.3476044Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.3476117Z return func(*args, **kwargs) 2025-12-04T08:56:50.3476373Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T08:56:50.3476474Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:56:50.3476724Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.3476804Z return func(*args, **kwargs) 2025-12-04T08:56:50.3477054Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 164, in forward 2025-12-04T08:56:50.3477167Z value_states = self.v_proj(hidden_states) 2025-12-04T08:56:50.3477174Z 2025-12-04T08:56:50.3477258Z cudagraph partition due to non gpu ops 2025-12-04T08:56:50.3477340Z cudagraph partition due to non gpu ops 2025-12-04T08:56:50.3477455Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:50.3477660Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:50.3477727Z res = mod(**inputs) 2025-12-04T08:56:50.3477959Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3478112Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3478386Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T08:56:50.3478464Z outputs = self.model.decoder( 2025-12-04T08:56:50.3478695Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3478780Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3479043Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T08:56:50.3479124Z layer_outputs = decoder_layer( 2025-12-04T08:56:50.3479377Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:56:50.3479463Z return super().__call__(*args, **kwargs) 2025-12-04T08:56:50.3479745Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.3479819Z return func(*args, **kwargs) 2025-12-04T08:56:50.3480082Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T08:56:50.3480197Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:56:50.3480469Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.3480541Z return func(*args, **kwargs) 2025-12-04T08:56:50.3480824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 179, in forward 2025-12-04T08:56:50.3480929Z attn_output, attn_weights = attention_interface( 2025-12-04T08:56:50.3481240Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T08:56:50.3481379Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T08:56:50.3481383Z 2025-12-04T08:56:50.3481489Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:50.3481708Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:50.3481775Z res = mod(**inputs) 2025-12-04T08:56:50.3482022Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3482117Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3482367Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T08:56:50.3482450Z outputs = self.model.decoder( 2025-12-04T08:56:50.3482672Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3482747Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3483001Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T08:56:50.3483075Z layer_outputs = decoder_layer( 2025-12-04T08:56:50.3483310Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:56:50.3483412Z return super().__call__(*args, **kwargs) 2025-12-04T08:56:50.3483676Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.3483759Z return func(*args, **kwargs) 2025-12-04T08:56:50.3484003Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T08:56:50.3484112Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:56:50.3484357Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.3484429Z return func(*args, **kwargs) 2025-12-04T08:56:50.3484681Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 191, in forward 2025-12-04T08:56:50.3484766Z attn_output = self.out_proj(attn_output) 2025-12-04T08:56:50.3484770Z 2025-12-04T08:56:50.3484877Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:50.3485093Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:50.3485164Z res = mod(**inputs) 2025-12-04T08:56:50.3485399Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3485479Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3485729Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T08:56:50.3485817Z outputs = self.model.decoder( 2025-12-04T08:56:50.3486042Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3486120Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3486379Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T08:56:50.3486458Z layer_outputs = decoder_layer( 2025-12-04T08:56:50.3486698Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:56:50.3486783Z return super().__call__(*args, **kwargs) 2025-12-04T08:56:50.3487053Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.3487134Z return func(*args, **kwargs) 2025-12-04T08:56:50.3487387Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 281, in forward 2025-12-04T08:56:50.3487476Z hidden_states = self.fc1(hidden_states) 2025-12-04T08:56:50.3487480Z 2025-12-04T08:56:50.3487587Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:50.3487789Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:50.3487860Z res = mod(**inputs) 2025-12-04T08:56:50.3488106Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3488199Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3488464Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T08:56:50.3488538Z outputs = self.model.decoder( 2025-12-04T08:56:50.3488768Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3488842Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3489088Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T08:56:50.3489169Z layer_outputs = decoder_layer( 2025-12-04T08:56:50.3489396Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:56:50.3489497Z return super().__call__(*args, **kwargs) 2025-12-04T08:56:50.3489754Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.3489826Z return func(*args, **kwargs) 2025-12-04T08:56:50.3490082Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 282, in forward 2025-12-04T08:56:50.3490181Z hidden_states = self.activation_fn(hidden_states) 2025-12-04T08:56:50.3490184Z 2025-12-04T08:56:50.3490288Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:50.3490500Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:50.3490576Z res = mod(**inputs) 2025-12-04T08:56:50.3490800Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3490872Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3491117Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T08:56:50.3491201Z outputs = self.model.decoder( 2025-12-04T08:56:50.3491428Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3491502Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3491760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T08:56:50.3491834Z layer_outputs = decoder_layer( 2025-12-04T08:56:50.3492068Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:56:50.3492148Z return super().__call__(*args, **kwargs) 2025-12-04T08:56:50.3492405Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.3492484Z return func(*args, **kwargs) 2025-12-04T08:56:50.3492724Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 284, in forward 2025-12-04T08:56:50.3492806Z hidden_states = self.fc2(hidden_states) 2025-12-04T08:56:50.3492816Z 2025-12-04T08:56:50.3492980Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:50.3493180Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:50.3493253Z res = mod(**inputs) 2025-12-04T08:56:50.3493474Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3493549Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3493804Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T08:56:50.3493878Z outputs = self.model.decoder( 2025-12-04T08:56:50.3494110Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3494201Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3494466Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T08:56:50.3494545Z layer_outputs = decoder_layer( 2025-12-04T08:56:50.3494771Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:56:50.3494850Z return super().__call__(*args, **kwargs) 2025-12-04T08:56:50.3495107Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.3495176Z return func(*args, **kwargs) 2025-12-04T08:56:50.3495425Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 287, in forward 2025-12-04T08:56:50.3495594Z hidden_states = (residual + hidden_states).view(hidden_states_shape) 2025-12-04T08:56:50.3495598Z 2025-12-04T08:56:50.3495705Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:50.3495921Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:50.3495987Z res = mod(**inputs) 2025-12-04T08:56:50.3496212Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3496293Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3496543Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T08:56:50.3496627Z outputs = self.model.decoder( 2025-12-04T08:56:50.3496856Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3496931Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3497189Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T08:56:50.3497266Z layer_outputs = decoder_layer( 2025-12-04T08:56:50.3497517Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:56:50.3497602Z return super().__call__(*args, **kwargs) 2025-12-04T08:56:50.3497917Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.3498006Z return func(*args, **kwargs) 2025-12-04T08:56:50.3498256Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T08:56:50.3498356Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:56:50.3498617Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.3498690Z return func(*args, **kwargs) 2025-12-04T08:56:50.3498951Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 160, in forward 2025-12-04T08:56:50.3499067Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T08:56:50.3499071Z 2025-12-04T08:56:50.3499197Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:50.3499413Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:50.3499479Z res = mod(**inputs) 2025-12-04T08:56:50.3499712Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3499787Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3500033Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T08:56:50.3500115Z outputs = self.model.decoder( 2025-12-04T08:56:50.3500342Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3500450Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3500712Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T08:56:50.3500785Z layer_outputs = decoder_layer( 2025-12-04T08:56:50.3501024Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:56:50.3501104Z return super().__call__(*args, **kwargs) 2025-12-04T08:56:50.3501356Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.3501433Z return func(*args, **kwargs) 2025-12-04T08:56:50.3501685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T08:56:50.3501801Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:56:50.3502062Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.3502133Z return func(*args, **kwargs) 2025-12-04T08:56:50.3502390Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 163, in forward 2025-12-04T08:56:50.3502472Z key_states = self.k_proj(hidden_states) 2025-12-04T08:56:50.3502476Z 2025-12-04T08:56:50.3502581Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:50.3502793Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:50.3502885Z res = mod(**inputs) 2025-12-04T08:56:50.3503117Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3503191Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3503445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T08:56:50.3503530Z outputs = self.model.decoder( 2025-12-04T08:56:50.3503754Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3503829Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3504085Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T08:56:50.3504157Z layer_outputs = decoder_layer( 2025-12-04T08:56:50.3504393Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:56:50.3504474Z return super().__call__(*args, **kwargs) 2025-12-04T08:56:50.3504724Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.3504804Z return func(*args, **kwargs) 2025-12-04T08:56:50.3505056Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T08:56:50.3505156Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:56:50.3505444Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.3505517Z return func(*args, **kwargs) 2025-12-04T08:56:50.3505776Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 164, in forward 2025-12-04T08:56:50.3505863Z value_states = self.v_proj(hidden_states) 2025-12-04T08:56:50.3505867Z 2025-12-04T08:56:50.3505950Z cudagraph partition due to non gpu ops 2025-12-04T08:56:50.3506039Z cudagraph partition due to non gpu ops 2025-12-04T08:56:50.3506144Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:50.3506357Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:50.3506426Z res = mod(**inputs) 2025-12-04T08:56:50.3506665Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3506766Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3507014Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T08:56:50.3507087Z outputs = self.model.decoder( 2025-12-04T08:56:50.3507318Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3507393Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3507648Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T08:56:50.3507721Z layer_outputs = decoder_layer( 2025-12-04T08:56:50.3507964Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:56:50.3508055Z return super().__call__(*args, **kwargs) 2025-12-04T08:56:50.3508306Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.3508376Z return func(*args, **kwargs) 2025-12-04T08:56:50.3508631Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T08:56:50.3508730Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:56:50.3508984Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.3509055Z return func(*args, **kwargs) 2025-12-04T08:56:50.3509300Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 179, in forward 2025-12-04T08:56:50.3509408Z attn_output, attn_weights = attention_interface( 2025-12-04T08:56:50.3509713Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T08:56:50.3509858Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T08:56:50.3509864Z 2025-12-04T08:56:50.3509968Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:50.3510171Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:50.3510242Z res = mod(**inputs) 2025-12-04T08:56:50.3510467Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3510541Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3510798Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T08:56:50.3510872Z outputs = self.model.decoder( 2025-12-04T08:56:50.3511105Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3511181Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3511448Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T08:56:50.3511531Z layer_outputs = decoder_layer( 2025-12-04T08:56:50.3511761Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:56:50.3511841Z return super().__call__(*args, **kwargs) 2025-12-04T08:56:50.3512100Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.3512169Z return func(*args, **kwargs) 2025-12-04T08:56:50.3512423Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T08:56:50.3512524Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:56:50.3512792Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.3512907Z return func(*args, **kwargs) 2025-12-04T08:56:50.3513162Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 191, in forward 2025-12-04T08:56:50.3513255Z attn_output = self.out_proj(attn_output) 2025-12-04T08:56:50.3513258Z 2025-12-04T08:56:50.3513363Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:50.3513566Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:50.3513639Z res = mod(**inputs) 2025-12-04T08:56:50.3513866Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3513942Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3514221Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T08:56:50.3514298Z outputs = self.model.decoder( 2025-12-04T08:56:50.3514533Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3514608Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3514859Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T08:56:50.3514942Z layer_outputs = decoder_layer( 2025-12-04T08:56:50.3515179Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:56:50.3515260Z return super().__call__(*args, **kwargs) 2025-12-04T08:56:50.3515521Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.3515593Z return func(*args, **kwargs) 2025-12-04T08:56:50.3515851Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 281, in forward 2025-12-04T08:56:50.3515935Z hidden_states = self.fc1(hidden_states) 2025-12-04T08:56:50.3515941Z 2025-12-04T08:56:50.3516046Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:50.3516260Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:50.3516325Z res = mod(**inputs) 2025-12-04T08:56:50.3516559Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3516634Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3516881Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T08:56:50.3516962Z outputs = self.model.decoder( 2025-12-04T08:56:50.3517193Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3517267Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3517539Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T08:56:50.3517614Z layer_outputs = decoder_layer( 2025-12-04T08:56:50.3517855Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:56:50.3517935Z return super().__call__(*args, **kwargs) 2025-12-04T08:56:50.3518286Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.3518376Z return func(*args, **kwargs) 2025-12-04T08:56:50.3518626Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 282, in forward 2025-12-04T08:56:50.3518731Z hidden_states = self.activation_fn(hidden_states) 2025-12-04T08:56:50.3518743Z 2025-12-04T08:56:50.3518877Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:50.3519130Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:50.3519216Z res = mod(**inputs) 2025-12-04T08:56:50.3519482Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3519563Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3519847Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T08:56:50.3519927Z outputs = self.model.decoder( 2025-12-04T08:56:50.3520182Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3520262Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3520541Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T08:56:50.3520625Z layer_outputs = decoder_layer( 2025-12-04T08:56:50.3521133Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:56:50.3521225Z return super().__call__(*args, **kwargs) 2025-12-04T08:56:50.3521519Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.3521597Z return func(*args, **kwargs) 2025-12-04T08:56:50.3521876Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 284, in forward 2025-12-04T08:56:50.3521968Z hidden_states = self.fc2(hidden_states) 2025-12-04T08:56:50.3521972Z 2025-12-04T08:56:50.3522086Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:50.3522318Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:50.3522393Z res = mod(**inputs) 2025-12-04T08:56:50.3522640Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3522733Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3523004Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T08:56:50.3523095Z outputs = self.model.decoder( 2025-12-04T08:56:50.3523342Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3523423Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3523702Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T08:56:50.3523782Z layer_outputs = decoder_layer( 2025-12-04T08:56:50.3524041Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:56:50.3524130Z return super().__call__(*args, **kwargs) 2025-12-04T08:56:50.3524460Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.3524547Z return func(*args, **kwargs) 2025-12-04T08:56:50.3524817Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T08:56:50.3524926Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:56:50.3525204Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.3525280Z return func(*args, **kwargs) 2025-12-04T08:56:50.3525558Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 160, in forward 2025-12-04T08:56:50.3525685Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T08:56:50.3525689Z 2025-12-04T08:56:50.3525834Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:50.3526096Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:50.3526170Z res = mod(**inputs) 2025-12-04T08:56:50.3526425Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3526508Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3526785Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T08:56:50.3526874Z outputs = self.model.decoder( 2025-12-04T08:56:50.3527126Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3527206Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3527515Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T08:56:50.3527593Z layer_outputs = decoder_layer( 2025-12-04T08:56:50.3527832Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:56:50.3527914Z return super().__call__(*args, **kwargs) 2025-12-04T08:56:50.3528160Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.3528238Z return func(*args, **kwargs) 2025-12-04T08:56:50.3528486Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T08:56:50.3528587Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:56:50.3528845Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.3528918Z return func(*args, **kwargs) 2025-12-04T08:56:50.3529179Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 163, in forward 2025-12-04T08:56:50.3529264Z key_states = self.k_proj(hidden_states) 2025-12-04T08:56:50.3529268Z 2025-12-04T08:56:50.3529373Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:50.3529588Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:50.3529655Z res = mod(**inputs) 2025-12-04T08:56:50.3529891Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3529966Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3530213Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T08:56:50.3530293Z outputs = self.model.decoder( 2025-12-04T08:56:50.3530521Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3530598Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3530873Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T08:56:50.3530949Z layer_outputs = decoder_layer( 2025-12-04T08:56:50.3531187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:56:50.3531270Z return super().__call__(*args, **kwargs) 2025-12-04T08:56:50.3531519Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.3531598Z return func(*args, **kwargs) 2025-12-04T08:56:50.3531845Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T08:56:50.3531946Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:56:50.3532223Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.3532310Z return func(*args, **kwargs) 2025-12-04T08:56:50.3532568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 164, in forward 2025-12-04T08:56:50.3532657Z value_states = self.v_proj(hidden_states) 2025-12-04T08:56:50.3532661Z 2025-12-04T08:56:50.3532744Z cudagraph partition due to non gpu ops 2025-12-04T08:56:50.3532834Z cudagraph partition due to non gpu ops 2025-12-04T08:56:50.3532939Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:50.3533142Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:50.3533217Z res = mod(**inputs) 2025-12-04T08:56:50.3533442Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3533541Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3533789Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T08:56:50.3533863Z outputs = self.model.decoder( 2025-12-04T08:56:50.3534093Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3534167Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3534423Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T08:56:50.3534498Z layer_outputs = decoder_layer( 2025-12-04T08:56:50.3534724Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:56:50.3534811Z return super().__call__(*args, **kwargs) 2025-12-04T08:56:50.3535061Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.3535135Z return func(*args, **kwargs) 2025-12-04T08:56:50.3535401Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T08:56:50.3535509Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:56:50.3535791Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.3535865Z return func(*args, **kwargs) 2025-12-04T08:56:50.3536126Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 179, in forward 2025-12-04T08:56:50.3536240Z attn_output, attn_weights = attention_interface( 2025-12-04T08:56:50.3536560Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T08:56:50.3536710Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T08:56:50.3536721Z 2025-12-04T08:56:50.3536833Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:50.3537071Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:50.3537159Z res = mod(**inputs) 2025-12-04T08:56:50.3537387Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3537464Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3537732Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T08:56:50.3537808Z outputs = self.model.decoder( 2025-12-04T08:56:50.3538042Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3538115Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3538391Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T08:56:50.3538496Z layer_outputs = decoder_layer( 2025-12-04T08:56:50.3538745Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:56:50.3538831Z return super().__call__(*args, **kwargs) 2025-12-04T08:56:50.3539116Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.3539191Z return func(*args, **kwargs) 2025-12-04T08:56:50.3539462Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T08:56:50.3539567Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:56:50.3539835Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.3539930Z return func(*args, **kwargs) 2025-12-04T08:56:50.3540178Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 191, in forward 2025-12-04T08:56:50.3540270Z attn_output = self.out_proj(attn_output) 2025-12-04T08:56:50.3540275Z 2025-12-04T08:56:50.3540380Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:50.3540582Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:50.3540654Z res = mod(**inputs) 2025-12-04T08:56:50.3540876Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3540951Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3541213Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T08:56:50.3541286Z outputs = self.model.decoder( 2025-12-04T08:56:50.3541517Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3541593Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3541842Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T08:56:50.3541923Z layer_outputs = decoder_layer( 2025-12-04T08:56:50.3542147Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:56:50.3542227Z return super().__call__(*args, **kwargs) 2025-12-04T08:56:50.3542484Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.3542552Z return func(*args, **kwargs) 2025-12-04T08:56:50.3542805Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 281, in forward 2025-12-04T08:56:50.3542888Z hidden_states = self.fc1(hidden_states) 2025-12-04T08:56:50.3542892Z 2025-12-04T08:56:50.3542997Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:50.3543212Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:50.3543298Z res = mod(**inputs) 2025-12-04T08:56:50.3543529Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3543603Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3543848Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T08:56:50.3543932Z outputs = self.model.decoder( 2025-12-04T08:56:50.3544153Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3544228Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3544485Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T08:56:50.3544608Z layer_outputs = decoder_layer( 2025-12-04T08:56:50.3544845Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:56:50.3544926Z return super().__call__(*args, **kwargs) 2025-12-04T08:56:50.3545175Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.3545258Z return func(*args, **kwargs) 2025-12-04T08:56:50.3545519Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 282, in forward 2025-12-04T08:56:50.3545625Z hidden_states = self.activation_fn(hidden_states) 2025-12-04T08:56:50.3545636Z 2025-12-04T08:56:50.3545746Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:50.3545971Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:50.3546062Z res = mod(**inputs) 2025-12-04T08:56:50.3546290Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3546367Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3546620Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T08:56:50.3546697Z outputs = self.model.decoder( 2025-12-04T08:56:50.3546927Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3547004Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3547248Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T08:56:50.3547331Z layer_outputs = decoder_layer( 2025-12-04T08:56:50.3547559Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:56:50.3547639Z return super().__call__(*args, **kwargs) 2025-12-04T08:56:50.3547900Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.3547971Z return func(*args, **kwargs) 2025-12-04T08:56:50.3548223Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 284, in forward 2025-12-04T08:56:50.3548305Z hidden_states = self.fc2(hidden_states) 2025-12-04T08:56:50.3548309Z 2025-12-04T08:56:50.3548415Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:50.3548623Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:50.3548689Z res = mod(**inputs) 2025-12-04T08:56:50.3548919Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3549003Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3549251Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T08:56:50.3549335Z outputs = self.model.decoder( 2025-12-04T08:56:50.3549572Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3549649Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3549906Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T08:56:50.3549979Z layer_outputs = decoder_layer( 2025-12-04T08:56:50.3550214Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:56:50.3550295Z return super().__call__(*args, **kwargs) 2025-12-04T08:56:50.3550542Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.3550641Z return func(*args, **kwargs) 2025-12-04T08:56:50.3550903Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 287, in forward 2025-12-04T08:56:50.3551044Z hidden_states = (residual + hidden_states).view(hidden_states_shape) 2025-12-04T08:56:50.3551047Z 2025-12-04T08:56:50.3551159Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:50.3551362Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:50.3551435Z res = mod(**inputs) 2025-12-04T08:56:50.3551658Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3551733Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3551988Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T08:56:50.3552080Z outputs = self.model.decoder( 2025-12-04T08:56:50.3552310Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3552399Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3552657Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T08:56:50.3552737Z layer_outputs = decoder_layer( 2025-12-04T08:56:50.3552971Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:56:50.3553051Z return super().__call__(*args, **kwargs) 2025-12-04T08:56:50.3553313Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.3553383Z return func(*args, **kwargs) 2025-12-04T08:56:50.3553644Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T08:56:50.3553747Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:56:50.3554003Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.3554079Z return func(*args, **kwargs) 2025-12-04T08:56:50.3554332Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 160, in forward 2025-12-04T08:56:50.3554446Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T08:56:50.3554458Z 2025-12-04T08:56:50.3554562Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:50.3554765Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:50.3554838Z res = mod(**inputs) 2025-12-04T08:56:50.3555066Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3555142Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3555410Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T08:56:50.3555506Z outputs = self.model.decoder( 2025-12-04T08:56:50.3555751Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3555829Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3556090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T08:56:50.3556175Z layer_outputs = decoder_layer( 2025-12-04T08:56:50.3556418Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:56:50.3556505Z return super().__call__(*args, **kwargs) 2025-12-04T08:56:50.3556780Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.3556874Z return func(*args, **kwargs) 2025-12-04T08:56:50.3557164Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T08:56:50.3557270Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:56:50.3557532Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.3557613Z return func(*args, **kwargs) 2025-12-04T08:56:50.3557877Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 163, in forward 2025-12-04T08:56:50.3557964Z key_states = self.k_proj(hidden_states) 2025-12-04T08:56:50.3557975Z 2025-12-04T08:56:50.3558176Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:50.3558409Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:50.3558513Z res = mod(**inputs) 2025-12-04T08:56:50.3558768Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3558852Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3559135Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T08:56:50.3559217Z outputs = self.model.decoder( 2025-12-04T08:56:50.3559476Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3559556Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3559819Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T08:56:50.3559905Z layer_outputs = decoder_layer( 2025-12-04T08:56:50.3560150Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:56:50.3560237Z return super().__call__(*args, **kwargs) 2025-12-04T08:56:50.3560513Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.3560589Z return func(*args, **kwargs) 2025-12-04T08:56:50.3560859Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T08:56:50.3560964Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:56:50.3561229Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.3561312Z return func(*args, **kwargs) 2025-12-04T08:56:50.3561575Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 164, in forward 2025-12-04T08:56:50.3561664Z value_states = self.v_proj(hidden_states) 2025-12-04T08:56:50.3561677Z 2025-12-04T08:56:50.3561761Z cudagraph partition due to non gpu ops 2025-12-04T08:56:50.3561846Z cudagraph partition due to non gpu ops 2025-12-04T08:56:50.3561959Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:50.3562188Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:50.3562256Z res = mod(**inputs) 2025-12-04T08:56:50.3562496Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3562572Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3562817Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T08:56:50.3562899Z outputs = self.model.decoder( 2025-12-04T08:56:50.3563129Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3563213Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3563476Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T08:56:50.3563565Z layer_outputs = decoder_layer( 2025-12-04T08:56:50.3563817Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:56:50.3563897Z return super().__call__(*args, **kwargs) 2025-12-04T08:56:50.3564151Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.3564221Z return func(*args, **kwargs) 2025-12-04T08:56:50.3564469Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T08:56:50.3564574Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:56:50.3564842Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.3564915Z return func(*args, **kwargs) 2025-12-04T08:56:50.3565174Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 179, in forward 2025-12-04T08:56:50.3565276Z attn_output, attn_weights = attention_interface( 2025-12-04T08:56:50.3565584Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T08:56:50.3565721Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T08:56:50.3565725Z 2025-12-04T08:56:50.3565828Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:50.3566040Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:50.3566106Z res = mod(**inputs) 2025-12-04T08:56:50.3566337Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3566416Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3566666Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T08:56:50.3566761Z outputs = self.model.decoder( 2025-12-04T08:56:50.3566985Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3567058Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3567308Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T08:56:50.3567381Z layer_outputs = decoder_layer( 2025-12-04T08:56:50.3567614Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:56:50.3567696Z return super().__call__(*args, **kwargs) 2025-12-04T08:56:50.3567949Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.3568033Z return func(*args, **kwargs) 2025-12-04T08:56:50.3568330Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T08:56:50.3568437Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:56:50.3568712Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.3568788Z return func(*args, **kwargs) 2025-12-04T08:56:50.3569068Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 191, in forward 2025-12-04T08:56:50.3569160Z attn_output = self.out_proj(attn_output) 2025-12-04T08:56:50.3569164Z 2025-12-04T08:56:50.3569278Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:50.3569507Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:50.3569597Z res = mod(**inputs) 2025-12-04T08:56:50.3569862Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3569944Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3570218Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T08:56:50.3570306Z outputs = self.model.decoder( 2025-12-04T08:56:50.3570541Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3570616Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3570869Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T08:56:50.3570941Z layer_outputs = decoder_layer( 2025-12-04T08:56:50.3571199Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:56:50.3571282Z return super().__call__(*args, **kwargs) 2025-12-04T08:56:50.3571529Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.3571605Z return func(*args, **kwargs) 2025-12-04T08:56:50.3571867Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 281, in forward 2025-12-04T08:56:50.3571954Z hidden_states = self.fc1(hidden_states) 2025-12-04T08:56:50.3571964Z 2025-12-04T08:56:50.3572073Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:50.3572289Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:50.3572365Z res = mod(**inputs) 2025-12-04T08:56:50.3572601Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3572682Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3572952Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T08:56:50.3573031Z outputs = self.model.decoder( 2025-12-04T08:56:50.3573273Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3573352Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3573611Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T08:56:50.3573696Z layer_outputs = decoder_layer( 2025-12-04T08:56:50.3573937Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:56:50.3574021Z return super().__call__(*args, **kwargs) 2025-12-04T08:56:50.3574292Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.3574370Z return func(*args, **kwargs) 2025-12-04T08:56:50.3574657Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 282, in forward 2025-12-04T08:56:50.3574773Z hidden_states = self.activation_fn(hidden_states) 2025-12-04T08:56:50.3574777Z 2025-12-04T08:56:50.3574882Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:50.3575100Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:50.3575167Z res = mod(**inputs) 2025-12-04T08:56:50.3575395Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3575478Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3575725Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T08:56:50.3575809Z outputs = self.model.decoder( 2025-12-04T08:56:50.3576061Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3576153Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3576410Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T08:56:50.3576484Z layer_outputs = decoder_layer( 2025-12-04T08:56:50.3576718Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:56:50.3576799Z return super().__call__(*args, **kwargs) 2025-12-04T08:56:50.3577050Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.3577127Z return func(*args, **kwargs) 2025-12-04T08:56:50.3577393Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 284, in forward 2025-12-04T08:56:50.3577477Z hidden_states = self.fc2(hidden_states) 2025-12-04T08:56:50.3577482Z 2025-12-04T08:56:50.3577594Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:50.3577799Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:50.3577871Z res = mod(**inputs) 2025-12-04T08:56:50.3578096Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3578173Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3578427Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T08:56:50.3578502Z outputs = self.model.decoder( 2025-12-04T08:56:50.3578724Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3578807Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3579056Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T08:56:50.3579137Z layer_outputs = decoder_layer( 2025-12-04T08:56:50.3579369Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:56:50.3579449Z return super().__call__(*args, **kwargs) 2025-12-04T08:56:50.3579709Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.3579778Z return func(*args, **kwargs) 2025-12-04T08:56:50.3580032Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T08:56:50.3580132Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:56:50.3580383Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.3580464Z return func(*args, **kwargs) 2025-12-04T08:56:50.3580732Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 160, in forward 2025-12-04T08:56:50.3580852Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T08:56:50.3580864Z 2025-12-04T08:56:50.3580975Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:50.3581188Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:50.3581264Z res = mod(**inputs) 2025-12-04T08:56:50.3581503Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3581583Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3581853Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T08:56:50.3581935Z outputs = self.model.decoder( 2025-12-04T08:56:50.3582196Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3582287Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3582534Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T08:56:50.3582615Z layer_outputs = decoder_layer( 2025-12-04T08:56:50.3582843Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:56:50.3582924Z return super().__call__(*args, **kwargs) 2025-12-04T08:56:50.3583185Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.3583256Z return func(*args, **kwargs) 2025-12-04T08:56:50.3583513Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T08:56:50.3583634Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:56:50.3583890Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.3583973Z return func(*args, **kwargs) 2025-12-04T08:56:50.3584235Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 163, in forward 2025-12-04T08:56:50.3584322Z key_states = self.k_proj(hidden_states) 2025-12-04T08:56:50.3584334Z 2025-12-04T08:56:50.3584444Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:50.3584661Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:50.3584738Z res = mod(**inputs) 2025-12-04T08:56:50.3584981Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3585063Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3585333Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T08:56:50.3585416Z outputs = self.model.decoder( 2025-12-04T08:56:50.3585662Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3585740Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3586003Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T08:56:50.3586088Z layer_outputs = decoder_layer( 2025-12-04T08:56:50.3586328Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:56:50.3586413Z return super().__call__(*args, **kwargs) 2025-12-04T08:56:50.3586684Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.3586761Z return func(*args, **kwargs) 2025-12-04T08:56:50.3587030Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T08:56:50.3587154Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:56:50.3587418Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.3587500Z return func(*args, **kwargs) 2025-12-04T08:56:50.3587760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 164, in forward 2025-12-04T08:56:50.3587852Z value_states = self.v_proj(hidden_states) 2025-12-04T08:56:50.3587862Z 2025-12-04T08:56:50.3587949Z cudagraph partition due to non gpu ops 2025-12-04T08:56:50.3588036Z cudagraph partition due to non gpu ops 2025-12-04T08:56:50.3588155Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:50.3588390Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:50.3588476Z res = mod(**inputs) 2025-12-04T08:56:50.3588732Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3588811Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3589076Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T08:56:50.3589164Z outputs = self.model.decoder( 2025-12-04T08:56:50.3589401Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3589487Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3589751Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T08:56:50.3589894Z layer_outputs = decoder_layer( 2025-12-04T08:56:50.3590146Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:56:50.3590232Z return super().__call__(*args, **kwargs) 2025-12-04T08:56:50.3590504Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.3590580Z return func(*args, **kwargs) 2025-12-04T08:56:50.3590845Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T08:56:50.3590959Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:56:50.3591225Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.3591301Z return func(*args, **kwargs) 2025-12-04T08:56:50.3591574Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 179, in forward 2025-12-04T08:56:50.3591684Z attn_output, attn_weights = attention_interface( 2025-12-04T08:56:50.3592018Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T08:56:50.3592165Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T08:56:50.3592169Z 2025-12-04T08:56:50.3592279Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:50.3592505Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:50.3592574Z res = mod(**inputs) 2025-12-04T08:56:50.3592823Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3592904Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3593167Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T08:56:50.3593257Z outputs = self.model.decoder( 2025-12-04T08:56:50.3593500Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3593606Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3593877Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T08:56:50.3593957Z layer_outputs = decoder_layer( 2025-12-04T08:56:50.3594206Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:56:50.3594291Z return super().__call__(*args, **kwargs) 2025-12-04T08:56:50.3594557Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.3594639Z return func(*args, **kwargs) 2025-12-04T08:56:50.3594903Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T08:56:50.3595064Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:56:50.3595339Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.3595413Z return func(*args, **kwargs) 2025-12-04T08:56:50.3595683Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 191, in forward 2025-12-04T08:56:50.3595772Z attn_output = self.out_proj(attn_output) 2025-12-04T08:56:50.3595776Z 2025-12-04T08:56:50.3595888Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:50.3596115Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:50.3596184Z res = mod(**inputs) 2025-12-04T08:56:50.3596429Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3596528Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3596793Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T08:56:50.3596883Z outputs = self.model.decoder( 2025-12-04T08:56:50.3597121Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3597200Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3597467Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T08:56:50.3597545Z layer_outputs = decoder_layer( 2025-12-04T08:56:50.3597792Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:56:50.3597878Z return super().__call__(*args, **kwargs) 2025-12-04T08:56:50.3598246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.3598344Z return func(*args, **kwargs) 2025-12-04T08:56:50.3598627Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 281, in forward 2025-12-04T08:56:50.3598718Z hidden_states = self.fc1(hidden_states) 2025-12-04T08:56:50.3598730Z 2025-12-04T08:56:50.3598845Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:50.3599070Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:50.3599150Z res = mod(**inputs) 2025-12-04T08:56:50.3599398Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3599480Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3599762Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T08:56:50.3599846Z outputs = self.model.decoder( 2025-12-04T08:56:50.3600099Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3600207Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3600479Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T08:56:50.3600575Z layer_outputs = decoder_layer( 2025-12-04T08:56:50.3600823Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:56:50.3600907Z return super().__call__(*args, **kwargs) 2025-12-04T08:56:50.3601176Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.3601248Z return func(*args, **kwargs) 2025-12-04T08:56:50.3601516Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 282, in forward 2025-12-04T08:56:50.3601638Z hidden_states = self.activation_fn(hidden_states) 2025-12-04T08:56:50.3601656Z 2025-12-04T08:56:50.3601765Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:50.3601990Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:50.3602060Z res = mod(**inputs) 2025-12-04T08:56:50.3602300Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3602386Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3602648Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T08:56:50.3602732Z outputs = self.model.decoder( 2025-12-04T08:56:50.3602969Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3603066Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3603339Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T08:56:50.3603418Z layer_outputs = decoder_layer( 2025-12-04T08:56:50.3603667Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:56:50.3603751Z return super().__call__(*args, **kwargs) 2025-12-04T08:56:50.3604013Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.3604093Z return func(*args, **kwargs) 2025-12-04T08:56:50.3604353Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 284, in forward 2025-12-04T08:56:50.3604440Z hidden_states = self.fc2(hidden_states) 2025-12-04T08:56:50.3604445Z 2025-12-04T08:56:50.3604564Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:50.3604783Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:50.3604860Z res = mod(**inputs) 2025-12-04T08:56:50.3605099Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3605179Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3605448Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T08:56:50.3605525Z outputs = self.model.decoder( 2025-12-04T08:56:50.3605761Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3605847Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3606108Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T08:56:50.3606196Z layer_outputs = decoder_layer( 2025-12-04T08:56:50.3606436Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:56:50.3606522Z return super().__call__(*args, **kwargs) 2025-12-04T08:56:50.3606820Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.3606896Z return func(*args, **kwargs) 2025-12-04T08:56:50.3607171Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 287, in forward 2025-12-04T08:56:50.3607318Z hidden_states = (residual + hidden_states).view(hidden_states_shape) 2025-12-04T08:56:50.3607322Z 2025-12-04T08:56:50.3607432Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:50.3607656Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:50.3607727Z res = mod(**inputs) 2025-12-04T08:56:50.3607985Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3608089Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3608355Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T08:56:50.3608443Z outputs = self.model.decoder( 2025-12-04T08:56:50.3608679Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3608759Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3609030Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T08:56:50.3609109Z layer_outputs = decoder_layer( 2025-12-04T08:56:50.3609353Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:56:50.3609466Z return super().__call__(*args, **kwargs) 2025-12-04T08:56:50.3609730Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.3609814Z return func(*args, **kwargs) 2025-12-04T08:56:50.3610077Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T08:56:50.3610185Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:56:50.3610452Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.3610526Z return func(*args, **kwargs) 2025-12-04T08:56:50.3610791Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 160, in forward 2025-12-04T08:56:50.3610912Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T08:56:50.3610917Z 2025-12-04T08:56:50.3611026Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:50.3611248Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:50.3611320Z res = mod(**inputs) 2025-12-04T08:56:50.3611555Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3611643Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3611904Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T08:56:50.3611990Z outputs = self.model.decoder( 2025-12-04T08:56:50.3612223Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3612301Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3612567Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T08:56:50.3612647Z layer_outputs = decoder_layer( 2025-12-04T08:56:50.3612895Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:56:50.3613004Z return super().__call__(*args, **kwargs) 2025-12-04T08:56:50.3613267Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.3613349Z return func(*args, **kwargs) 2025-12-04T08:56:50.3613611Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T08:56:50.3613716Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:56:50.3613987Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.3614061Z return func(*args, **kwargs) 2025-12-04T08:56:50.3614333Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 163, in forward 2025-12-04T08:56:50.3614444Z key_states = self.k_proj(hidden_states) 2025-12-04T08:56:50.3614470Z 2025-12-04T08:56:50.3614581Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:50.3614811Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:50.3614879Z res = mod(**inputs) 2025-12-04T08:56:50.3615119Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3615201Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3615459Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T08:56:50.3615541Z outputs = self.model.decoder( 2025-12-04T08:56:50.3615768Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3615861Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3616116Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T08:56:50.3616194Z layer_outputs = decoder_layer( 2025-12-04T08:56:50.3616450Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:56:50.3616533Z return super().__call__(*args, **kwargs) 2025-12-04T08:56:50.3616806Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.3616883Z return func(*args, **kwargs) 2025-12-04T08:56:50.3617131Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T08:56:50.3617228Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:56:50.3617487Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.3617562Z return func(*args, **kwargs) 2025-12-04T08:56:50.3617831Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 164, in forward 2025-12-04T08:56:50.3617922Z value_states = self.v_proj(hidden_states) 2025-12-04T08:56:50.3617926Z 2025-12-04T08:56:50.3618011Z cudagraph partition due to non gpu ops 2025-12-04T08:56:50.3618103Z cudagraph partition due to non gpu ops 2025-12-04T08:56:50.3618213Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:50.3618434Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:50.3618510Z res = mod(**inputs) 2025-12-04T08:56:50.3618757Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3618842Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3619104Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T08:56:50.3619186Z outputs = self.model.decoder( 2025-12-04T08:56:50.3619453Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3619533Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3619795Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T08:56:50.3619881Z layer_outputs = decoder_layer( 2025-12-04T08:56:50.3620121Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:56:50.3620212Z return super().__call__(*args, **kwargs) 2025-12-04T08:56:50.3620482Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.3620558Z return func(*args, **kwargs) 2025-12-04T08:56:50.3621036Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T08:56:50.3621176Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:56:50.3621460Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.3621535Z return func(*args, **kwargs) 2025-12-04T08:56:50.3621806Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 179, in forward 2025-12-04T08:56:50.3621912Z attn_output, attn_weights = attention_interface( 2025-12-04T08:56:50.3622214Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T08:56:50.3622354Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T08:56:50.3622400Z 2025-12-04T08:56:50.3622514Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:50.3622732Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:50.3622813Z res = mod(**inputs) 2025-12-04T08:56:50.3623061Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3623138Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3623394Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T08:56:50.3623471Z outputs = self.model.decoder( 2025-12-04T08:56:50.3623706Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3623782Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3624033Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T08:56:50.3624118Z layer_outputs = decoder_layer( 2025-12-04T08:56:50.3624365Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:56:50.3624454Z return super().__call__(*args, **kwargs) 2025-12-04T08:56:50.3624737Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.3624810Z return func(*args, **kwargs) 2025-12-04T08:56:50.3625084Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 255, in forward 2025-12-04T08:56:50.3625191Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:56:50.3625463Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.3625546Z return func(*args, **kwargs) 2025-12-04T08:56:50.3625815Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 191, in forward 2025-12-04T08:56:50.3625905Z attn_output = self.out_proj(attn_output) 2025-12-04T08:56:50.3625930Z 2025-12-04T08:56:50.3626067Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:50.3626284Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:50.3626360Z res = mod(**inputs) 2025-12-04T08:56:50.3626598Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3626677Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3626945Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T08:56:50.3627024Z outputs = self.model.decoder( 2025-12-04T08:56:50.3627268Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3627349Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3627647Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T08:56:50.3627737Z layer_outputs = decoder_layer( 2025-12-04T08:56:50.3627984Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:56:50.3628069Z return super().__call__(*args, **kwargs) 2025-12-04T08:56:50.3628349Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.3628426Z return func(*args, **kwargs) 2025-12-04T08:56:50.3628712Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 281, in forward 2025-12-04T08:56:50.3628798Z hidden_states = self.fc1(hidden_states) 2025-12-04T08:56:50.3628824Z 2025-12-04T08:56:50.3628936Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:50.3629163Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:50.3629232Z res = mod(**inputs) 2025-12-04T08:56:50.3629474Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3629562Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3629828Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T08:56:50.3629914Z outputs = self.model.decoder( 2025-12-04T08:56:50.3630153Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3630233Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3630514Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T08:56:50.3630594Z layer_outputs = decoder_layer( 2025-12-04T08:56:50.3630843Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:56:50.3630932Z return super().__call__(*args, **kwargs) 2025-12-04T08:56:50.3631195Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.3631275Z return func(*args, **kwargs) 2025-12-04T08:56:50.3631539Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 282, in forward 2025-12-04T08:56:50.3631645Z hidden_states = self.activation_fn(hidden_states) 2025-12-04T08:56:50.3631657Z 2025-12-04T08:56:50.3631767Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:50.3631994Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:50.3632072Z res = mod(**inputs) 2025-12-04T08:56:50.3632311Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3632392Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3632679Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 818, in forward 2025-12-04T08:56:50.3632759Z outputs = self.model.decoder( 2025-12-04T08:56:50.3633004Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3633082Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3633344Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward 2025-12-04T08:56:50.3633426Z layer_outputs = decoder_layer( 2025-12-04T08:56:50.3633671Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:56:50.3633758Z return super().__call__(*args, **kwargs) 2025-12-04T08:56:50.3634047Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:56:50.3634138Z return func(*args, **kwargs) 2025-12-04T08:56:50.3634418Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 284, in forward 2025-12-04T08:56:50.3634507Z hidden_states = self.fc2(hidden_states) 2025-12-04T08:56:50.3634511Z 2025-12-04T08:56:50.3665734Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:50.3666199Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:50.3666274Z res = mod(**inputs) 2025-12-04T08:56:50.3666547Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3666644Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3667041Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 833, in forward 2025-12-04T08:56:50.3667160Z logits = self.lm_head(outputs[0]).contiguous() 2025-12-04T08:56:50.3667167Z 2025-12-04T08:56:50.3667290Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:56:50.3667517Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:56:50.3667600Z res = mod(**inputs) 2025-12-04T08:56:50.3667845Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 918, in wrapper 2025-12-04T08:56:50.3667942Z output = func(self, *args, **kwargs) 2025-12-04T08:56:50.3668214Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 839, in forward 2025-12-04T08:56:50.3668296Z loss = self.loss_function( 2025-12-04T08:56:50.3668562Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/loss/loss_utils.py", line 67, in ForCausalLMLoss 2025-12-04T08:56:50.3668755Z loss = fixed_cross_entropy(logits, shift_labels, num_items_in_batch, ignore_index, **kwargs) 2025-12-04T08:56:50.3669029Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/loss/loss_utils.py", line 36, in fixed_cross_entropy 2025-12-04T08:56:50.3669250Z loss = nn.functional.cross_entropy(source, target, ignore_index=ignore_index, reduction=reduction) 2025-12-04T08:56:50.3669255Z 2025-12-04T08:57:05.5964549Z Compilation time (from dynamo_timed): 18.936035283 2025-12-04T08:57:05.6439150Z pass 2025-12-04T08:57:05.6439660Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T08:57:05.6440565Z TIMING: _recursive_pre_grad_passes:0.00848 _recursive_joint_graph_passes:0.67329 _recursive_post_grad_passes:0.08772 async_compile.wait:1.602 code_gen:11.83079 inductor_compile:13.15852 backend_compile:16.3582 gc:0.00115 entire_frame_compile:18.93604 total_wall_time:18.93604 2025-12-04T08:57:05.6441657Z STATS: call_* op count: 379 | FakeTensorMode.__torch_dispatch__:7020 | FakeTensor.__torch_dispatch__:4035 | ProxyTorchDispatchMode.__torch_dispatch__:1957 2025-12-04T08:57:05.6442519Z Dynamo produced 1 graphs covering 379 ops with 0 graph breaks (0 unique) 2025-12-04T08:57:08.1177327Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/__init__.py:65: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-12-04T08:57:08.1178257Z import pynvml # type: ignore[import] 2025-12-04T08:57:11.5773277Z 2025-12-04T08:57:12.9695585Z loading model: 0it [00:00, ?it/s] 2025-12-04T08:57:12.9696038Z loading model: 0it [00:01, ?it/s] 2025-12-04T08:57:12.9709405Z cpu eval PLBartForCausalLM 2025-12-04T08:57:14.8580826Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T08:57:15.7292612Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T08:57:16.5661672Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T08:57:21.2189818Z cudagraph partition due to non gpu ops 2025-12-04T08:57:21.2190122Z cudagraph partition due to non gpu ops 2025-12-04T08:57:21.2190408Z cudagraph partition due to non gpu ops 2025-12-04T08:57:21.2190681Z cudagraph partition due to non gpu ops 2025-12-04T08:57:21.2190946Z cudagraph partition due to non gpu ops 2025-12-04T08:57:21.2191215Z cudagraph partition due to non gpu ops 2025-12-04T08:57:21.2191508Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:21.2192273Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:21.2192697Z res = mod(**inputs) 2025-12-04T08:57:21.2193519Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1680, in forward 2025-12-04T08:57:21.2194021Z outputs = self.model.decoder( 2025-12-04T08:57:21.2194603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1039, in forward 2025-12-04T08:57:21.2195094Z layer_outputs = decoder_layer( 2025-12-04T08:57:21.2195541Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:57:21.2195994Z return super().__call__(*args, **kwargs) 2025-12-04T08:57:21.2196437Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:21.2196920Z return func(*args, **kwargs) 2025-12-04T08:57:21.2197374Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 766, in forward 2025-12-04T08:57:21.2197901Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:57:21.2198512Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:21.2198935Z return func(*args, **kwargs) 2025-12-04T08:57:21.2199361Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 404, in forward 2025-12-04T08:57:21.2199900Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T08:57:21.2200153Z 2025-12-04T08:57:21.2200277Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:21.2200684Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:21.2201046Z res = mod(**inputs) 2025-12-04T08:57:21.2201461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1680, in forward 2025-12-04T08:57:21.2201905Z outputs = self.model.decoder( 2025-12-04T08:57:21.2202338Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1039, in forward 2025-12-04T08:57:21.2202777Z layer_outputs = decoder_layer( 2025-12-04T08:57:21.2203252Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:57:21.2203655Z return super().__call__(*args, **kwargs) 2025-12-04T08:57:21.2204158Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:21.2204579Z return func(*args, **kwargs) 2025-12-04T08:57:21.2204988Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 766, in forward 2025-12-04T08:57:21.2205428Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:57:21.2205910Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:21.2206407Z return func(*args, **kwargs) 2025-12-04T08:57:21.2206879Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 424, in forward 2025-12-04T08:57:21.2207303Z key_states = self.k_proj(current_states) 2025-12-04T08:57:21.2207455Z 2025-12-04T08:57:21.2207568Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:21.2207948Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:21.2208296Z res = mod(**inputs) 2025-12-04T08:57:21.2208680Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1680, in forward 2025-12-04T08:57:21.2209101Z outputs = self.model.decoder( 2025-12-04T08:57:21.2209517Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1039, in forward 2025-12-04T08:57:21.2209948Z layer_outputs = decoder_layer( 2025-12-04T08:57:21.2210306Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:57:21.2210681Z return super().__call__(*args, **kwargs) 2025-12-04T08:57:21.2211072Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:21.2211452Z return func(*args, **kwargs) 2025-12-04T08:57:21.2211845Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 766, in forward 2025-12-04T08:57:21.2212279Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:57:21.2212689Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:21.2213066Z return func(*args, **kwargs) 2025-12-04T08:57:21.2213458Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 425, in forward 2025-12-04T08:57:21.2213879Z value_states = self.v_proj(current_states) 2025-12-04T08:57:21.2214027Z 2025-12-04T08:57:21.2214117Z cudagraph partition due to non gpu ops 2025-12-04T08:57:21.2214362Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:21.2214729Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:21.2215058Z res = mod(**inputs) 2025-12-04T08:57:21.2215431Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1680, in forward 2025-12-04T08:57:21.2215835Z outputs = self.model.decoder( 2025-12-04T08:57:21.2216236Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1039, in forward 2025-12-04T08:57:21.2216642Z layer_outputs = decoder_layer( 2025-12-04T08:57:21.2216998Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:57:21.2217373Z return super().__call__(*args, **kwargs) 2025-12-04T08:57:21.2217784Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:21.2218165Z return func(*args, **kwargs) 2025-12-04T08:57:21.2218559Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 766, in forward 2025-12-04T08:57:21.2218990Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:57:21.2219407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:21.2219766Z return func(*args, **kwargs) 2025-12-04T08:57:21.2220139Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 443, in forward 2025-12-04T08:57:21.2220555Z attn_output, attn_weights = attention_interface( 2025-12-04T08:57:21.2221215Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T08:57:21.2221731Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T08:57:21.2221927Z 2025-12-04T08:57:21.2222031Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:21.2222393Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:21.2222707Z res = mod(**inputs) 2025-12-04T08:57:21.2223072Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1680, in forward 2025-12-04T08:57:21.2223472Z outputs = self.model.decoder( 2025-12-04T08:57:21.2223862Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1039, in forward 2025-12-04T08:57:21.2224332Z layer_outputs = decoder_layer( 2025-12-04T08:57:21.2224681Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:57:21.2225051Z return super().__call__(*args, **kwargs) 2025-12-04T08:57:21.2225430Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:21.2225801Z return func(*args, **kwargs) 2025-12-04T08:57:21.2226178Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 766, in forward 2025-12-04T08:57:21.2226592Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:57:21.2226982Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:21.2227357Z return func(*args, **kwargs) 2025-12-04T08:57:21.2227742Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 457, in forward 2025-12-04T08:57:21.2228155Z attn_output = self.out_proj(attn_output) 2025-12-04T08:57:21.2228294Z 2025-12-04T08:57:21.2228399Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:21.2228763Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:21.2229086Z res = mod(**inputs) 2025-12-04T08:57:21.2229445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1680, in forward 2025-12-04T08:57:21.2229847Z outputs = self.model.decoder( 2025-12-04T08:57:21.2230238Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1039, in forward 2025-12-04T08:57:21.2230641Z layer_outputs = decoder_layer( 2025-12-04T08:57:21.2231005Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:57:21.2231372Z return super().__call__(*args, **kwargs) 2025-12-04T08:57:21.2231755Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:21.2232156Z return func(*args, **kwargs) 2025-12-04T08:57:21.2232548Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 798, in forward 2025-12-04T08:57:21.2232993Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:57:21.2233175Z 2025-12-04T08:57:21.2233281Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:21.2233645Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:21.2233966Z res = mod(**inputs) 2025-12-04T08:57:21.2234339Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1680, in forward 2025-12-04T08:57:21.2234741Z outputs = self.model.decoder( 2025-12-04T08:57:21.2235152Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1039, in forward 2025-12-04T08:57:21.2235579Z layer_outputs = decoder_layer( 2025-12-04T08:57:21.2235945Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:57:21.2236319Z return super().__call__(*args, **kwargs) 2025-12-04T08:57:21.2236704Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:21.2237094Z return func(*args, **kwargs) 2025-12-04T08:57:21.2237489Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 798, in forward 2025-12-04T08:57:21.2237945Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:57:21.2238442Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T08:57:21.2238856Z return self.act(input) 2025-12-04T08:57:21.2238977Z 2025-12-04T08:57:21.2239100Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:21.2239494Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:21.2239831Z res = mod(**inputs) 2025-12-04T08:57:21.2240212Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1680, in forward 2025-12-04T08:57:21.2240620Z outputs = self.model.decoder( 2025-12-04T08:57:21.2241016Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1039, in forward 2025-12-04T08:57:21.2241440Z layer_outputs = decoder_layer( 2025-12-04T08:57:21.2241830Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:57:21.2242221Z return super().__call__(*args, **kwargs) 2025-12-04T08:57:21.2242620Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:21.2243005Z return func(*args, **kwargs) 2025-12-04T08:57:21.2243399Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 800, in forward 2025-12-04T08:57:21.2243811Z hidden_states = self.fc2(hidden_states) 2025-12-04T08:57:21.2243960Z 2025-12-04T08:57:21.2244067Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:21.2244436Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:21.2244770Z res = mod(**inputs) 2025-12-04T08:57:21.2245141Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1680, in forward 2025-12-04T08:57:21.2245548Z outputs = self.model.decoder( 2025-12-04T08:57:21.2245953Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1039, in forward 2025-12-04T08:57:21.2246353Z layer_outputs = decoder_layer( 2025-12-04T08:57:21.2246732Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:57:21.2247115Z return super().__call__(*args, **kwargs) 2025-12-04T08:57:21.2247519Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:21.2247912Z return func(*args, **kwargs) 2025-12-04T08:57:21.2248316Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 766, in forward 2025-12-04T08:57:21.2248760Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:57:21.2249180Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:21.2249579Z return func(*args, **kwargs) 2025-12-04T08:57:21.2250017Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 404, in forward 2025-12-04T08:57:21.2250537Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T08:57:21.2250743Z 2025-12-04T08:57:21.2250849Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:21.2251212Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:21.2251537Z res = mod(**inputs) 2025-12-04T08:57:21.2251910Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1680, in forward 2025-12-04T08:57:21.2252305Z outputs = self.model.decoder( 2025-12-04T08:57:21.2252693Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1039, in forward 2025-12-04T08:57:21.2253106Z layer_outputs = decoder_layer( 2025-12-04T08:57:21.2253454Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:57:21.2253825Z return super().__call__(*args, **kwargs) 2025-12-04T08:57:21.2254354Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:21.2254743Z return func(*args, **kwargs) 2025-12-04T08:57:21.2255132Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 766, in forward 2025-12-04T08:57:21.2255582Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:57:21.2255991Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:21.2256365Z return func(*args, **kwargs) 2025-12-04T08:57:21.2256751Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 424, in forward 2025-12-04T08:57:21.2257164Z key_states = self.k_proj(current_states) 2025-12-04T08:57:21.2257305Z 2025-12-04T08:57:21.2257422Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:21.2257788Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:21.2258122Z res = mod(**inputs) 2025-12-04T08:57:21.2258495Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1680, in forward 2025-12-04T08:57:21.2258895Z outputs = self.model.decoder( 2025-12-04T08:57:21.2259280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1039, in forward 2025-12-04T08:57:21.2259678Z layer_outputs = decoder_layer( 2025-12-04T08:57:21.2260040Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:57:21.2260411Z return super().__call__(*args, **kwargs) 2025-12-04T08:57:21.2260808Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:21.2261218Z return func(*args, **kwargs) 2025-12-04T08:57:21.2261616Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 766, in forward 2025-12-04T08:57:21.2262045Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:57:21.2262573Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:21.2262968Z return func(*args, **kwargs) 2025-12-04T08:57:21.2263358Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 425, in forward 2025-12-04T08:57:21.2263791Z value_states = self.v_proj(current_states) 2025-12-04T08:57:21.2263948Z 2025-12-04T08:57:21.2264032Z cudagraph partition due to non gpu ops 2025-12-04T08:57:21.2264299Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:21.2264678Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:21.2265007Z res = mod(**inputs) 2025-12-04T08:57:21.2265384Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1680, in forward 2025-12-04T08:57:21.2265787Z outputs = self.model.decoder( 2025-12-04T08:57:21.2266175Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1039, in forward 2025-12-04T08:57:21.2266574Z layer_outputs = decoder_layer( 2025-12-04T08:57:21.2266924Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:57:21.2267287Z return super().__call__(*args, **kwargs) 2025-12-04T08:57:21.2267699Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:21.2268075Z return func(*args, **kwargs) 2025-12-04T08:57:21.2268457Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 766, in forward 2025-12-04T08:57:21.2268871Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:57:21.2269273Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:21.2269647Z return func(*args, **kwargs) 2025-12-04T08:57:21.2270023Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 443, in forward 2025-12-04T08:57:21.2270444Z attn_output, attn_weights = attention_interface( 2025-12-04T08:57:21.2270894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T08:57:21.2271435Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T08:57:21.2271625Z 2025-12-04T08:57:21.2271732Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:21.2272107Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:21.2272441Z res = mod(**inputs) 2025-12-04T08:57:21.2272818Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1680, in forward 2025-12-04T08:57:21.2273220Z outputs = self.model.decoder( 2025-12-04T08:57:21.2273621Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1039, in forward 2025-12-04T08:57:21.2274023Z layer_outputs = decoder_layer( 2025-12-04T08:57:21.2274394Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:57:21.2274800Z return super().__call__(*args, **kwargs) 2025-12-04T08:57:21.2275228Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:21.2275640Z return func(*args, **kwargs) 2025-12-04T08:57:21.2276072Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 766, in forward 2025-12-04T08:57:21.2276535Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:57:21.2276974Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:21.2277380Z return func(*args, **kwargs) 2025-12-04T08:57:21.2277793Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 457, in forward 2025-12-04T08:57:21.2278291Z attn_output = self.out_proj(attn_output) 2025-12-04T08:57:21.2278450Z 2025-12-04T08:57:21.2278570Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:21.2278976Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:21.2279352Z res = mod(**inputs) 2025-12-04T08:57:21.2279756Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1680, in forward 2025-12-04T08:57:21.2280190Z outputs = self.model.decoder( 2025-12-04T08:57:21.2280619Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1039, in forward 2025-12-04T08:57:21.2281030Z layer_outputs = decoder_layer( 2025-12-04T08:57:21.2281395Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:57:21.2281768Z return super().__call__(*args, **kwargs) 2025-12-04T08:57:21.2282165Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:21.2282572Z return func(*args, **kwargs) 2025-12-04T08:57:21.2282966Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 798, in forward 2025-12-04T08:57:21.2283414Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:57:21.2283600Z 2025-12-04T08:57:21.2283708Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:21.2284080Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:21.2284415Z res = mod(**inputs) 2025-12-04T08:57:21.2284788Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1680, in forward 2025-12-04T08:57:21.2285199Z outputs = self.model.decoder( 2025-12-04T08:57:21.2285597Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1039, in forward 2025-12-04T08:57:21.2285998Z layer_outputs = decoder_layer( 2025-12-04T08:57:21.2286362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:57:21.2286736Z return super().__call__(*args, **kwargs) 2025-12-04T08:57:21.2287130Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:21.2287508Z return func(*args, **kwargs) 2025-12-04T08:57:21.2287907Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 798, in forward 2025-12-04T08:57:21.2288358Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:57:21.2288754Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T08:57:21.2289112Z return self.act(input) 2025-12-04T08:57:21.2289231Z 2025-12-04T08:57:21.2289340Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:21.2289713Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:21.2290038Z res = mod(**inputs) 2025-12-04T08:57:21.2290441Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1680, in forward 2025-12-04T08:57:21.2290854Z outputs = self.model.decoder( 2025-12-04T08:57:21.2291254Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1039, in forward 2025-12-04T08:57:21.2291654Z layer_outputs = decoder_layer( 2025-12-04T08:57:21.2292013Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:57:21.2292398Z return super().__call__(*args, **kwargs) 2025-12-04T08:57:21.2292783Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:21.2293172Z return func(*args, **kwargs) 2025-12-04T08:57:21.2293579Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 800, in forward 2025-12-04T08:57:21.2294016Z hidden_states = self.fc2(hidden_states) 2025-12-04T08:57:21.2294160Z 2025-12-04T08:57:21.2294267Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:21.2294638Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:21.2294973Z res = mod(**inputs) 2025-12-04T08:57:21.2295348Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1680, in forward 2025-12-04T08:57:21.2295757Z outputs = self.model.decoder( 2025-12-04T08:57:21.2296157Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1039, in forward 2025-12-04T08:57:21.2296580Z layer_outputs = decoder_layer( 2025-12-04T08:57:21.2296935Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:57:21.2297323Z return super().__call__(*args, **kwargs) 2025-12-04T08:57:21.2297715Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:21.2298100Z return func(*args, **kwargs) 2025-12-04T08:57:21.2298489Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 766, in forward 2025-12-04T08:57:21.2298922Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:57:21.2299332Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:21.2299709Z return func(*args, **kwargs) 2025-12-04T08:57:21.2300102Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 404, in forward 2025-12-04T08:57:21.2300589Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T08:57:21.2300800Z 2025-12-04T08:57:21.2300914Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:21.2301277Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:21.2301615Z res = mod(**inputs) 2025-12-04T08:57:21.2301996Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1680, in forward 2025-12-04T08:57:21.2302400Z outputs = self.model.decoder( 2025-12-04T08:57:21.2302791Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1039, in forward 2025-12-04T08:57:21.2303196Z layer_outputs = decoder_layer( 2025-12-04T08:57:21.2303558Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:57:21.2303928Z return super().__call__(*args, **kwargs) 2025-12-04T08:57:21.2304328Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:21.2304742Z return func(*args, **kwargs) 2025-12-04T08:57:21.2305135Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 766, in forward 2025-12-04T08:57:21.2305549Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:57:21.2305978Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:21.2306369Z return func(*args, **kwargs) 2025-12-04T08:57:21.2306761Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 424, in forward 2025-12-04T08:57:21.2307254Z key_states = self.k_proj(current_states) 2025-12-04T08:57:21.2307400Z 2025-12-04T08:57:21.2307508Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:21.2308311Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:21.2308653Z res = mod(**inputs) 2025-12-04T08:57:21.2309030Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1680, in forward 2025-12-04T08:57:21.2309430Z outputs = self.model.decoder( 2025-12-04T08:57:21.2309832Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1039, in forward 2025-12-04T08:57:21.2310233Z layer_outputs = decoder_layer( 2025-12-04T08:57:21.2310597Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:57:21.2310976Z return super().__call__(*args, **kwargs) 2025-12-04T08:57:21.2311362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:21.2311775Z return func(*args, **kwargs) 2025-12-04T08:57:21.2312171Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 766, in forward 2025-12-04T08:57:21.2312604Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:57:21.2313014Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:21.2313394Z return func(*args, **kwargs) 2025-12-04T08:57:21.2313786Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 425, in forward 2025-12-04T08:57:21.2314200Z value_states = self.v_proj(current_states) 2025-12-04T08:57:21.2314354Z 2025-12-04T08:57:21.2314443Z cudagraph partition due to non gpu ops 2025-12-04T08:57:21.2314706Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:21.2315105Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:21.2315464Z res = mod(**inputs) 2025-12-04T08:57:21.2315883Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1680, in forward 2025-12-04T08:57:21.2316319Z outputs = self.model.decoder( 2025-12-04T08:57:21.2316741Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1039, in forward 2025-12-04T08:57:21.2317167Z layer_outputs = decoder_layer( 2025-12-04T08:57:21.2317548Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:57:21.2317948Z return super().__call__(*args, **kwargs) 2025-12-04T08:57:21.2318459Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:21.2318894Z return func(*args, **kwargs) 2025-12-04T08:57:21.2319339Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 766, in forward 2025-12-04T08:57:21.2319779Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:57:21.2320249Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:21.2320674Z return func(*args, **kwargs) 2025-12-04T08:57:21.2321269Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 443, in forward 2025-12-04T08:57:21.2321736Z attn_output, attn_weights = attention_interface( 2025-12-04T08:57:21.2322241Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T08:57:21.2322789Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T08:57:21.2322999Z 2025-12-04T08:57:21.2323128Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:21.2323586Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:21.2323979Z res = mod(**inputs) 2025-12-04T08:57:21.2324406Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1680, in forward 2025-12-04T08:57:21.2324854Z outputs = self.model.decoder( 2025-12-04T08:57:21.2325281Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1039, in forward 2025-12-04T08:57:21.2325722Z layer_outputs = decoder_layer( 2025-12-04T08:57:21.2326118Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:57:21.2326521Z return super().__call__(*args, **kwargs) 2025-12-04T08:57:21.2326952Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:21.2327389Z return func(*args, **kwargs) 2025-12-04T08:57:21.2327784Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 766, in forward 2025-12-04T08:57:21.2328213Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:57:21.2328628Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:21.2329012Z return func(*args, **kwargs) 2025-12-04T08:57:21.2329406Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 457, in forward 2025-12-04T08:57:21.2329811Z attn_output = self.out_proj(attn_output) 2025-12-04T08:57:21.2329959Z 2025-12-04T08:57:21.2330174Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:21.2330546Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:21.2330877Z res = mod(**inputs) 2025-12-04T08:57:21.2331284Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1680, in forward 2025-12-04T08:57:21.2331722Z outputs = self.model.decoder( 2025-12-04T08:57:21.2332150Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1039, in forward 2025-12-04T08:57:21.2332573Z layer_outputs = decoder_layer( 2025-12-04T08:57:21.2332959Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:57:21.2333363Z return super().__call__(*args, **kwargs) 2025-12-04T08:57:21.2333749Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:21.2334136Z return func(*args, **kwargs) 2025-12-04T08:57:21.2334527Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 798, in forward 2025-12-04T08:57:21.2334981Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:57:21.2335170Z 2025-12-04T08:57:21.2335273Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:21.2335658Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:21.2335984Z res = mod(**inputs) 2025-12-04T08:57:21.2336355Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1680, in forward 2025-12-04T08:57:21.2336749Z outputs = self.model.decoder( 2025-12-04T08:57:21.2337140Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1039, in forward 2025-12-04T08:57:21.2337538Z layer_outputs = decoder_layer( 2025-12-04T08:57:21.2337884Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:57:21.2338266Z return super().__call__(*args, **kwargs) 2025-12-04T08:57:21.2338671Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:21.2339062Z return func(*args, **kwargs) 2025-12-04T08:57:21.2339435Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 798, in forward 2025-12-04T08:57:21.2339873Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:57:21.2340264Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T08:57:21.2340613Z return self.act(input) 2025-12-04T08:57:21.2340725Z 2025-12-04T08:57:21.2340827Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:21.2341191Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:21.2341532Z res = mod(**inputs) 2025-12-04T08:57:21.2341894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1680, in forward 2025-12-04T08:57:21.2342296Z outputs = self.model.decoder( 2025-12-04T08:57:21.2342685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1039, in forward 2025-12-04T08:57:21.2343073Z layer_outputs = decoder_layer( 2025-12-04T08:57:21.2343417Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:57:21.2343780Z return super().__call__(*args, **kwargs) 2025-12-04T08:57:21.2344161Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:21.2344528Z return func(*args, **kwargs) 2025-12-04T08:57:21.2344906Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 800, in forward 2025-12-04T08:57:21.2345321Z hidden_states = self.fc2(hidden_states) 2025-12-04T08:57:21.2345458Z 2025-12-04T08:57:21.2345568Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:21.2345911Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:21.2346224Z res = mod(**inputs) 2025-12-04T08:57:21.2346580Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1680, in forward 2025-12-04T08:57:21.2346982Z outputs = self.model.decoder( 2025-12-04T08:57:21.2347348Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1039, in forward 2025-12-04T08:57:21.2347731Z layer_outputs = decoder_layer( 2025-12-04T08:57:21.2348071Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:57:21.2348421Z return super().__call__(*args, **kwargs) 2025-12-04T08:57:21.2348794Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:21.2349187Z return func(*args, **kwargs) 2025-12-04T08:57:21.2349578Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 766, in forward 2025-12-04T08:57:21.2349980Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:57:21.2350368Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:21.2350727Z return func(*args, **kwargs) 2025-12-04T08:57:21.2351091Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 404, in forward 2025-12-04T08:57:21.2351549Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T08:57:21.2351757Z 2025-12-04T08:57:21.2351857Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:21.2352222Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:21.2352549Z res = mod(**inputs) 2025-12-04T08:57:21.2352909Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1680, in forward 2025-12-04T08:57:21.2353296Z outputs = self.model.decoder( 2025-12-04T08:57:21.2353679Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1039, in forward 2025-12-04T08:57:21.2354067Z layer_outputs = decoder_layer( 2025-12-04T08:57:21.2354418Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:57:21.2354780Z return super().__call__(*args, **kwargs) 2025-12-04T08:57:21.2355154Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:21.2355564Z return func(*args, **kwargs) 2025-12-04T08:57:21.2355952Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 766, in forward 2025-12-04T08:57:21.2356374Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:57:21.2356778Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:21.2357168Z return func(*args, **kwargs) 2025-12-04T08:57:21.2357560Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 424, in forward 2025-12-04T08:57:21.2357970Z key_states = self.k_proj(current_states) 2025-12-04T08:57:21.2358198Z 2025-12-04T08:57:21.2358312Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:21.2358737Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:21.2359115Z res = mod(**inputs) 2025-12-04T08:57:21.2359541Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1680, in forward 2025-12-04T08:57:21.2360064Z outputs = self.model.decoder( 2025-12-04T08:57:21.2360476Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1039, in forward 2025-12-04T08:57:21.2360884Z layer_outputs = decoder_layer( 2025-12-04T08:57:21.2361222Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:57:21.2361583Z return super().__call__(*args, **kwargs) 2025-12-04T08:57:21.2361958Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:21.2362326Z return func(*args, **kwargs) 2025-12-04T08:57:21.2362714Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 766, in forward 2025-12-04T08:57:21.2363142Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:57:21.2363570Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:21.2363946Z return func(*args, **kwargs) 2025-12-04T08:57:21.2364334Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 425, in forward 2025-12-04T08:57:21.2364752Z value_states = self.v_proj(current_states) 2025-12-04T08:57:21.2364890Z 2025-12-04T08:57:21.2364980Z cudagraph partition due to non gpu ops 2025-12-04T08:57:21.2365215Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:21.2365575Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:21.2365905Z res = mod(**inputs) 2025-12-04T08:57:21.2366268Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1680, in forward 2025-12-04T08:57:21.2366705Z outputs = self.model.decoder( 2025-12-04T08:57:21.2367103Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1039, in forward 2025-12-04T08:57:21.2367493Z layer_outputs = decoder_layer( 2025-12-04T08:57:21.2367839Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:57:21.2368200Z return super().__call__(*args, **kwargs) 2025-12-04T08:57:21.2368578Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:21.2368949Z return func(*args, **kwargs) 2025-12-04T08:57:21.2369329Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 766, in forward 2025-12-04T08:57:21.2369772Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:57:21.2370176Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:21.2370547Z return func(*args, **kwargs) 2025-12-04T08:57:21.2370935Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 443, in forward 2025-12-04T08:57:21.2371355Z attn_output, attn_weights = attention_interface( 2025-12-04T08:57:21.2371800Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T08:57:21.2372289Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T08:57:21.2372479Z 2025-12-04T08:57:21.2372583Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:21.2372944Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:21.2373263Z res = mod(**inputs) 2025-12-04T08:57:21.2373634Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1680, in forward 2025-12-04T08:57:21.2374036Z outputs = self.model.decoder( 2025-12-04T08:57:21.2374428Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1039, in forward 2025-12-04T08:57:21.2374812Z layer_outputs = decoder_layer( 2025-12-04T08:57:21.2375165Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:57:21.2375533Z return super().__call__(*args, **kwargs) 2025-12-04T08:57:21.2375910Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:21.2376287Z return func(*args, **kwargs) 2025-12-04T08:57:21.2376669Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 766, in forward 2025-12-04T08:57:21.2377092Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:57:21.2377489Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:21.2377893Z return func(*args, **kwargs) 2025-12-04T08:57:21.2378274Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 457, in forward 2025-12-04T08:57:21.2378668Z attn_output = self.out_proj(attn_output) 2025-12-04T08:57:21.2378802Z 2025-12-04T08:57:21.2378904Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:21.2379253Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:21.2379567Z res = mod(**inputs) 2025-12-04T08:57:21.2379919Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1680, in forward 2025-12-04T08:57:21.2380309Z outputs = self.model.decoder( 2025-12-04T08:57:21.2380714Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1039, in forward 2025-12-04T08:57:21.2381130Z layer_outputs = decoder_layer( 2025-12-04T08:57:21.2381476Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:57:21.2381847Z return super().__call__(*args, **kwargs) 2025-12-04T08:57:21.2382217Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:21.2382573Z return func(*args, **kwargs) 2025-12-04T08:57:21.2382942Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 798, in forward 2025-12-04T08:57:21.2383372Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:57:21.2383559Z 2025-12-04T08:57:21.2383666Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:21.2384015Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:21.2384343Z res = mod(**inputs) 2025-12-04T08:57:21.2384716Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1680, in forward 2025-12-04T08:57:21.2385111Z outputs = self.model.decoder( 2025-12-04T08:57:21.2385494Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1039, in forward 2025-12-04T08:57:21.2385888Z layer_outputs = decoder_layer( 2025-12-04T08:57:21.2386240Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:57:21.2386597Z return super().__call__(*args, **kwargs) 2025-12-04T08:57:21.2386979Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:21.2387356Z return func(*args, **kwargs) 2025-12-04T08:57:21.2387739Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 798, in forward 2025-12-04T08:57:21.2388171Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:57:21.2388563Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T08:57:21.2388916Z return self.act(input) 2025-12-04T08:57:21.2389050Z 2025-12-04T08:57:21.2389160Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:21.2389514Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:21.2389839Z res = mod(**inputs) 2025-12-04T08:57:21.2390209Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1680, in forward 2025-12-04T08:57:21.2390602Z outputs = self.model.decoder( 2025-12-04T08:57:21.2390995Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1039, in forward 2025-12-04T08:57:21.2391398Z layer_outputs = decoder_layer( 2025-12-04T08:57:21.2391779Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:57:21.2392148Z return super().__call__(*args, **kwargs) 2025-12-04T08:57:21.2392540Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:21.2392939Z return func(*args, **kwargs) 2025-12-04T08:57:21.2393333Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 800, in forward 2025-12-04T08:57:21.2393757Z hidden_states = self.fc2(hidden_states) 2025-12-04T08:57:21.2393910Z 2025-12-04T08:57:21.2394022Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:21.2394415Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:21.2394765Z res = mod(**inputs) 2025-12-04T08:57:21.2395157Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1680, in forward 2025-12-04T08:57:21.2395589Z outputs = self.model.decoder( 2025-12-04T08:57:21.2396017Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1039, in forward 2025-12-04T08:57:21.2396455Z layer_outputs = decoder_layer( 2025-12-04T08:57:21.2396840Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:57:21.2397249Z return super().__call__(*args, **kwargs) 2025-12-04T08:57:21.2397667Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:21.2398182Z return func(*args, **kwargs) 2025-12-04T08:57:21.2398611Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 766, in forward 2025-12-04T08:57:21.2399084Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:57:21.2399512Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:21.2399899Z return func(*args, **kwargs) 2025-12-04T08:57:21.2400292Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 404, in forward 2025-12-04T08:57:21.2400847Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T08:57:21.2401066Z 2025-12-04T08:57:21.2401174Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:21.2401541Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:21.2401871Z res = mod(**inputs) 2025-12-04T08:57:21.2402245Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1680, in forward 2025-12-04T08:57:21.2402655Z outputs = self.model.decoder( 2025-12-04T08:57:21.2403059Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1039, in forward 2025-12-04T08:57:21.2403460Z layer_outputs = decoder_layer( 2025-12-04T08:57:21.2403814Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:57:21.2404190Z return super().__call__(*args, **kwargs) 2025-12-04T08:57:21.2404577Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:21.2404956Z return func(*args, **kwargs) 2025-12-04T08:57:21.2405346Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 766, in forward 2025-12-04T08:57:21.2405775Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:57:21.2406210Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:21.2406591Z return func(*args, **kwargs) 2025-12-04T08:57:21.2406962Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 424, in forward 2025-12-04T08:57:21.2407369Z key_states = self.k_proj(current_states) 2025-12-04T08:57:21.2407505Z 2025-12-04T08:57:21.2407610Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:21.2407972Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:21.2408299Z res = mod(**inputs) 2025-12-04T08:57:21.2408665Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1680, in forward 2025-12-04T08:57:21.2409059Z outputs = self.model.decoder( 2025-12-04T08:57:21.2409468Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1039, in forward 2025-12-04T08:57:21.2409895Z layer_outputs = decoder_layer( 2025-12-04T08:57:21.2410250Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:57:21.2410608Z return super().__call__(*args, **kwargs) 2025-12-04T08:57:21.2410988Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:21.2411366Z return func(*args, **kwargs) 2025-12-04T08:57:21.2411740Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 766, in forward 2025-12-04T08:57:21.2412158Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:57:21.2412597Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:21.2412960Z return func(*args, **kwargs) 2025-12-04T08:57:21.2413324Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 425, in forward 2025-12-04T08:57:21.2413721Z value_states = self.v_proj(current_states) 2025-12-04T08:57:21.2413858Z 2025-12-04T08:57:21.2413943Z cudagraph partition due to non gpu ops 2025-12-04T08:57:21.2414171Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:21.2414523Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:21.2414833Z res = mod(**inputs) 2025-12-04T08:57:21.2415190Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1680, in forward 2025-12-04T08:57:21.2415565Z outputs = self.model.decoder( 2025-12-04T08:57:21.2415946Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1039, in forward 2025-12-04T08:57:21.2416337Z layer_outputs = decoder_layer( 2025-12-04T08:57:21.2416693Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:57:21.2417049Z return super().__call__(*args, **kwargs) 2025-12-04T08:57:21.2417422Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:21.2417792Z return func(*args, **kwargs) 2025-12-04T08:57:21.2418165Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 766, in forward 2025-12-04T08:57:21.2418581Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:57:21.2418981Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:21.2419355Z return func(*args, **kwargs) 2025-12-04T08:57:21.2419727Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 443, in forward 2025-12-04T08:57:21.2420166Z attn_output, attn_weights = attention_interface( 2025-12-04T08:57:21.2420630Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T08:57:21.2421273Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T08:57:21.2421491Z 2025-12-04T08:57:21.2421598Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:21.2421966Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:21.2422292Z res = mod(**inputs) 2025-12-04T08:57:21.2422654Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1680, in forward 2025-12-04T08:57:21.2423061Z outputs = self.model.decoder( 2025-12-04T08:57:21.2423499Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1039, in forward 2025-12-04T08:57:21.2423920Z layer_outputs = decoder_layer( 2025-12-04T08:57:21.2424268Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:57:21.2424636Z return super().__call__(*args, **kwargs) 2025-12-04T08:57:21.2425020Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:21.2425388Z return func(*args, **kwargs) 2025-12-04T08:57:21.2425772Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 766, in forward 2025-12-04T08:57:21.2426192Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:57:21.2426641Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:21.2427085Z return func(*args, **kwargs) 2025-12-04T08:57:21.2427471Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 457, in forward 2025-12-04T08:57:21.2427875Z attn_output = self.out_proj(attn_output) 2025-12-04T08:57:21.2428013Z 2025-12-04T08:57:21.2428125Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:21.2428478Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:21.2428802Z res = mod(**inputs) 2025-12-04T08:57:21.2429169Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1680, in forward 2025-12-04T08:57:21.2429560Z outputs = self.model.decoder( 2025-12-04T08:57:21.2429948Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1039, in forward 2025-12-04T08:57:21.2430344Z layer_outputs = decoder_layer( 2025-12-04T08:57:21.2430699Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:57:21.2431057Z return super().__call__(*args, **kwargs) 2025-12-04T08:57:21.2431438Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:21.2431813Z return func(*args, **kwargs) 2025-12-04T08:57:21.2432186Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 798, in forward 2025-12-04T08:57:21.2432625Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:57:21.2432804Z 2025-12-04T08:57:21.2432908Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:21.2433264Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:21.2433589Z res = mod(**inputs) 2025-12-04T08:57:21.2433969Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1680, in forward 2025-12-04T08:57:21.2434377Z outputs = self.model.decoder( 2025-12-04T08:57:21.2434798Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1039, in forward 2025-12-04T08:57:21.2435200Z layer_outputs = decoder_layer( 2025-12-04T08:57:21.2435567Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:57:21.2435946Z return super().__call__(*args, **kwargs) 2025-12-04T08:57:21.2436347Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:21.2436734Z return func(*args, **kwargs) 2025-12-04T08:57:21.2437129Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 798, in forward 2025-12-04T08:57:21.2437602Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:57:21.2438022Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T08:57:21.2438456Z return self.act(input) 2025-12-04T08:57:21.2438581Z 2025-12-04T08:57:21.2438704Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:21.2439090Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:21.2439440Z res = mod(**inputs) 2025-12-04T08:57:21.2439839Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1680, in forward 2025-12-04T08:57:21.2440270Z outputs = self.model.decoder( 2025-12-04T08:57:21.2440685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1039, in forward 2025-12-04T08:57:21.2441151Z layer_outputs = decoder_layer( 2025-12-04T08:57:21.2441522Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:57:21.2441892Z return super().__call__(*args, **kwargs) 2025-12-04T08:57:21.2442266Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:21.2442666Z return func(*args, **kwargs) 2025-12-04T08:57:21.2443085Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 800, in forward 2025-12-04T08:57:21.2443513Z hidden_states = self.fc2(hidden_states) 2025-12-04T08:57:21.2443674Z 2025-12-04T08:57:21.2443787Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:21.2444177Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:21.2444531Z res = mod(**inputs) 2025-12-04T08:57:21.2444926Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1680, in forward 2025-12-04T08:57:21.2445362Z outputs = self.model.decoder( 2025-12-04T08:57:21.2445789Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1039, in forward 2025-12-04T08:57:21.2446215Z layer_outputs = decoder_layer( 2025-12-04T08:57:21.2446600Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:57:21.2447007Z return super().__call__(*args, **kwargs) 2025-12-04T08:57:21.2447423Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:21.2447829Z return func(*args, **kwargs) 2025-12-04T08:57:21.2448243Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 766, in forward 2025-12-04T08:57:21.2448705Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:57:21.2449251Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:21.2449643Z return func(*args, **kwargs) 2025-12-04T08:57:21.2450030Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 404, in forward 2025-12-04T08:57:21.2450494Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T08:57:21.2450693Z 2025-12-04T08:57:21.2450794Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:21.2451150Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:21.2451467Z res = mod(**inputs) 2025-12-04T08:57:21.2451829Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1680, in forward 2025-12-04T08:57:21.2452210Z outputs = self.model.decoder( 2025-12-04T08:57:21.2452604Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1039, in forward 2025-12-04T08:57:21.2453062Z layer_outputs = decoder_layer( 2025-12-04T08:57:21.2453397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:57:21.2453751Z return super().__call__(*args, **kwargs) 2025-12-04T08:57:21.2454122Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:21.2454487Z return func(*args, **kwargs) 2025-12-04T08:57:21.2454851Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 766, in forward 2025-12-04T08:57:21.2455260Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:57:21.2455679Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:21.2456049Z return func(*args, **kwargs) 2025-12-04T08:57:21.2456428Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 424, in forward 2025-12-04T08:57:21.2456831Z key_states = self.k_proj(current_states) 2025-12-04T08:57:21.2456965Z 2025-12-04T08:57:21.2457074Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:21.2457421Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:21.2457742Z res = mod(**inputs) 2025-12-04T08:57:21.2458109Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1680, in forward 2025-12-04T08:57:21.2458508Z outputs = self.model.decoder( 2025-12-04T08:57:21.2458888Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1039, in forward 2025-12-04T08:57:21.2459286Z layer_outputs = decoder_layer( 2025-12-04T08:57:21.2459637Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:57:21.2459996Z return super().__call__(*args, **kwargs) 2025-12-04T08:57:21.2460374Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:21.2460746Z return func(*args, **kwargs) 2025-12-04T08:57:21.2461124Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 766, in forward 2025-12-04T08:57:21.2461536Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:57:21.2461940Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:21.2462314Z return func(*args, **kwargs) 2025-12-04T08:57:21.2462701Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 425, in forward 2025-12-04T08:57:21.2463104Z value_states = self.v_proj(current_states) 2025-12-04T08:57:21.2463254Z 2025-12-04T08:57:21.2463354Z cudagraph partition due to non gpu ops 2025-12-04T08:57:21.2463591Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:21.2463934Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:21.2464262Z res = mod(**inputs) 2025-12-04T08:57:21.2464624Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1680, in forward 2025-12-04T08:57:21.2465025Z outputs = self.model.decoder( 2025-12-04T08:57:21.2465415Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1039, in forward 2025-12-04T08:57:21.2465812Z layer_outputs = decoder_layer( 2025-12-04T08:57:21.2466190Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:57:21.2466571Z return super().__call__(*args, **kwargs) 2025-12-04T08:57:21.2466948Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:21.2467321Z return func(*args, **kwargs) 2025-12-04T08:57:21.2467704Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 766, in forward 2025-12-04T08:57:21.2468115Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:57:21.2468519Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:21.2468891Z return func(*args, **kwargs) 2025-12-04T08:57:21.2469273Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 443, in forward 2025-12-04T08:57:21.2469710Z attn_output, attn_weights = attention_interface( 2025-12-04T08:57:21.2470181Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T08:57:21.2470709Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T08:57:21.2470907Z 2025-12-04T08:57:21.2471018Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:21.2471412Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:21.2471778Z res = mod(**inputs) 2025-12-04T08:57:21.2472179Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1680, in forward 2025-12-04T08:57:21.2472602Z outputs = self.model.decoder( 2025-12-04T08:57:21.2473025Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1039, in forward 2025-12-04T08:57:21.2473458Z layer_outputs = decoder_layer( 2025-12-04T08:57:21.2473823Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:57:21.2474191Z return super().__call__(*args, **kwargs) 2025-12-04T08:57:21.2474583Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:21.2474996Z return func(*args, **kwargs) 2025-12-04T08:57:21.2475403Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 766, in forward 2025-12-04T08:57:21.2475857Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:57:21.2476289Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:21.2476700Z return func(*args, **kwargs) 2025-12-04T08:57:21.2477110Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 457, in forward 2025-12-04T08:57:21.2477549Z attn_output = self.out_proj(attn_output) 2025-12-04T08:57:21.2477698Z 2025-12-04T08:57:21.2477837Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:21.2478284Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:21.2478644Z res = mod(**inputs) 2025-12-04T08:57:21.2479050Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1680, in forward 2025-12-04T08:57:21.2479491Z outputs = self.model.decoder( 2025-12-04T08:57:21.2479918Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1039, in forward 2025-12-04T08:57:21.2480364Z layer_outputs = decoder_layer( 2025-12-04T08:57:21.2480734Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:57:21.2481120Z return super().__call__(*args, **kwargs) 2025-12-04T08:57:21.2481513Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:21.2481898Z return func(*args, **kwargs) 2025-12-04T08:57:21.2482297Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 798, in forward 2025-12-04T08:57:21.2482752Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:57:21.2482941Z 2025-12-04T08:57:21.2483051Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:21.2483428Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:21.2483767Z res = mod(**inputs) 2025-12-04T08:57:21.2484143Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1680, in forward 2025-12-04T08:57:21.2484574Z outputs = self.model.decoder( 2025-12-04T08:57:21.2484977Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1039, in forward 2025-12-04T08:57:21.2485379Z layer_outputs = decoder_layer( 2025-12-04T08:57:21.2485739Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:57:21.2486115Z return super().__call__(*args, **kwargs) 2025-12-04T08:57:21.2486507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:21.2486879Z return func(*args, **kwargs) 2025-12-04T08:57:21.2487271Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 798, in forward 2025-12-04T08:57:21.2487721Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:57:21.2488125Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T08:57:21.2488475Z return self.act(input) 2025-12-04T08:57:21.2488598Z 2025-12-04T08:57:21.2488703Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:21.2489077Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:21.2489400Z res = mod(**inputs) 2025-12-04T08:57:21.2489782Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1680, in forward 2025-12-04T08:57:21.2490182Z outputs = self.model.decoder( 2025-12-04T08:57:21.2490580Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1039, in forward 2025-12-04T08:57:21.2490979Z layer_outputs = decoder_layer( 2025-12-04T08:57:21.2491340Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:57:21.2491714Z return super().__call__(*args, **kwargs) 2025-12-04T08:57:21.2492099Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:21.2492497Z return func(*args, **kwargs) 2025-12-04T08:57:21.2492894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 800, in forward 2025-12-04T08:57:21.2493305Z hidden_states = self.fc2(hidden_states) 2025-12-04T08:57:21.2493445Z 2025-12-04T08:57:21.2493552Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:21.2493921Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:21.2494252Z res = mod(**inputs) 2025-12-04T08:57:21.2494626Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1696, in forward 2025-12-04T08:57:21.2495029Z logits = self.lm_head(outputs[0]) 2025-12-04T08:57:21.2495170Z 2025-12-04T08:57:21.2495293Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:21.2495692Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:21.2496034Z res = mod(**inputs) 2025-12-04T08:57:21.2496411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1702, in forward 2025-12-04T08:57:21.2496894Z loss = loss_fct(logits.view(-1, self.config.vocab_size), labels.view(-1)) 2025-12-04T08:57:21.2497099Z 2025-12-04T08:57:31.3870749Z Compilation time (from dynamo_timed): 13.289824806 2025-12-04T08:57:31.4267464Z pass 2025-12-04T08:57:31.4267876Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T08:57:31.4268762Z TIMING: _recursive_pre_grad_passes:0.00526 _recursive_joint_graph_passes:0.26146 _recursive_post_grad_passes:0.04983 async_compile.wait:0.81206 code_gen:9.26365 inductor_compile:9.98394 backend_compile:11.79716 gc:0.00116 entire_frame_compile:13.28982 total_wall_time:13.28982 2025-12-04T08:57:31.4270015Z STATS: call_* op count: 180 | FakeTensorMode.__torch_dispatch__:4080 | FakeTensor.__torch_dispatch__:2342 | ProxyTorchDispatchMode.__torch_dispatch__:1092 2025-12-04T08:57:31.4270520Z Dynamo produced 1 graphs covering 180 ops with 0 graph breaks (0 unique) 2025-12-04T08:57:33.7600945Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/__init__.py:65: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-12-04T08:57:33.7601963Z import pynvml # type: ignore[import] 2025-12-04T08:57:37.2618305Z 2025-12-04T08:57:40.6793165Z loading model: 0it [00:00, ?it/s] 2025-12-04T08:57:40.6793457Z loading model: 0it [00:03, ?it/s] 2025-12-04T08:57:40.6810303Z cpu eval PegasusForCausalLM 2025-12-04T08:57:41.5349747Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T08:57:41.9120686Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T08:57:42.2510852Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T08:57:49.5214378Z cudagraph partition due to non gpu ops 2025-12-04T08:57:49.5214958Z cudagraph partition due to non gpu ops 2025-12-04T08:57:49.5215326Z cudagraph partition due to non gpu ops 2025-12-04T08:57:49.5215679Z cudagraph partition due to non gpu ops 2025-12-04T08:57:49.5216050Z cudagraph partition due to non gpu ops 2025-12-04T08:57:49.5216407Z cudagraph partition due to non gpu ops 2025-12-04T08:57:49.5216766Z cudagraph partition due to non gpu ops 2025-12-04T08:57:49.5217094Z cudagraph partition due to non gpu ops 2025-12-04T08:57:49.5217528Z cudagraph partition due to non gpu ops 2025-12-04T08:57:49.5217840Z cudagraph partition due to non gpu ops 2025-12-04T08:57:49.5218121Z cudagraph partition due to non gpu ops 2025-12-04T08:57:49.5218386Z cudagraph partition due to non gpu ops 2025-12-04T08:57:49.5219132Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:49.5219660Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:49.5220149Z res = mod(**inputs) 2025-12-04T08:57:49.5220957Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T08:57:49.5221592Z outputs = self.model.decoder( 2025-12-04T08:57:49.5222192Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T08:57:49.5222803Z layer_outputs = decoder_layer( 2025-12-04T08:57:49.5223331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:57:49.5223917Z return super().__call__(*args, **kwargs) 2025-12-04T08:57:49.5224529Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5225097Z return func(*args, **kwargs) 2025-12-04T08:57:49.5225728Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T08:57:49.5226388Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:57:49.5226995Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5227560Z return func(*args, **kwargs) 2025-12-04T08:57:49.5228120Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 216, in forward 2025-12-04T08:57:49.5228857Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T08:57:49.5229131Z 2025-12-04T08:57:49.5229287Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:49.5229848Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:49.5230356Z res = mod(**inputs) 2025-12-04T08:57:49.5230806Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T08:57:49.5231231Z outputs = self.model.decoder( 2025-12-04T08:57:49.5231844Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T08:57:49.5232397Z layer_outputs = decoder_layer( 2025-12-04T08:57:49.5232794Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:57:49.5233198Z return super().__call__(*args, **kwargs) 2025-12-04T08:57:49.5233594Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5233970Z return func(*args, **kwargs) 2025-12-04T08:57:49.5234459Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T08:57:49.5235018Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:57:49.5235654Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5236234Z return func(*args, **kwargs) 2025-12-04T08:57:49.5236881Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 236, in forward 2025-12-04T08:57:49.5237561Z key_states = self.k_proj(current_states) 2025-12-04T08:57:49.5237766Z 2025-12-04T08:57:49.5237921Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:49.5238699Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:49.5239210Z res = mod(**inputs) 2025-12-04T08:57:49.5239884Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T08:57:49.5240476Z outputs = self.model.decoder( 2025-12-04T08:57:49.5240924Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T08:57:49.5241501Z layer_outputs = decoder_layer( 2025-12-04T08:57:49.5242005Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:57:49.5242505Z return super().__call__(*args, **kwargs) 2025-12-04T08:57:49.5243091Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5243687Z return func(*args, **kwargs) 2025-12-04T08:57:49.5244332Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T08:57:49.5245049Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:57:49.5245674Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5246260Z return func(*args, **kwargs) 2025-12-04T08:57:49.5246850Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 237, in forward 2025-12-04T08:57:49.5247431Z value_states = self.v_proj(current_states) 2025-12-04T08:57:49.5247643Z 2025-12-04T08:57:49.5247745Z cudagraph partition due to non gpu ops 2025-12-04T08:57:49.5248051Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:49.5248605Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:49.5249135Z res = mod(**inputs) 2025-12-04T08:57:49.5249665Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T08:57:49.5250282Z outputs = self.model.decoder( 2025-12-04T08:57:49.5250854Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T08:57:49.5251413Z layer_outputs = decoder_layer( 2025-12-04T08:57:49.5251933Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:57:49.5252389Z return super().__call__(*args, **kwargs) 2025-12-04T08:57:49.5252944Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5253328Z return func(*args, **kwargs) 2025-12-04T08:57:49.5253719Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T08:57:49.5254154Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:57:49.5254564Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5254937Z return func(*args, **kwargs) 2025-12-04T08:57:49.5255319Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 255, in forward 2025-12-04T08:57:49.5255743Z attn_output, attn_weights = attention_interface( 2025-12-04T08:57:49.5256197Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T08:57:49.5256691Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T08:57:49.5256877Z 2025-12-04T08:57:49.5256984Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:49.5257354Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:49.5257685Z res = mod(**inputs) 2025-12-04T08:57:49.5258084Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T08:57:49.5258492Z outputs = self.model.decoder( 2025-12-04T08:57:49.5258891Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T08:57:49.5259289Z layer_outputs = decoder_layer( 2025-12-04T08:57:49.5259639Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:57:49.5260007Z return super().__call__(*args, **kwargs) 2025-12-04T08:57:49.5260422Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5260807Z return func(*args, **kwargs) 2025-12-04T08:57:49.5261248Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T08:57:49.5261702Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:57:49.5262107Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5262497Z return func(*args, **kwargs) 2025-12-04T08:57:49.5262886Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 269, in forward 2025-12-04T08:57:49.5263295Z attn_output = self.out_proj(attn_output) 2025-12-04T08:57:49.5263434Z 2025-12-04T08:57:49.5263547Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:49.5263901Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:49.5264223Z res = mod(**inputs) 2025-12-04T08:57:49.5264623Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T08:57:49.5265024Z outputs = self.model.decoder( 2025-12-04T08:57:49.5265421Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T08:57:49.5265825Z layer_outputs = decoder_layer( 2025-12-04T08:57:49.5266176Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:57:49.5266535Z return super().__call__(*args, **kwargs) 2025-12-04T08:57:49.5266917Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5267307Z return func(*args, **kwargs) 2025-12-04T08:57:49.5267693Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 441, in forward 2025-12-04T08:57:49.5268140Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:57:49.5268353Z 2025-12-04T08:57:49.5268463Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:49.5268827Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:49.5269148Z res = mod(**inputs) 2025-12-04T08:57:49.5269525Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T08:57:49.5269951Z outputs = self.model.decoder( 2025-12-04T08:57:49.5270360Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T08:57:49.5270765Z layer_outputs = decoder_layer( 2025-12-04T08:57:49.5271128Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:57:49.5271511Z return super().__call__(*args, **kwargs) 2025-12-04T08:57:49.5271903Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5272291Z return func(*args, **kwargs) 2025-12-04T08:57:49.5272710Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 441, in forward 2025-12-04T08:57:49.5273168Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:57:49.5273560Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T08:57:49.5273917Z return self.act(input) 2025-12-04T08:57:49.5274032Z 2025-12-04T08:57:49.5274147Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:49.5274521Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:49.5274845Z res = mod(**inputs) 2025-12-04T08:57:49.5275249Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T08:57:49.5275682Z outputs = self.model.decoder( 2025-12-04T08:57:49.5276083Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T08:57:49.5276501Z layer_outputs = decoder_layer( 2025-12-04T08:57:49.5276873Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:57:49.5277279Z return super().__call__(*args, **kwargs) 2025-12-04T08:57:49.5277697Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5278195Z return func(*args, **kwargs) 2025-12-04T08:57:49.5278644Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 443, in forward 2025-12-04T08:57:49.5279137Z hidden_states = self.fc2(hidden_states) 2025-12-04T08:57:49.5279301Z 2025-12-04T08:57:49.5279422Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:49.5279842Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:49.5280182Z res = mod(**inputs) 2025-12-04T08:57:49.5280564Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T08:57:49.5280985Z outputs = self.model.decoder( 2025-12-04T08:57:49.5281399Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T08:57:49.5281894Z layer_outputs = decoder_layer( 2025-12-04T08:57:49.5282251Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:57:49.5282632Z return super().__call__(*args, **kwargs) 2025-12-04T08:57:49.5283028Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5283413Z return func(*args, **kwargs) 2025-12-04T08:57:49.5283821Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T08:57:49.5284264Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:57:49.5284680Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5285058Z return func(*args, **kwargs) 2025-12-04T08:57:49.5285457Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 216, in forward 2025-12-04T08:57:49.5285951Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T08:57:49.5286161Z 2025-12-04T08:57:49.5286278Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:49.5286648Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:49.5286983Z res = mod(**inputs) 2025-12-04T08:57:49.5287393Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T08:57:49.5287806Z outputs = self.model.decoder( 2025-12-04T08:57:49.5288210Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T08:57:49.5288618Z layer_outputs = decoder_layer( 2025-12-04T08:57:49.5288980Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:57:49.5289347Z return super().__call__(*args, **kwargs) 2025-12-04T08:57:49.5289740Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5290125Z return func(*args, **kwargs) 2025-12-04T08:57:49.5290534Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T08:57:49.5290998Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:57:49.5291413Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5291809Z return func(*args, **kwargs) 2025-12-04T08:57:49.5292206Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 236, in forward 2025-12-04T08:57:49.5292634Z key_states = self.k_proj(current_states) 2025-12-04T08:57:49.5292778Z 2025-12-04T08:57:49.5292897Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:49.5293273Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:49.5293735Z res = mod(**inputs) 2025-12-04T08:57:49.5294127Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T08:57:49.5294560Z outputs = self.model.decoder( 2025-12-04T08:57:49.5294951Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T08:57:49.5295362Z layer_outputs = decoder_layer( 2025-12-04T08:57:49.5295727Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:57:49.5296105Z return super().__call__(*args, **kwargs) 2025-12-04T08:57:49.5296499Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5296880Z return func(*args, **kwargs) 2025-12-04T08:57:49.5297270Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T08:57:49.5297697Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:57:49.5298105Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5298485Z return func(*args, **kwargs) 2025-12-04T08:57:49.5298875Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 237, in forward 2025-12-04T08:57:49.5299288Z value_states = self.v_proj(current_states) 2025-12-04T08:57:49.5299438Z 2025-12-04T08:57:49.5299521Z cudagraph partition due to non gpu ops 2025-12-04T08:57:49.5299768Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:49.5300131Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:49.5300447Z res = mod(**inputs) 2025-12-04T08:57:49.5300826Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T08:57:49.5301237Z outputs = self.model.decoder( 2025-12-04T08:57:49.5301652Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T08:57:49.5302056Z layer_outputs = decoder_layer( 2025-12-04T08:57:49.5302421Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:57:49.5302800Z return super().__call__(*args, **kwargs) 2025-12-04T08:57:49.5303192Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5303613Z return func(*args, **kwargs) 2025-12-04T08:57:49.5304004Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T08:57:49.5304426Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:57:49.5304850Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5305245Z return func(*args, **kwargs) 2025-12-04T08:57:49.5305637Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 255, in forward 2025-12-04T08:57:49.5306061Z attn_output, attn_weights = attention_interface( 2025-12-04T08:57:49.5306516Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T08:57:49.5307009Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T08:57:49.5307194Z 2025-12-04T08:57:49.5307307Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:49.5307712Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:49.5308060Z res = mod(**inputs) 2025-12-04T08:57:49.5308441Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T08:57:49.5308842Z outputs = self.model.decoder( 2025-12-04T08:57:49.5309238Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T08:57:49.5309639Z layer_outputs = decoder_layer( 2025-12-04T08:57:49.5309990Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:57:49.5310350Z return super().__call__(*args, **kwargs) 2025-12-04T08:57:49.5310731Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5311111Z return func(*args, **kwargs) 2025-12-04T08:57:49.5311488Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T08:57:49.5311917Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:57:49.5312317Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5312694Z return func(*args, **kwargs) 2025-12-04T08:57:49.5313075Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 269, in forward 2025-12-04T08:57:49.5313473Z attn_output = self.out_proj(attn_output) 2025-12-04T08:57:49.5313613Z 2025-12-04T08:57:49.5313715Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:49.5314309Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:49.5314621Z res = mod(**inputs) 2025-12-04T08:57:49.5314993Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T08:57:49.5315403Z outputs = self.model.decoder( 2025-12-04T08:57:49.5315797Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T08:57:49.5316193Z layer_outputs = decoder_layer( 2025-12-04T08:57:49.5316582Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:57:49.5316949Z return super().__call__(*args, **kwargs) 2025-12-04T08:57:49.5317327Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5317714Z return func(*args, **kwargs) 2025-12-04T08:57:49.5318198Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 441, in forward 2025-12-04T08:57:49.5318696Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:57:49.5318898Z 2025-12-04T08:57:49.5319017Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:49.5319429Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:49.5319788Z res = mod(**inputs) 2025-12-04T08:57:49.5320171Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T08:57:49.5320638Z outputs = self.model.decoder( 2025-12-04T08:57:49.5321255Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T08:57:49.5321655Z layer_outputs = decoder_layer( 2025-12-04T08:57:49.5321994Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:57:49.5322357Z return super().__call__(*args, **kwargs) 2025-12-04T08:57:49.5322736Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5323179Z return func(*args, **kwargs) 2025-12-04T08:57:49.5323560Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 441, in forward 2025-12-04T08:57:49.5323999Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:57:49.5324380Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T08:57:49.5324716Z return self.act(input) 2025-12-04T08:57:49.5324833Z 2025-12-04T08:57:49.5324938Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:49.5325298Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:49.5325619Z res = mod(**inputs) 2025-12-04T08:57:49.5325978Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T08:57:49.5326376Z outputs = self.model.decoder( 2025-12-04T08:57:49.5326763Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T08:57:49.5327148Z layer_outputs = decoder_layer( 2025-12-04T08:57:49.5327494Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:57:49.5327856Z return super().__call__(*args, **kwargs) 2025-12-04T08:57:49.5328227Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5328585Z return func(*args, **kwargs) 2025-12-04T08:57:49.5328959Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 443, in forward 2025-12-04T08:57:49.5329354Z hidden_states = self.fc2(hidden_states) 2025-12-04T08:57:49.5329487Z 2025-12-04T08:57:49.5329597Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:49.5329944Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:49.5330261Z res = mod(**inputs) 2025-12-04T08:57:49.5330652Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T08:57:49.5331043Z outputs = self.model.decoder( 2025-12-04T08:57:49.5331428Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T08:57:49.5331820Z layer_outputs = decoder_layer( 2025-12-04T08:57:49.5332163Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:57:49.5332509Z return super().__call__(*args, **kwargs) 2025-12-04T08:57:49.5332878Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5333244Z return func(*args, **kwargs) 2025-12-04T08:57:49.5333639Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T08:57:49.5334081Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:57:49.5334483Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5334856Z return func(*args, **kwargs) 2025-12-04T08:57:49.5335233Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 216, in forward 2025-12-04T08:57:49.5335709Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T08:57:49.5335923Z 2025-12-04T08:57:49.5336033Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:49.5336385Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:49.5336719Z res = mod(**inputs) 2025-12-04T08:57:49.5337095Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T08:57:49.5337499Z outputs = self.model.decoder( 2025-12-04T08:57:49.5337887Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T08:57:49.5338287Z layer_outputs = decoder_layer( 2025-12-04T08:57:49.5338639Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:57:49.5339003Z return super().__call__(*args, **kwargs) 2025-12-04T08:57:49.5339375Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5339748Z return func(*args, **kwargs) 2025-12-04T08:57:49.5340134Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T08:57:49.5340558Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:57:49.5340954Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5341330Z return func(*args, **kwargs) 2025-12-04T08:57:49.5341715Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 236, in forward 2025-12-04T08:57:49.5342116Z key_states = self.k_proj(current_states) 2025-12-04T08:57:49.5342263Z 2025-12-04T08:57:49.5342367Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:49.5342728Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:49.5343051Z res = mod(**inputs) 2025-12-04T08:57:49.5343420Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T08:57:49.5343830Z outputs = self.model.decoder( 2025-12-04T08:57:49.5344227Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T08:57:49.5344640Z layer_outputs = decoder_layer( 2025-12-04T08:57:49.5344996Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:57:49.5345359Z return super().__call__(*args, **kwargs) 2025-12-04T08:57:49.5345740Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5346105Z return func(*args, **kwargs) 2025-12-04T08:57:49.5346492Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T08:57:49.5346915Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:57:49.5347340Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5347724Z return func(*args, **kwargs) 2025-12-04T08:57:49.5348251Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 237, in forward 2025-12-04T08:57:49.5348675Z value_states = self.v_proj(current_states) 2025-12-04T08:57:49.5348819Z 2025-12-04T08:57:49.5348903Z cudagraph partition due to non gpu ops 2025-12-04T08:57:49.5349150Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:49.5349515Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:49.5349841Z res = mod(**inputs) 2025-12-04T08:57:49.5350210Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T08:57:49.5350650Z outputs = self.model.decoder( 2025-12-04T08:57:49.5351052Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T08:57:49.5351458Z layer_outputs = decoder_layer( 2025-12-04T08:57:49.5351819Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:57:49.5352195Z return super().__call__(*args, **kwargs) 2025-12-04T08:57:49.5352587Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5352967Z return func(*args, **kwargs) 2025-12-04T08:57:49.5353361Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T08:57:49.5353793Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:57:49.5354192Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5354583Z return func(*args, **kwargs) 2025-12-04T08:57:49.5354979Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 255, in forward 2025-12-04T08:57:49.5355420Z attn_output, attn_weights = attention_interface( 2025-12-04T08:57:49.5355879Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T08:57:49.5356384Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T08:57:49.5356581Z 2025-12-04T08:57:49.5356693Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:49.5357092Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:49.5357443Z res = mod(**inputs) 2025-12-04T08:57:49.5357870Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T08:57:49.5358405Z outputs = self.model.decoder( 2025-12-04T08:57:49.5358850Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T08:57:49.5359329Z layer_outputs = decoder_layer( 2025-12-04T08:57:49.5359711Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:57:49.5360094Z return super().__call__(*args, **kwargs) 2025-12-04T08:57:49.5360486Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5360874Z return func(*args, **kwargs) 2025-12-04T08:57:49.5361280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T08:57:49.5361723Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:57:49.5362154Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5362561Z return func(*args, **kwargs) 2025-12-04T08:57:49.5362964Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 269, in forward 2025-12-04T08:57:49.5363379Z attn_output = self.out_proj(attn_output) 2025-12-04T08:57:49.5363535Z 2025-12-04T08:57:49.5363648Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:49.5364023Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:49.5364358Z res = mod(**inputs) 2025-12-04T08:57:49.5364741Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T08:57:49.5365158Z outputs = self.model.decoder( 2025-12-04T08:57:49.5365611Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T08:57:49.5366023Z layer_outputs = decoder_layer( 2025-12-04T08:57:49.5366378Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:57:49.5366751Z return super().__call__(*args, **kwargs) 2025-12-04T08:57:49.5367145Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5367523Z return func(*args, **kwargs) 2025-12-04T08:57:49.5367922Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 441, in forward 2025-12-04T08:57:49.5368377Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:57:49.5368556Z 2025-12-04T08:57:49.5368671Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:49.5369039Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:49.5369372Z res = mod(**inputs) 2025-12-04T08:57:49.5369783Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T08:57:49.5370198Z outputs = self.model.decoder( 2025-12-04T08:57:49.5370605Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T08:57:49.5371017Z layer_outputs = decoder_layer( 2025-12-04T08:57:49.5371378Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:57:49.5371747Z return super().__call__(*args, **kwargs) 2025-12-04T08:57:49.5372140Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5372526Z return func(*args, **kwargs) 2025-12-04T08:57:49.5372923Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 441, in forward 2025-12-04T08:57:49.5373376Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:57:49.5373775Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T08:57:49.5374117Z return self.act(input) 2025-12-04T08:57:49.5374227Z 2025-12-04T08:57:49.5374330Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:49.5374690Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:49.5375005Z res = mod(**inputs) 2025-12-04T08:57:49.5375370Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T08:57:49.5375760Z outputs = self.model.decoder( 2025-12-04T08:57:49.5376143Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T08:57:49.5376550Z layer_outputs = decoder_layer( 2025-12-04T08:57:49.5376902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:57:49.5377262Z return super().__call__(*args, **kwargs) 2025-12-04T08:57:49.5377636Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5378000Z return func(*args, **kwargs) 2025-12-04T08:57:49.5378366Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 443, in forward 2025-12-04T08:57:49.5378762Z hidden_states = self.fc2(hidden_states) 2025-12-04T08:57:49.5378895Z 2025-12-04T08:57:49.5379003Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:49.5379359Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:49.5379692Z res = mod(**inputs) 2025-12-04T08:57:49.5380058Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T08:57:49.5380462Z outputs = self.model.decoder( 2025-12-04T08:57:49.5380846Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T08:57:49.5381243Z layer_outputs = decoder_layer( 2025-12-04T08:57:49.5381592Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:57:49.5381956Z return super().__call__(*args, **kwargs) 2025-12-04T08:57:49.5382327Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5382697Z return func(*args, **kwargs) 2025-12-04T08:57:49.5383083Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 445, in forward 2025-12-04T08:57:49.5383480Z hidden_states = residual + hidden_states 2025-12-04T08:57:49.5383626Z 2025-12-04T08:57:49.5383733Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:49.5384094Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:49.5384416Z res = mod(**inputs) 2025-12-04T08:57:49.5384780Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T08:57:49.5385177Z outputs = self.model.decoder( 2025-12-04T08:57:49.5385569Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T08:57:49.5385966Z layer_outputs = decoder_layer( 2025-12-04T08:57:49.5386307Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:57:49.5386671Z return super().__call__(*args, **kwargs) 2025-12-04T08:57:49.5387064Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5387457Z return func(*args, **kwargs) 2025-12-04T08:57:49.5387830Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T08:57:49.5388236Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:57:49.5388619Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5388968Z return func(*args, **kwargs) 2025-12-04T08:57:49.5389333Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 216, in forward 2025-12-04T08:57:49.5389781Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T08:57:49.5389976Z 2025-12-04T08:57:49.5390097Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:49.5390446Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:49.5390753Z res = mod(**inputs) 2025-12-04T08:57:49.5391111Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T08:57:49.5391486Z outputs = self.model.decoder( 2025-12-04T08:57:49.5391862Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T08:57:49.5392294Z layer_outputs = decoder_layer( 2025-12-04T08:57:49.5392634Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:57:49.5392981Z return super().__call__(*args, **kwargs) 2025-12-04T08:57:49.5393372Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5393735Z return func(*args, **kwargs) 2025-12-04T08:57:49.5394105Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T08:57:49.5394524Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:57:49.5394912Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5395275Z return func(*args, **kwargs) 2025-12-04T08:57:49.5395644Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 236, in forward 2025-12-04T08:57:49.5396050Z key_states = self.k_proj(current_states) 2025-12-04T08:57:49.5396187Z 2025-12-04T08:57:49.5396300Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:49.5396662Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:49.5396990Z res = mod(**inputs) 2025-12-04T08:57:49.5397377Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T08:57:49.5397791Z outputs = self.model.decoder( 2025-12-04T08:57:49.5398323Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T08:57:49.5398758Z layer_outputs = decoder_layer( 2025-12-04T08:57:49.5399161Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:57:49.5399576Z return super().__call__(*args, **kwargs) 2025-12-04T08:57:49.5399999Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5400404Z return func(*args, **kwargs) 2025-12-04T08:57:49.5400797Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T08:57:49.5401218Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:57:49.5401646Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5402014Z return func(*args, **kwargs) 2025-12-04T08:57:49.5402395Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 237, in forward 2025-12-04T08:57:49.5402795Z value_states = self.v_proj(current_states) 2025-12-04T08:57:49.5402944Z 2025-12-04T08:57:49.5403027Z cudagraph partition due to non gpu ops 2025-12-04T08:57:49.5403270Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:49.5403620Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:49.5403947Z res = mod(**inputs) 2025-12-04T08:57:49.5404337Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T08:57:49.5404753Z outputs = self.model.decoder( 2025-12-04T08:57:49.5405130Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T08:57:49.5405524Z layer_outputs = decoder_layer( 2025-12-04T08:57:49.5405867Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:57:49.5406223Z return super().__call__(*args, **kwargs) 2025-12-04T08:57:49.5406590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5406958Z return func(*args, **kwargs) 2025-12-04T08:57:49.5407339Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T08:57:49.5407779Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:57:49.5408172Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5408543Z return func(*args, **kwargs) 2025-12-04T08:57:49.5408920Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 255, in forward 2025-12-04T08:57:49.5409328Z attn_output, attn_weights = attention_interface( 2025-12-04T08:57:49.5409766Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T08:57:49.5410240Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T08:57:49.5410420Z 2025-12-04T08:57:49.5410529Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:49.5410877Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:49.5411200Z res = mod(**inputs) 2025-12-04T08:57:49.5411568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T08:57:49.5411961Z outputs = self.model.decoder( 2025-12-04T08:57:49.5412349Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T08:57:49.5412746Z layer_outputs = decoder_layer( 2025-12-04T08:57:49.5413088Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:57:49.5413443Z return super().__call__(*args, **kwargs) 2025-12-04T08:57:49.5413818Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5414191Z return func(*args, **kwargs) 2025-12-04T08:57:49.5414565Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T08:57:49.5414988Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:57:49.5415397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5415763Z return func(*args, **kwargs) 2025-12-04T08:57:49.5416132Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 269, in forward 2025-12-04T08:57:49.5416535Z attn_output = self.out_proj(attn_output) 2025-12-04T08:57:49.5416672Z 2025-12-04T08:57:49.5416785Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:49.5417145Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:49.5417459Z res = mod(**inputs) 2025-12-04T08:57:49.5417852Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T08:57:49.5418322Z outputs = self.model.decoder( 2025-12-04T08:57:49.5418719Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T08:57:49.5419131Z layer_outputs = decoder_layer( 2025-12-04T08:57:49.5419478Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:57:49.5419838Z return super().__call__(*args, **kwargs) 2025-12-04T08:57:49.5420207Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5420577Z return func(*args, **kwargs) 2025-12-04T08:57:49.5421202Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 441, in forward 2025-12-04T08:57:49.5421698Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:57:49.5421876Z 2025-12-04T08:57:49.5421982Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:49.5422342Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:49.5422670Z res = mod(**inputs) 2025-12-04T08:57:49.5423048Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T08:57:49.5423460Z outputs = self.model.decoder( 2025-12-04T08:57:49.5423865Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T08:57:49.5424335Z layer_outputs = decoder_layer( 2025-12-04T08:57:49.5424681Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:57:49.5425053Z return super().__call__(*args, **kwargs) 2025-12-04T08:57:49.5425439Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5425812Z return func(*args, **kwargs) 2025-12-04T08:57:49.5426207Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 441, in forward 2025-12-04T08:57:49.5426650Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:57:49.5427036Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T08:57:49.5427382Z return self.act(input) 2025-12-04T08:57:49.5427499Z 2025-12-04T08:57:49.5427604Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:49.5427972Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:49.5428292Z res = mod(**inputs) 2025-12-04T08:57:49.5428670Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T08:57:49.5429078Z outputs = self.model.decoder( 2025-12-04T08:57:49.5429501Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T08:57:49.5429899Z layer_outputs = decoder_layer( 2025-12-04T08:57:49.5430252Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:57:49.5430623Z return super().__call__(*args, **kwargs) 2025-12-04T08:57:49.5431011Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5431386Z return func(*args, **kwargs) 2025-12-04T08:57:49.5431778Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 443, in forward 2025-12-04T08:57:49.5432193Z hidden_states = self.fc2(hidden_states) 2025-12-04T08:57:49.5432332Z 2025-12-04T08:57:49.5432468Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:49.5432853Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:49.5433181Z res = mod(**inputs) 2025-12-04T08:57:49.5433556Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T08:57:49.5433953Z outputs = self.model.decoder( 2025-12-04T08:57:49.5434349Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T08:57:49.5434749Z layer_outputs = decoder_layer( 2025-12-04T08:57:49.5435090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:57:49.5435456Z return super().__call__(*args, **kwargs) 2025-12-04T08:57:49.5435870Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5436256Z return func(*args, **kwargs) 2025-12-04T08:57:49.5436649Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T08:57:49.5437087Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:57:49.5437502Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5437891Z return func(*args, **kwargs) 2025-12-04T08:57:49.5438381Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 216, in forward 2025-12-04T08:57:49.5438928Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T08:57:49.5439165Z 2025-12-04T08:57:49.5439279Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:49.5439649Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:49.5439985Z res = mod(**inputs) 2025-12-04T08:57:49.5440375Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T08:57:49.5440795Z outputs = self.model.decoder( 2025-12-04T08:57:49.5441190Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T08:57:49.5441604Z layer_outputs = decoder_layer( 2025-12-04T08:57:49.5441965Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:57:49.5442343Z return super().__call__(*args, **kwargs) 2025-12-04T08:57:49.5442729Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5443106Z return func(*args, **kwargs) 2025-12-04T08:57:49.5443492Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T08:57:49.5443928Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:57:49.5444353Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5444729Z return func(*args, **kwargs) 2025-12-04T08:57:49.5445116Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 236, in forward 2025-12-04T08:57:49.5445514Z key_states = self.k_proj(current_states) 2025-12-04T08:57:49.5445661Z 2025-12-04T08:57:49.5445768Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:49.5446126Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:49.5446443Z res = mod(**inputs) 2025-12-04T08:57:49.5446843Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T08:57:49.5447280Z outputs = self.model.decoder( 2025-12-04T08:57:49.5447680Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T08:57:49.5448074Z layer_outputs = decoder_layer( 2025-12-04T08:57:49.5448429Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:57:49.5448807Z return super().__call__(*args, **kwargs) 2025-12-04T08:57:49.5449200Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5449579Z return func(*args, **kwargs) 2025-12-04T08:57:49.5449978Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T08:57:49.5450432Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:57:49.5450839Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5451227Z return func(*args, **kwargs) 2025-12-04T08:57:49.5451628Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 237, in forward 2025-12-04T08:57:49.5452052Z value_states = self.v_proj(current_states) 2025-12-04T08:57:49.5452197Z 2025-12-04T08:57:49.5452282Z cudagraph partition due to non gpu ops 2025-12-04T08:57:49.5452531Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:49.5452904Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:49.5453228Z res = mod(**inputs) 2025-12-04T08:57:49.5453619Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T08:57:49.5454037Z outputs = self.model.decoder( 2025-12-04T08:57:49.5454443Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T08:57:49.5454851Z layer_outputs = decoder_layer( 2025-12-04T08:57:49.5455211Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:57:49.5455586Z return super().__call__(*args, **kwargs) 2025-12-04T08:57:49.5455975Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5456349Z return func(*args, **kwargs) 2025-12-04T08:57:49.5456744Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T08:57:49.5457175Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:57:49.5457583Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5457964Z return func(*args, **kwargs) 2025-12-04T08:57:49.5458378Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 255, in forward 2025-12-04T08:57:49.5458823Z attn_output, attn_weights = attention_interface( 2025-12-04T08:57:49.5459289Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T08:57:49.5459800Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T08:57:49.5459995Z 2025-12-04T08:57:49.5460115Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:49.5460500Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:49.5460840Z res = mod(**inputs) 2025-12-04T08:57:49.5461260Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T08:57:49.5461697Z outputs = self.model.decoder( 2025-12-04T08:57:49.5462104Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T08:57:49.5462521Z layer_outputs = decoder_layer( 2025-12-04T08:57:49.5462883Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:57:49.5463263Z return super().__call__(*args, **kwargs) 2025-12-04T08:57:49.5463655Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5464043Z return func(*args, **kwargs) 2025-12-04T08:57:49.5464450Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T08:57:49.5464888Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:57:49.5465299Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5465679Z return func(*args, **kwargs) 2025-12-04T08:57:49.5466069Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 269, in forward 2025-12-04T08:57:49.5466475Z attn_output = self.out_proj(attn_output) 2025-12-04T08:57:49.5466623Z 2025-12-04T08:57:49.5466729Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:49.5467093Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:49.5467420Z res = mod(**inputs) 2025-12-04T08:57:49.5467788Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T08:57:49.5468192Z outputs = self.model.decoder( 2025-12-04T08:57:49.5468589Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T08:57:49.5468984Z layer_outputs = decoder_layer( 2025-12-04T08:57:49.5469335Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:57:49.5469699Z return super().__call__(*args, **kwargs) 2025-12-04T08:57:49.5470079Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5470445Z return func(*args, **kwargs) 2025-12-04T08:57:49.5470831Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 441, in forward 2025-12-04T08:57:49.5471291Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:57:49.5471470Z 2025-12-04T08:57:49.5471595Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:49.5471950Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:49.5472273Z res = mod(**inputs) 2025-12-04T08:57:49.5472668Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T08:57:49.5473066Z outputs = self.model.decoder( 2025-12-04T08:57:49.5473462Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T08:57:49.5473864Z layer_outputs = decoder_layer( 2025-12-04T08:57:49.5474214Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:57:49.5474572Z return super().__call__(*args, **kwargs) 2025-12-04T08:57:49.5474960Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5475340Z return func(*args, **kwargs) 2025-12-04T08:57:49.5475746Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 441, in forward 2025-12-04T08:57:49.5476214Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:57:49.5476611Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T08:57:49.5476967Z return self.act(input) 2025-12-04T08:57:49.5477083Z 2025-12-04T08:57:49.5477194Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:49.5477573Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:49.5477907Z res = mod(**inputs) 2025-12-04T08:57:49.5478411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T08:57:49.5478894Z outputs = self.model.decoder( 2025-12-04T08:57:49.5479335Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T08:57:49.5479783Z layer_outputs = decoder_layer( 2025-12-04T08:57:49.5480154Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:57:49.5480580Z return super().__call__(*args, **kwargs) 2025-12-04T08:57:49.5480977Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5481365Z return func(*args, **kwargs) 2025-12-04T08:57:49.5481760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 443, in forward 2025-12-04T08:57:49.5482181Z hidden_states = self.fc2(hidden_states) 2025-12-04T08:57:49.5482324Z 2025-12-04T08:57:49.5482441Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:49.5482814Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:49.5483204Z res = mod(**inputs) 2025-12-04T08:57:49.5483592Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T08:57:49.5484009Z outputs = self.model.decoder( 2025-12-04T08:57:49.5484408Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T08:57:49.5484824Z layer_outputs = decoder_layer( 2025-12-04T08:57:49.5485186Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:57:49.5485554Z return super().__call__(*args, **kwargs) 2025-12-04T08:57:49.5485954Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5486344Z return func(*args, **kwargs) 2025-12-04T08:57:49.5486748Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 445, in forward 2025-12-04T08:57:49.5487165Z hidden_states = residual + hidden_states 2025-12-04T08:57:49.5487338Z 2025-12-04T08:57:49.5487449Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:49.5487823Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:49.5488158Z res = mod(**inputs) 2025-12-04T08:57:49.5488542Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T08:57:49.5488957Z outputs = self.model.decoder( 2025-12-04T08:57:49.5489360Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T08:57:49.5489769Z layer_outputs = decoder_layer( 2025-12-04T08:57:49.5490148Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:57:49.5490543Z return super().__call__(*args, **kwargs) 2025-12-04T08:57:49.5490941Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5491320Z return func(*args, **kwargs) 2025-12-04T08:57:49.5491720Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T08:57:49.5492162Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:57:49.5492579Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5492964Z return func(*args, **kwargs) 2025-12-04T08:57:49.5493351Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 216, in forward 2025-12-04T08:57:49.5493861Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T08:57:49.5494068Z 2025-12-04T08:57:49.5494172Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:49.5494556Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:49.5494897Z res = mod(**inputs) 2025-12-04T08:57:49.5495304Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T08:57:49.5495760Z outputs = self.model.decoder( 2025-12-04T08:57:49.5496174Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T08:57:49.5496592Z layer_outputs = decoder_layer( 2025-12-04T08:57:49.5496943Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:57:49.5497322Z return super().__call__(*args, **kwargs) 2025-12-04T08:57:49.5497706Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5498084Z return func(*args, **kwargs) 2025-12-04T08:57:49.5498468Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T08:57:49.5498891Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:57:49.5499298Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5499672Z return func(*args, **kwargs) 2025-12-04T08:57:49.5500052Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 236, in forward 2025-12-04T08:57:49.5500463Z key_states = self.k_proj(current_states) 2025-12-04T08:57:49.5500599Z 2025-12-04T08:57:49.5500712Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:49.5501067Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:49.5501394Z res = mod(**inputs) 2025-12-04T08:57:49.5501794Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T08:57:49.5502199Z outputs = self.model.decoder( 2025-12-04T08:57:49.5502587Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T08:57:49.5502988Z layer_outputs = decoder_layer( 2025-12-04T08:57:49.5503336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:57:49.5503693Z return super().__call__(*args, **kwargs) 2025-12-04T08:57:49.5504076Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5504469Z return func(*args, **kwargs) 2025-12-04T08:57:49.5504879Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T08:57:49.5505302Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:57:49.5505712Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5506093Z return func(*args, **kwargs) 2025-12-04T08:57:49.5506489Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 237, in forward 2025-12-04T08:57:49.5506906Z value_states = self.v_proj(current_states) 2025-12-04T08:57:49.5507057Z 2025-12-04T08:57:49.5507142Z cudagraph partition due to non gpu ops 2025-12-04T08:57:49.5507394Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:49.5507767Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:49.5508096Z res = mod(**inputs) 2025-12-04T08:57:49.5508482Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T08:57:49.5508885Z outputs = self.model.decoder( 2025-12-04T08:57:49.5509269Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T08:57:49.5509671Z layer_outputs = decoder_layer( 2025-12-04T08:57:49.5510029Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:57:49.5510394Z return super().__call__(*args, **kwargs) 2025-12-04T08:57:49.5510783Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5511176Z return func(*args, **kwargs) 2025-12-04T08:57:49.5511562Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T08:57:49.5511974Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:57:49.5512378Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5512749Z return func(*args, **kwargs) 2025-12-04T08:57:49.5513133Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 255, in forward 2025-12-04T08:57:49.5513546Z attn_output, attn_weights = attention_interface( 2025-12-04T08:57:49.5513994Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T08:57:49.5514477Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T08:57:49.5514658Z 2025-12-04T08:57:49.5514762Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:49.5515126Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:49.5515450Z res = mod(**inputs) 2025-12-04T08:57:49.5515839Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T08:57:49.5516237Z outputs = self.model.decoder( 2025-12-04T08:57:49.5516631Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T08:57:49.5517043Z layer_outputs = decoder_layer( 2025-12-04T08:57:49.5517398Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:57:49.5517773Z return super().__call__(*args, **kwargs) 2025-12-04T08:57:49.5518263Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5518694Z return func(*args, **kwargs) 2025-12-04T08:57:49.5519162Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T08:57:49.5519646Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:57:49.5520074Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5520462Z return func(*args, **kwargs) 2025-12-04T08:57:49.5521104Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 269, in forward 2025-12-04T08:57:49.5521539Z attn_output = self.out_proj(attn_output) 2025-12-04T08:57:49.5521683Z 2025-12-04T08:57:49.5521799Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:49.5522173Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:49.5522554Z res = mod(**inputs) 2025-12-04T08:57:49.5522937Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T08:57:49.5523347Z outputs = self.model.decoder( 2025-12-04T08:57:49.5523738Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T08:57:49.5524146Z layer_outputs = decoder_layer( 2025-12-04T08:57:49.5524490Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:57:49.5524858Z return super().__call__(*args, **kwargs) 2025-12-04T08:57:49.5525235Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5525615Z return func(*args, **kwargs) 2025-12-04T08:57:49.5526005Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 441, in forward 2025-12-04T08:57:49.5526451Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:57:49.5526631Z 2025-12-04T08:57:49.5526738Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:49.5527098Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:49.5527421Z res = mod(**inputs) 2025-12-04T08:57:49.5527791Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T08:57:49.5528191Z outputs = self.model.decoder( 2025-12-04T08:57:49.5528618Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T08:57:49.5529017Z layer_outputs = decoder_layer( 2025-12-04T08:57:49.5529368Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:57:49.5529739Z return super().__call__(*args, **kwargs) 2025-12-04T08:57:49.5530124Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5530580Z return func(*args, **kwargs) 2025-12-04T08:57:49.5530970Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 441, in forward 2025-12-04T08:57:49.5531414Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:57:49.5531806Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T08:57:49.5532146Z return self.act(input) 2025-12-04T08:57:49.5532263Z 2025-12-04T08:57:49.5532368Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:49.5532729Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:49.5533049Z res = mod(**inputs) 2025-12-04T08:57:49.5533448Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T08:57:49.5533880Z outputs = self.model.decoder( 2025-12-04T08:57:49.5534275Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T08:57:49.5534674Z layer_outputs = decoder_layer( 2025-12-04T08:57:49.5535024Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:57:49.5535386Z return super().__call__(*args, **kwargs) 2025-12-04T08:57:49.5535761Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5536138Z return func(*args, **kwargs) 2025-12-04T08:57:49.5536527Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 443, in forward 2025-12-04T08:57:49.5536957Z hidden_states = self.fc2(hidden_states) 2025-12-04T08:57:49.5537104Z 2025-12-04T08:57:49.5537213Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:49.5537590Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:49.5537922Z res = mod(**inputs) 2025-12-04T08:57:49.5538308Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T08:57:49.5538719Z outputs = self.model.decoder( 2025-12-04T08:57:49.5539123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T08:57:49.5539590Z layer_outputs = decoder_layer( 2025-12-04T08:57:49.5539937Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:57:49.5540324Z return super().__call__(*args, **kwargs) 2025-12-04T08:57:49.5540720Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5541112Z return func(*args, **kwargs) 2025-12-04T08:57:49.5541512Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T08:57:49.5541951Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:57:49.5542368Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5542746Z return func(*args, **kwargs) 2025-12-04T08:57:49.5543149Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 216, in forward 2025-12-04T08:57:49.5543637Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T08:57:49.5543850Z 2025-12-04T08:57:49.5543969Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:49.5544336Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:49.5544688Z res = mod(**inputs) 2025-12-04T08:57:49.5545075Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T08:57:49.5545494Z outputs = self.model.decoder( 2025-12-04T08:57:49.5545905Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T08:57:49.5546324Z layer_outputs = decoder_layer( 2025-12-04T08:57:49.5546691Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:57:49.5547066Z return super().__call__(*args, **kwargs) 2025-12-04T08:57:49.5547467Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5547903Z return func(*args, **kwargs) 2025-12-04T08:57:49.5548308Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T08:57:49.5548741Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:57:49.5549158Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5549545Z return func(*args, **kwargs) 2025-12-04T08:57:49.5549935Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 236, in forward 2025-12-04T08:57:49.5550353Z key_states = self.k_proj(current_states) 2025-12-04T08:57:49.5550502Z 2025-12-04T08:57:49.5550611Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:49.5551005Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:49.5551335Z res = mod(**inputs) 2025-12-04T08:57:49.5551721Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T08:57:49.5552132Z outputs = self.model.decoder( 2025-12-04T08:57:49.5552535Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T08:57:49.5552937Z layer_outputs = decoder_layer( 2025-12-04T08:57:49.5553296Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:57:49.5553668Z return super().__call__(*args, **kwargs) 2025-12-04T08:57:49.5554053Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5554439Z return func(*args, **kwargs) 2025-12-04T08:57:49.5554838Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T08:57:49.5555272Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:57:49.5555679Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5556061Z return func(*args, **kwargs) 2025-12-04T08:57:49.5556462Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 237, in forward 2025-12-04T08:57:49.5556874Z value_states = self.v_proj(current_states) 2025-12-04T08:57:49.5557021Z 2025-12-04T08:57:49.5557104Z cudagraph partition due to non gpu ops 2025-12-04T08:57:49.5557356Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:49.5557727Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:49.5558055Z res = mod(**inputs) 2025-12-04T08:57:49.5558517Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T08:57:49.5558937Z outputs = self.model.decoder( 2025-12-04T08:57:49.5559373Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T08:57:49.5559806Z layer_outputs = decoder_layer( 2025-12-04T08:57:49.5560190Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:57:49.5560590Z return super().__call__(*args, **kwargs) 2025-12-04T08:57:49.5561005Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5561417Z return func(*args, **kwargs) 2025-12-04T08:57:49.5561823Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T08:57:49.5562288Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:57:49.5562715Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5563102Z return func(*args, **kwargs) 2025-12-04T08:57:49.5563497Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 255, in forward 2025-12-04T08:57:49.5563918Z attn_output, attn_weights = attention_interface( 2025-12-04T08:57:49.5564359Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T08:57:49.5564843Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T08:57:49.5565028Z 2025-12-04T08:57:49.5565143Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:49.5565535Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:49.5565873Z res = mod(**inputs) 2025-12-04T08:57:49.5566272Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T08:57:49.5566683Z outputs = self.model.decoder( 2025-12-04T08:57:49.5567077Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T08:57:49.5567484Z layer_outputs = decoder_layer( 2025-12-04T08:57:49.5567840Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:57:49.5568204Z return super().__call__(*args, **kwargs) 2025-12-04T08:57:49.5568590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5568970Z return func(*args, **kwargs) 2025-12-04T08:57:49.5569368Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T08:57:49.5569776Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:57:49.5570171Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5570535Z return func(*args, **kwargs) 2025-12-04T08:57:49.5570907Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 269, in forward 2025-12-04T08:57:49.5571307Z attn_output = self.out_proj(attn_output) 2025-12-04T08:57:49.5571452Z 2025-12-04T08:57:49.5571555Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:49.5571916Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:49.5572233Z res = mod(**inputs) 2025-12-04T08:57:49.5572613Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T08:57:49.5573021Z outputs = self.model.decoder( 2025-12-04T08:57:49.5573436Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T08:57:49.5573837Z layer_outputs = decoder_layer( 2025-12-04T08:57:49.5574188Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:57:49.5574557Z return super().__call__(*args, **kwargs) 2025-12-04T08:57:49.5574934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5575313Z return func(*args, **kwargs) 2025-12-04T08:57:49.5575698Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 441, in forward 2025-12-04T08:57:49.5576144Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:57:49.5576315Z 2025-12-04T08:57:49.5576440Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:49.5576825Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:49.5577157Z res = mod(**inputs) 2025-12-04T08:57:49.5577536Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T08:57:49.5577931Z outputs = self.model.decoder( 2025-12-04T08:57:49.5578325Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T08:57:49.5578733Z layer_outputs = decoder_layer( 2025-12-04T08:57:49.5579079Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:57:49.5579450Z return super().__call__(*args, **kwargs) 2025-12-04T08:57:49.5579863Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5580237Z return func(*args, **kwargs) 2025-12-04T08:57:49.5580617Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 441, in forward 2025-12-04T08:57:49.5581070Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:57:49.5581456Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T08:57:49.5581801Z return self.act(input) 2025-12-04T08:57:49.5581911Z 2025-12-04T08:57:49.5582015Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:49.5582379Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:49.5582700Z res = mod(**inputs) 2025-12-04T08:57:49.5583079Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T08:57:49.5583486Z outputs = self.model.decoder( 2025-12-04T08:57:49.5583880Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T08:57:49.5584278Z layer_outputs = decoder_layer( 2025-12-04T08:57:49.5584628Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:57:49.5585004Z return super().__call__(*args, **kwargs) 2025-12-04T08:57:49.5585392Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5585770Z return func(*args, **kwargs) 2025-12-04T08:57:49.5586164Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 443, in forward 2025-12-04T08:57:49.5586582Z hidden_states = self.fc2(hidden_states) 2025-12-04T08:57:49.5586721Z 2025-12-04T08:57:49.5586837Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:49.5587200Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:49.5587560Z res = mod(**inputs) 2025-12-04T08:57:49.5587932Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T08:57:49.5588332Z outputs = self.model.decoder( 2025-12-04T08:57:49.5588715Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T08:57:49.5589115Z layer_outputs = decoder_layer( 2025-12-04T08:57:49.5589463Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:57:49.5589818Z return super().__call__(*args, **kwargs) 2025-12-04T08:57:49.5590199Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5590642Z return func(*args, **kwargs) 2025-12-04T08:57:49.5591045Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 445, in forward 2025-12-04T08:57:49.5591455Z hidden_states = residual + hidden_states 2025-12-04T08:57:49.5591599Z 2025-12-04T08:57:49.5591717Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:49.5592078Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:49.5592397Z res = mod(**inputs) 2025-12-04T08:57:49.5592769Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T08:57:49.5593182Z outputs = self.model.decoder( 2025-12-04T08:57:49.5593586Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T08:57:49.5594014Z layer_outputs = decoder_layer( 2025-12-04T08:57:49.5594375Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:57:49.5594754Z return super().__call__(*args, **kwargs) 2025-12-04T08:57:49.5595142Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5595534Z return func(*args, **kwargs) 2025-12-04T08:57:49.5595934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T08:57:49.5596371Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:57:49.5596807Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5597225Z return func(*args, **kwargs) 2025-12-04T08:57:49.5597651Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 216, in forward 2025-12-04T08:57:49.5598241Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T08:57:49.5598470Z 2025-12-04T08:57:49.5598590Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:49.5598989Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:49.5599354Z res = mod(**inputs) 2025-12-04T08:57:49.5599777Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T08:57:49.5600234Z outputs = self.model.decoder( 2025-12-04T08:57:49.5600663Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T08:57:49.5601081Z layer_outputs = decoder_layer( 2025-12-04T08:57:49.5601438Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:57:49.5601815Z return super().__call__(*args, **kwargs) 2025-12-04T08:57:49.5602228Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5602614Z return func(*args, **kwargs) 2025-12-04T08:57:49.5603000Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T08:57:49.5603434Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:57:49.5603843Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5604221Z return func(*args, **kwargs) 2025-12-04T08:57:49.5604616Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 236, in forward 2025-12-04T08:57:49.5605032Z key_states = self.k_proj(current_states) 2025-12-04T08:57:49.5605186Z 2025-12-04T08:57:49.5605318Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:49.5605683Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:49.5606017Z res = mod(**inputs) 2025-12-04T08:57:49.5606399Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T08:57:49.5606810Z outputs = self.model.decoder( 2025-12-04T08:57:49.5607206Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T08:57:49.5607602Z layer_outputs = decoder_layer( 2025-12-04T08:57:49.5607947Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:57:49.5608323Z return super().__call__(*args, **kwargs) 2025-12-04T08:57:49.5608709Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5609086Z return func(*args, **kwargs) 2025-12-04T08:57:49.5609474Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T08:57:49.5609893Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:57:49.5610299Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5610674Z return func(*args, **kwargs) 2025-12-04T08:57:49.5611054Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 237, in forward 2025-12-04T08:57:49.5611469Z value_states = self.v_proj(current_states) 2025-12-04T08:57:49.5611617Z 2025-12-04T08:57:49.5611702Z cudagraph partition due to non gpu ops 2025-12-04T08:57:49.5611946Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:49.5612303Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:49.5612630Z res = mod(**inputs) 2025-12-04T08:57:49.5613009Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T08:57:49.5613411Z outputs = self.model.decoder( 2025-12-04T08:57:49.5613806Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T08:57:49.5614211Z layer_outputs = decoder_layer( 2025-12-04T08:57:49.5614561Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:57:49.5614924Z return super().__call__(*args, **kwargs) 2025-12-04T08:57:49.5615309Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5615690Z return func(*args, **kwargs) 2025-12-04T08:57:49.5616084Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T08:57:49.5616524Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:57:49.5616944Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5617334Z return func(*args, **kwargs) 2025-12-04T08:57:49.5617725Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 255, in forward 2025-12-04T08:57:49.5618162Z attn_output, attn_weights = attention_interface( 2025-12-04T08:57:49.5618627Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T08:57:49.5619130Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T08:57:49.5619134Z 2025-12-04T08:57:49.5619284Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:49.5619492Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:49.5619568Z res = mod(**inputs) 2025-12-04T08:57:49.5619856Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T08:57:49.5619940Z outputs = self.model.decoder( 2025-12-04T08:57:49.5620208Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T08:57:49.5620280Z layer_outputs = decoder_layer( 2025-12-04T08:57:49.5620513Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:57:49.5620613Z return super().__call__(*args, **kwargs) 2025-12-04T08:57:49.5621043Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5621128Z return func(*args, **kwargs) 2025-12-04T08:57:49.5621399Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T08:57:49.5621505Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:57:49.5621746Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5621817Z return func(*args, **kwargs) 2025-12-04T08:57:49.5622091Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 269, in forward 2025-12-04T08:57:49.5622176Z attn_output = self.out_proj(attn_output) 2025-12-04T08:57:49.5622180Z 2025-12-04T08:57:49.5622292Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:49.5622498Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:49.5622566Z res = mod(**inputs) 2025-12-04T08:57:49.5622844Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T08:57:49.5622925Z outputs = self.model.decoder( 2025-12-04T08:57:49.5623198Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T08:57:49.5623278Z layer_outputs = decoder_layer( 2025-12-04T08:57:49.5623516Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:57:49.5623601Z return super().__call__(*args, **kwargs) 2025-12-04T08:57:49.5623845Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5623915Z return func(*args, **kwargs) 2025-12-04T08:57:49.5624190Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 441, in forward 2025-12-04T08:57:49.5624349Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:57:49.5624353Z 2025-12-04T08:57:49.5624464Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:49.5624663Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:49.5624728Z res = mod(**inputs) 2025-12-04T08:57:49.5625006Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T08:57:49.5625082Z outputs = self.model.decoder( 2025-12-04T08:57:49.5625361Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T08:57:49.5625443Z layer_outputs = decoder_layer( 2025-12-04T08:57:49.5625699Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:57:49.5625824Z return super().__call__(*args, **kwargs) 2025-12-04T08:57:49.5626073Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5626145Z return func(*args, **kwargs) 2025-12-04T08:57:49.5626423Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 441, in forward 2025-12-04T08:57:49.5626548Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:57:49.5626768Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T08:57:49.5626847Z return self.act(input) 2025-12-04T08:57:49.5626851Z 2025-12-04T08:57:49.5626955Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:49.5627194Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:49.5627273Z res = mod(**inputs) 2025-12-04T08:57:49.5627538Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T08:57:49.5627619Z outputs = self.model.decoder( 2025-12-04T08:57:49.5627886Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T08:57:49.5627968Z layer_outputs = decoder_layer( 2025-12-04T08:57:49.5628196Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:57:49.5628276Z return super().__call__(*args, **kwargs) 2025-12-04T08:57:49.5628534Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5628607Z return func(*args, **kwargs) 2025-12-04T08:57:49.5628880Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 443, in forward 2025-12-04T08:57:49.5628975Z hidden_states = self.fc2(hidden_states) 2025-12-04T08:57:49.5628979Z 2025-12-04T08:57:49.5629086Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:49.5629294Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:49.5629361Z res = mod(**inputs) 2025-12-04T08:57:49.5629633Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T08:57:49.5629716Z outputs = self.model.decoder( 2025-12-04T08:57:49.5629985Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T08:57:49.5630067Z layer_outputs = decoder_layer( 2025-12-04T08:57:49.5630298Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:57:49.5630380Z return super().__call__(*args, **kwargs) 2025-12-04T08:57:49.5630657Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5630732Z return func(*args, **kwargs) 2025-12-04T08:57:49.5631003Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T08:57:49.5631114Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:57:49.5631365Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5631445Z return func(*args, **kwargs) 2025-12-04T08:57:49.5631725Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 216, in forward 2025-12-04T08:57:49.5631931Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T08:57:49.5631953Z 2025-12-04T08:57:49.5632067Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:49.5632271Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:49.5632345Z res = mod(**inputs) 2025-12-04T08:57:49.5632618Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T08:57:49.5632691Z outputs = self.model.decoder( 2025-12-04T08:57:49.5632971Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T08:57:49.5633042Z layer_outputs = decoder_layer( 2025-12-04T08:57:49.5633268Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:57:49.5633375Z return super().__call__(*args, **kwargs) 2025-12-04T08:57:49.5633626Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5633705Z return func(*args, **kwargs) 2025-12-04T08:57:49.5633976Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T08:57:49.5634077Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:57:49.5634336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5634405Z return func(*args, **kwargs) 2025-12-04T08:57:49.5634676Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 236, in forward 2025-12-04T08:57:49.5634764Z key_states = self.k_proj(current_states) 2025-12-04T08:57:49.5634769Z 2025-12-04T08:57:49.5634872Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:49.5635091Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:49.5635162Z res = mod(**inputs) 2025-12-04T08:57:49.5635451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T08:57:49.5635537Z outputs = self.model.decoder( 2025-12-04T08:57:49.5635834Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T08:57:49.5635917Z layer_outputs = decoder_layer( 2025-12-04T08:57:49.5636159Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:57:49.5636243Z return super().__call__(*args, **kwargs) 2025-12-04T08:57:49.5636519Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5636595Z return func(*args, **kwargs) 2025-12-04T08:57:49.5636884Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T08:57:49.5637016Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:57:49.5637294Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5637374Z return func(*args, **kwargs) 2025-12-04T08:57:49.5637673Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 237, in forward 2025-12-04T08:57:49.5637766Z value_states = self.v_proj(current_states) 2025-12-04T08:57:49.5637770Z 2025-12-04T08:57:49.5637864Z cudagraph partition due to non gpu ops 2025-12-04T08:57:49.5637974Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:49.5638252Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:49.5638350Z res = mod(**inputs) 2025-12-04T08:57:49.5638657Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T08:57:49.5638745Z outputs = self.model.decoder( 2025-12-04T08:57:49.5639044Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T08:57:49.5639119Z layer_outputs = decoder_layer( 2025-12-04T08:57:49.5639364Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:57:49.5639448Z return super().__call__(*args, **kwargs) 2025-12-04T08:57:49.5639725Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5639799Z return func(*args, **kwargs) 2025-12-04T08:57:49.5640112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T08:57:49.5640232Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:57:49.5640496Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5640570Z return func(*args, **kwargs) 2025-12-04T08:57:49.5640869Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 255, in forward 2025-12-04T08:57:49.5640973Z attn_output, attn_weights = attention_interface( 2025-12-04T08:57:49.5641309Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T08:57:49.5641448Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T08:57:49.5641453Z 2025-12-04T08:57:49.5641558Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:49.5641773Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:49.5641840Z res = mod(**inputs) 2025-12-04T08:57:49.5642123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T08:57:49.5642199Z outputs = self.model.decoder( 2025-12-04T08:57:49.5642470Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T08:57:49.5642551Z layer_outputs = decoder_layer( 2025-12-04T08:57:49.5642780Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:57:49.5642858Z return super().__call__(*args, **kwargs) 2025-12-04T08:57:49.5643115Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5643187Z return func(*args, **kwargs) 2025-12-04T08:57:49.5643464Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T08:57:49.5643584Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:57:49.5643835Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5643914Z return func(*args, **kwargs) 2025-12-04T08:57:49.5644187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 269, in forward 2025-12-04T08:57:49.5644278Z attn_output = self.out_proj(attn_output) 2025-12-04T08:57:49.5644281Z 2025-12-04T08:57:49.5644386Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:49.5644589Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:49.5644662Z res = mod(**inputs) 2025-12-04T08:57:49.5644953Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T08:57:49.5645044Z outputs = self.model.decoder( 2025-12-04T08:57:49.5645328Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T08:57:49.5645403Z layer_outputs = decoder_layer( 2025-12-04T08:57:49.5645639Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:57:49.5645720Z return super().__call__(*args, **kwargs) 2025-12-04T08:57:49.5645969Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5646047Z return func(*args, **kwargs) 2025-12-04T08:57:49.5646316Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 441, in forward 2025-12-04T08:57:49.5646462Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:57:49.5646473Z 2025-12-04T08:57:49.5646590Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:49.5646790Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:49.5646863Z res = mod(**inputs) 2025-12-04T08:57:49.5647127Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T08:57:49.5647200Z outputs = self.model.decoder( 2025-12-04T08:57:49.5647472Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T08:57:49.5647543Z layer_outputs = decoder_layer( 2025-12-04T08:57:49.5647774Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:57:49.5647852Z return super().__call__(*args, **kwargs) 2025-12-04T08:57:49.5648103Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5648180Z return func(*args, **kwargs) 2025-12-04T08:57:49.5648445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 441, in forward 2025-12-04T08:57:49.5648564Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:57:49.5648804Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T08:57:49.5648876Z return self.act(input) 2025-12-04T08:57:49.5648879Z 2025-12-04T08:57:49.5648989Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:49.5649199Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:49.5649263Z res = mod(**inputs) 2025-12-04T08:57:49.5649539Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T08:57:49.5649613Z outputs = self.model.decoder( 2025-12-04T08:57:49.5649905Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T08:57:49.5649977Z layer_outputs = decoder_layer( 2025-12-04T08:57:49.5650199Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:57:49.5650285Z return super().__call__(*args, **kwargs) 2025-12-04T08:57:49.5650530Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5650599Z return func(*args, **kwargs) 2025-12-04T08:57:49.5650868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 443, in forward 2025-12-04T08:57:49.5650978Z hidden_states = self.fc2(hidden_states) 2025-12-04T08:57:49.5650996Z 2025-12-04T08:57:49.5651108Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:49.5651304Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:49.5651368Z res = mod(**inputs) 2025-12-04T08:57:49.5651642Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T08:57:49.5651714Z outputs = self.model.decoder( 2025-12-04T08:57:49.5651987Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T08:57:49.5652057Z layer_outputs = decoder_layer( 2025-12-04T08:57:49.5652276Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:57:49.5652380Z return super().__call__(*args, **kwargs) 2025-12-04T08:57:49.5652621Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5652692Z return func(*args, **kwargs) 2025-12-04T08:57:49.5652962Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 445, in forward 2025-12-04T08:57:49.5653042Z hidden_states = residual + hidden_states 2025-12-04T08:57:49.5653046Z 2025-12-04T08:57:49.5653155Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:49.5653352Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:49.5653416Z res = mod(**inputs) 2025-12-04T08:57:49.5653685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T08:57:49.5653758Z outputs = self.model.decoder( 2025-12-04T08:57:49.5654025Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T08:57:49.5654106Z layer_outputs = decoder_layer( 2025-12-04T08:57:49.5654334Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:57:49.5654420Z return super().__call__(*args, **kwargs) 2025-12-04T08:57:49.5654677Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5654746Z return func(*args, **kwargs) 2025-12-04T08:57:49.5655015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T08:57:49.5655111Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:57:49.5655356Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5655427Z return func(*args, **kwargs) 2025-12-04T08:57:49.5655693Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 216, in forward 2025-12-04T08:57:49.5655865Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T08:57:49.5655868Z 2025-12-04T08:57:49.5655971Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:49.5656166Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:49.5656237Z res = mod(**inputs) 2025-12-04T08:57:49.5656510Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T08:57:49.5656591Z outputs = self.model.decoder( 2025-12-04T08:57:49.5656867Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T08:57:49.5656941Z layer_outputs = decoder_layer( 2025-12-04T08:57:49.5657200Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:57:49.5657300Z return super().__call__(*args, **kwargs) 2025-12-04T08:57:49.5657555Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5657625Z return func(*args, **kwargs) 2025-12-04T08:57:49.5657896Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T08:57:49.5658001Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:57:49.5658249Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5658321Z return func(*args, **kwargs) 2025-12-04T08:57:49.5658624Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 236, in forward 2025-12-04T08:57:49.5658708Z key_states = self.k_proj(current_states) 2025-12-04T08:57:49.5658712Z 2025-12-04T08:57:49.5658824Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:49.5659026Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:49.5659092Z res = mod(**inputs) 2025-12-04T08:57:49.5659374Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T08:57:49.5659449Z outputs = self.model.decoder( 2025-12-04T08:57:49.5659727Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T08:57:49.5659800Z layer_outputs = decoder_layer( 2025-12-04T08:57:49.5660026Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:57:49.5660116Z return super().__call__(*args, **kwargs) 2025-12-04T08:57:49.5660368Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5660440Z return func(*args, **kwargs) 2025-12-04T08:57:49.5660717Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T08:57:49.5660817Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:57:49.5661073Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5661144Z return func(*args, **kwargs) 2025-12-04T08:57:49.5661417Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 237, in forward 2025-12-04T08:57:49.5661515Z value_states = self.v_proj(current_states) 2025-12-04T08:57:49.5661519Z 2025-12-04T08:57:49.5661602Z cudagraph partition due to non gpu ops 2025-12-04T08:57:49.5661708Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:49.5661937Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:49.5662006Z res = mod(**inputs) 2025-12-04T08:57:49.5662287Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T08:57:49.5662362Z outputs = self.model.decoder( 2025-12-04T08:57:49.5662647Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T08:57:49.5662730Z layer_outputs = decoder_layer( 2025-12-04T08:57:49.5662956Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:57:49.5663046Z return super().__call__(*args, **kwargs) 2025-12-04T08:57:49.5663315Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5663403Z return func(*args, **kwargs) 2025-12-04T08:57:49.5663685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T08:57:49.5663784Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:57:49.5664033Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5664112Z return func(*args, **kwargs) 2025-12-04T08:57:49.5664382Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 255, in forward 2025-12-04T08:57:49.5664487Z attn_output, attn_weights = attention_interface( 2025-12-04T08:57:49.5664795Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T08:57:49.5664953Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T08:57:49.5664957Z 2025-12-04T08:57:49.5665072Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:49.5665277Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:49.5665349Z res = mod(**inputs) 2025-12-04T08:57:49.5665629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T08:57:49.5665703Z outputs = self.model.decoder( 2025-12-04T08:57:49.5665988Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T08:57:49.5666061Z layer_outputs = decoder_layer( 2025-12-04T08:57:49.5666291Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:57:49.5666383Z return super().__call__(*args, **kwargs) 2025-12-04T08:57:49.5666642Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5666727Z return func(*args, **kwargs) 2025-12-04T08:57:49.5667019Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T08:57:49.5667124Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:57:49.5667399Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5667473Z return func(*args, **kwargs) 2025-12-04T08:57:49.5667765Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 269, in forward 2025-12-04T08:57:49.5667862Z attn_output = self.out_proj(attn_output) 2025-12-04T08:57:49.5667866Z 2025-12-04T08:57:49.5667977Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:49.5668200Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:49.5668287Z res = mod(**inputs) 2025-12-04T08:57:49.5668585Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T08:57:49.5668670Z outputs = self.model.decoder( 2025-12-04T08:57:49.5668943Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T08:57:49.5669027Z layer_outputs = decoder_layer( 2025-12-04T08:57:49.5669254Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:57:49.5669332Z return super().__call__(*args, **kwargs) 2025-12-04T08:57:49.5669590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5669694Z return func(*args, **kwargs) 2025-12-04T08:57:49.5669968Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 441, in forward 2025-12-04T08:57:49.5670095Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:57:49.5670100Z 2025-12-04T08:57:49.5670203Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:49.5670408Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:49.5670474Z res = mod(**inputs) 2025-12-04T08:57:49.5670744Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T08:57:49.5670827Z outputs = self.model.decoder( 2025-12-04T08:57:49.5671096Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T08:57:49.5671201Z layer_outputs = decoder_layer( 2025-12-04T08:57:49.5671431Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:57:49.5671510Z return super().__call__(*args, **kwargs) 2025-12-04T08:57:49.5671774Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5671847Z return func(*args, **kwargs) 2025-12-04T08:57:49.5672135Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 441, in forward 2025-12-04T08:57:49.5672267Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:57:49.5672496Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T08:57:49.5672580Z return self.act(input) 2025-12-04T08:57:49.5672584Z 2025-12-04T08:57:49.5672696Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:49.5672913Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:49.5672990Z res = mod(**inputs) 2025-12-04T08:57:49.5673289Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T08:57:49.5673375Z outputs = self.model.decoder( 2025-12-04T08:57:49.5673672Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T08:57:49.5673749Z layer_outputs = decoder_layer( 2025-12-04T08:57:49.5674003Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:57:49.5674087Z return super().__call__(*args, **kwargs) 2025-12-04T08:57:49.5674351Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5674436Z return func(*args, **kwargs) 2025-12-04T08:57:49.5674744Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 443, in forward 2025-12-04T08:57:49.5674841Z hidden_states = self.fc2(hidden_states) 2025-12-04T08:57:49.5674845Z 2025-12-04T08:57:49.5674954Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:49.5675167Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:49.5675246Z res = mod(**inputs) 2025-12-04T08:57:49.5675539Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T08:57:49.5675618Z outputs = self.model.decoder( 2025-12-04T08:57:49.5675933Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T08:57:49.5676012Z layer_outputs = decoder_layer( 2025-12-04T08:57:49.5676286Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:57:49.5676391Z return super().__call__(*args, **kwargs) 2025-12-04T08:57:49.5676671Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5676755Z return func(*args, **kwargs) 2025-12-04T08:57:49.5677042Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T08:57:49.5677158Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:57:49.5677429Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5677503Z return func(*args, **kwargs) 2025-12-04T08:57:49.5677821Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 216, in forward 2025-12-04T08:57:49.5677985Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T08:57:49.5677990Z 2025-12-04T08:57:49.5678170Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:49.5678407Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:49.5678480Z res = mod(**inputs) 2025-12-04T08:57:49.5678783Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T08:57:49.5678866Z outputs = self.model.decoder( 2025-12-04T08:57:49.5679173Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T08:57:49.5679260Z layer_outputs = decoder_layer( 2025-12-04T08:57:49.5679522Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:57:49.5679618Z return super().__call__(*args, **kwargs) 2025-12-04T08:57:49.5679901Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5679976Z return func(*args, **kwargs) 2025-12-04T08:57:49.5680274Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T08:57:49.5680379Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:57:49.5680651Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5680736Z return func(*args, **kwargs) 2025-12-04T08:57:49.5681025Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 236, in forward 2025-12-04T08:57:49.5681124Z key_states = self.k_proj(current_states) 2025-12-04T08:57:49.5681129Z 2025-12-04T08:57:49.5681243Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:49.5681480Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:49.5681561Z res = mod(**inputs) 2025-12-04T08:57:49.5681851Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T08:57:49.5681937Z outputs = self.model.decoder( 2025-12-04T08:57:49.5682243Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T08:57:49.5682320Z layer_outputs = decoder_layer( 2025-12-04T08:57:49.5682577Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:57:49.5682661Z return super().__call__(*args, **kwargs) 2025-12-04T08:57:49.5682943Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5683049Z return func(*args, **kwargs) 2025-12-04T08:57:49.5683339Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T08:57:49.5683452Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:57:49.5683717Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5683790Z return func(*args, **kwargs) 2025-12-04T08:57:49.5684083Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 237, in forward 2025-12-04T08:57:49.5684176Z value_states = self.v_proj(current_states) 2025-12-04T08:57:49.5684180Z 2025-12-04T08:57:49.5684269Z cudagraph partition due to non gpu ops 2025-12-04T08:57:49.5684406Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:49.5684622Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:49.5684701Z res = mod(**inputs) 2025-12-04T08:57:49.5684991Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T08:57:49.5685071Z outputs = self.model.decoder( 2025-12-04T08:57:49.5685366Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T08:57:49.5685443Z layer_outputs = decoder_layer( 2025-12-04T08:57:49.5685690Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:57:49.5685774Z return super().__call__(*args, **kwargs) 2025-12-04T08:57:49.5686046Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5686124Z return func(*args, **kwargs) 2025-12-04T08:57:49.5686404Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T08:57:49.5686505Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:57:49.5686763Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5686835Z return func(*args, **kwargs) 2025-12-04T08:57:49.5687123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 255, in forward 2025-12-04T08:57:49.5687228Z attn_output, attn_weights = attention_interface( 2025-12-04T08:57:49.5687546Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T08:57:49.5687699Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T08:57:49.5687703Z 2025-12-04T08:57:49.5687814Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:49.5688038Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:49.5688123Z res = mod(**inputs) 2025-12-04T08:57:49.5688414Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T08:57:49.5688500Z outputs = self.model.decoder( 2025-12-04T08:57:49.5688789Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T08:57:49.5688862Z layer_outputs = decoder_layer( 2025-12-04T08:57:49.5689099Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:57:49.5689179Z return super().__call__(*args, **kwargs) 2025-12-04T08:57:49.5689454Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5689543Z return func(*args, **kwargs) 2025-12-04T08:57:49.5689820Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T08:57:49.5689927Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:57:49.5690176Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5690245Z return func(*args, **kwargs) 2025-12-04T08:57:49.5690523Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 269, in forward 2025-12-04T08:57:49.5690607Z attn_output = self.out_proj(attn_output) 2025-12-04T08:57:49.5690611Z 2025-12-04T08:57:49.5690722Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:49.5690946Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:49.5691013Z res = mod(**inputs) 2025-12-04T08:57:49.5691303Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T08:57:49.5691378Z outputs = self.model.decoder( 2025-12-04T08:57:49.5691670Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T08:57:49.5691743Z layer_outputs = decoder_layer( 2025-12-04T08:57:49.5691969Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:57:49.5692057Z return super().__call__(*args, **kwargs) 2025-12-04T08:57:49.5692304Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5692376Z return func(*args, **kwargs) 2025-12-04T08:57:49.5692654Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 441, in forward 2025-12-04T08:57:49.5692776Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:57:49.5692781Z 2025-12-04T08:57:49.5692896Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:49.5693099Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:49.5693163Z res = mod(**inputs) 2025-12-04T08:57:49.5693443Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T08:57:49.5693517Z outputs = self.model.decoder( 2025-12-04T08:57:49.5693794Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T08:57:49.5693870Z layer_outputs = decoder_layer( 2025-12-04T08:57:49.5694099Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:57:49.5694189Z return super().__call__(*args, **kwargs) 2025-12-04T08:57:49.5694453Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5694526Z return func(*args, **kwargs) 2025-12-04T08:57:49.5694816Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 441, in forward 2025-12-04T08:57:49.5694942Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:57:49.5695182Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T08:57:49.5695256Z return self.act(input) 2025-12-04T08:57:49.5695260Z 2025-12-04T08:57:49.5695370Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:49.5695595Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:49.5695684Z res = mod(**inputs) 2025-12-04T08:57:49.5696000Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T08:57:49.5696082Z outputs = self.model.decoder( 2025-12-04T08:57:49.5696374Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T08:57:49.5696456Z layer_outputs = decoder_layer( 2025-12-04T08:57:49.5696704Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:57:49.5696784Z return super().__call__(*args, **kwargs) 2025-12-04T08:57:49.5697040Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5697139Z return func(*args, **kwargs) 2025-12-04T08:57:49.5697418Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 443, in forward 2025-12-04T08:57:49.5697505Z hidden_states = self.fc2(hidden_states) 2025-12-04T08:57:49.5697508Z 2025-12-04T08:57:49.5697614Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:49.5697824Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:49.5697888Z res = mod(**inputs) 2025-12-04T08:57:49.5698159Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T08:57:49.5698242Z outputs = self.model.decoder( 2025-12-04T08:57:49.5698512Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T08:57:49.5698593Z layer_outputs = decoder_layer( 2025-12-04T08:57:49.5698819Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:57:49.5698903Z return super().__call__(*args, **kwargs) 2025-12-04T08:57:49.5699160Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5699231Z return func(*args, **kwargs) 2025-12-04T08:57:49.5699507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 445, in forward 2025-12-04T08:57:49.5699590Z hidden_states = residual + hidden_states 2025-12-04T08:57:49.5699593Z 2025-12-04T08:57:49.5699699Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:49.5699909Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:49.5699975Z res = mod(**inputs) 2025-12-04T08:57:49.5700245Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T08:57:49.5700332Z outputs = self.model.decoder( 2025-12-04T08:57:49.5700604Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T08:57:49.5700701Z layer_outputs = decoder_layer( 2025-12-04T08:57:49.5700929Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:57:49.5701009Z return super().__call__(*args, **kwargs) 2025-12-04T08:57:49.5701267Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5701337Z return func(*args, **kwargs) 2025-12-04T08:57:49.5701605Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T08:57:49.5701717Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:57:49.5701983Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5702086Z return func(*args, **kwargs) 2025-12-04T08:57:49.5702359Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 216, in forward 2025-12-04T08:57:49.5702514Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-12-04T08:57:49.5702518Z 2025-12-04T08:57:49.5702631Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:49.5702836Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:49.5702908Z res = mod(**inputs) 2025-12-04T08:57:49.5703187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T08:57:49.5703262Z outputs = self.model.decoder( 2025-12-04T08:57:49.5703559Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T08:57:49.5703633Z layer_outputs = decoder_layer( 2025-12-04T08:57:49.5703870Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:57:49.5703955Z return super().__call__(*args, **kwargs) 2025-12-04T08:57:49.5704200Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5704276Z return func(*args, **kwargs) 2025-12-04T08:57:49.5704543Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T08:57:49.5704642Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:57:49.5704895Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5704967Z return func(*args, **kwargs) 2025-12-04T08:57:49.5705244Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 236, in forward 2025-12-04T08:57:49.5705327Z key_states = self.k_proj(current_states) 2025-12-04T08:57:49.5705331Z 2025-12-04T08:57:49.5705435Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:49.5705644Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:49.5705710Z res = mod(**inputs) 2025-12-04T08:57:49.5705980Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T08:57:49.5706061Z outputs = self.model.decoder( 2025-12-04T08:57:49.5706332Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T08:57:49.5706413Z layer_outputs = decoder_layer( 2025-12-04T08:57:49.5706641Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:57:49.5706721Z return super().__call__(*args, **kwargs) 2025-12-04T08:57:49.5706996Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5707068Z return func(*args, **kwargs) 2025-12-04T08:57:49.5707349Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T08:57:49.5707456Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:57:49.5707692Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5707765Z return func(*args, **kwargs) 2025-12-04T08:57:49.5708019Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 237, in forward 2025-12-04T08:57:49.5708119Z value_states = self.v_proj(current_states) 2025-12-04T08:57:49.5708139Z 2025-12-04T08:57:49.5708228Z cudagraph partition due to non gpu ops 2025-12-04T08:57:49.5708328Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:49.5708529Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:49.5708592Z res = mod(**inputs) 2025-12-04T08:57:49.5708854Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T08:57:49.5708934Z outputs = self.model.decoder( 2025-12-04T08:57:49.5709192Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T08:57:49.5709262Z layer_outputs = decoder_layer( 2025-12-04T08:57:49.5709487Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:57:49.5709585Z return super().__call__(*args, **kwargs) 2025-12-04T08:57:49.5709831Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5709901Z return func(*args, **kwargs) 2025-12-04T08:57:49.5710162Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T08:57:49.5710266Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:57:49.5710506Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5710582Z return func(*args, **kwargs) 2025-12-04T08:57:49.5710843Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 255, in forward 2025-12-04T08:57:49.5710941Z attn_output, attn_weights = attention_interface( 2025-12-04T08:57:49.5711242Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward 2025-12-04T08:57:49.5711385Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T08:57:49.5711388Z 2025-12-04T08:57:49.5711497Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:49.5711693Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:49.5711757Z res = mod(**inputs) 2025-12-04T08:57:49.5712040Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T08:57:49.5712115Z outputs = self.model.decoder( 2025-12-04T08:57:49.5712382Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T08:57:49.5712464Z layer_outputs = decoder_layer( 2025-12-04T08:57:49.5712692Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:57:49.5712780Z return super().__call__(*args, **kwargs) 2025-12-04T08:57:49.5713046Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5713119Z return func(*args, **kwargs) 2025-12-04T08:57:49.5713397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 410, in forward 2025-12-04T08:57:49.5713495Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:57:49.5713745Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5713823Z return func(*args, **kwargs) 2025-12-04T08:57:49.5714091Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 269, in forward 2025-12-04T08:57:49.5714201Z attn_output = self.out_proj(attn_output) 2025-12-04T08:57:49.5714222Z 2025-12-04T08:57:49.5714328Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:49.5714533Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:49.5714608Z res = mod(**inputs) 2025-12-04T08:57:49.5714888Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T08:57:49.5714970Z outputs = self.model.decoder( 2025-12-04T08:57:49.5715290Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T08:57:49.5715363Z layer_outputs = decoder_layer( 2025-12-04T08:57:49.5715599Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:57:49.5715701Z return super().__call__(*args, **kwargs) 2025-12-04T08:57:49.5715951Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5716029Z return func(*args, **kwargs) 2025-12-04T08:57:49.5716300Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 441, in forward 2025-12-04T08:57:49.5716427Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:57:49.5716430Z 2025-12-04T08:57:49.5716534Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:49.5716738Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:49.5716811Z res = mod(**inputs) 2025-12-04T08:57:49.5717085Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T08:57:49.5717161Z outputs = self.model.decoder( 2025-12-04T08:57:49.5717442Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T08:57:49.5717515Z layer_outputs = decoder_layer( 2025-12-04T08:57:49.5717749Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:57:49.5717827Z return super().__call__(*args, **kwargs) 2025-12-04T08:57:49.5718078Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5718230Z return func(*args, **kwargs) 2025-12-04T08:57:49.5718508Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 441, in forward 2025-12-04T08:57:49.5718647Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:57:49.5718893Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T08:57:49.5718975Z return self.act(input) 2025-12-04T08:57:49.5718984Z 2025-12-04T08:57:49.5719108Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:49.5719367Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:49.5719439Z res = mod(**inputs) 2025-12-04T08:57:49.5719741Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1633, in forward 2025-12-04T08:57:49.5719821Z outputs = self.model.decoder( 2025-12-04T08:57:49.5720118Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1120, in forward 2025-12-04T08:57:49.5720194Z layer_outputs = decoder_layer( 2025-12-04T08:57:49.5720435Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:57:49.5720530Z return super().__call__(*args, **kwargs) 2025-12-04T08:57:49.5721128Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:57:49.5721262Z return func(*args, **kwargs) 2025-12-04T08:57:49.5721543Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 443, in forward 2025-12-04T08:57:49.5721624Z hidden_states = self.fc2(hidden_states) 2025-12-04T08:57:49.5721628Z 2025-12-04T08:57:49.5721741Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:49.5721941Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:49.5722007Z res = mod(**inputs) 2025-12-04T08:57:49.5722280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1649, in forward 2025-12-04T08:57:49.5722360Z logits = self.lm_head(outputs[0]) 2025-12-04T08:57:49.5722394Z 2025-12-04T08:57:49.5722505Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:57:49.5722706Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:57:49.5722770Z res = mod(**inputs) 2025-12-04T08:57:49.5723059Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1655, in forward 2025-12-04T08:57:49.5723211Z loss = loss_fct(logits.view(-1, self.config.vocab_size), labels.view(-1)) 2025-12-04T08:57:49.5723215Z 2025-12-04T08:58:00.0881938Z Compilation time (from dynamo_timed): 17.052186199 2025-12-04T08:58:00.0907697Z pass 2025-12-04T08:58:00.0908201Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T08:58:00.0909140Z TIMING: _recursive_pre_grad_passes:0.00708 _recursive_joint_graph_passes:0.67587 _recursive_post_grad_passes:0.06807 async_compile.wait:0.82686 code_gen:10.12242 inductor_compile:11.46455 backend_compile:14.63021 gc:0.001 entire_frame_compile:17.05219 total_wall_time:17.05219 2025-12-04T08:58:00.0910477Z STATS: call_* op count: 333 | FakeTensorMode.__torch_dispatch__:7342 | FakeTensor.__torch_dispatch__:4382 | ProxyTorchDispatchMode.__torch_dispatch__:2031 2025-12-04T08:58:00.0911126Z Dynamo produced 1 graphs covering 333 ops with 0 graph breaks (0 unique) 2025-12-04T08:58:02.6090674Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/__init__.py:65: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-12-04T08:58:02.6091676Z import pynvml # type: ignore[import] 2025-12-04T08:58:06.0818290Z 2025-12-04T08:58:06.0975047Z loading model: 0it [00:00, ?it/s]If you want to use `RobertaLMHeadModel` as a standalone, add `is_decoder=True.` 2025-12-04T08:58:06.0975718Z WARNING:transformers.models.roberta.modeling_roberta:If you want to use `RobertaLMHeadModel` as a standalone, add `is_decoder=True.` 2025-12-04T08:58:07.2297348Z We strongly recommend passing in an `attention_mask` since your input_ids may be padded. See https://huggingface.co/docs/transformers/troubleshooting#incorrect-output-when-padding-tokens-arent-masked. 2025-12-04T08:58:07.2300259Z You may ignore this warning if your `pad_token_id` (0) is identical to the `bos_token_id` (0), `eos_token_id` (2), or the `sep_token_id` (None), and your input is not padded. 2025-12-04T08:58:07.2301371Z WARNING:transformers.modeling_utils:We strongly recommend passing in an `attention_mask` since your input_ids may be padded. See https://huggingface.co/docs/transformers/troubleshooting#incorrect-output-when-padding-tokens-arent-masked. 2025-12-04T08:58:07.2302688Z You may ignore this warning if your `pad_token_id` (0) is identical to the `bos_token_id` (0), `eos_token_id` (2), or the `sep_token_id` (None), and your input is not padded. 2025-12-04T08:58:07.6265564Z 2025-12-04T08:58:07.6266384Z loading model: 0it [00:01, ?it/s] 2025-12-04T08:58:07.6277351Z cpu eval RobertaForCausalLM 2025-12-04T08:58:09.2097626Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T08:58:09.9337437Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T08:58:10.6622107Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T08:58:18.1832797Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:18.1833367Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:18.1834062Z res = mod(**inputs) 2025-12-04T08:58:18.1834611Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T08:58:18.1835172Z outputs = self.roberta( 2025-12-04T08:58:18.1836035Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 798, in forward 2025-12-04T08:58:18.1836973Z embedding_output = self.embeddings( 2025-12-04T08:58:18.1837682Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 87, in forward 2025-12-04T08:58:18.1838608Z position_ids = create_position_ids_from_input_ids(input_ids, self.padding_idx, past_key_values_length) 2025-12-04T08:58:18.1839374Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 1548, in create_position_ids_from_input_ids 2025-12-04T08:58:18.1839984Z mask = input_ids.ne(padding_idx).int() 2025-12-04T08:58:18.1840185Z 2025-12-04T08:58:18.1840349Z cudagraph partition due to non gpu ops 2025-12-04T08:58:18.1840695Z cudagraph partition due to non gpu ops 2025-12-04T08:58:18.1840964Z cudagraph partition due to non gpu ops 2025-12-04T08:58:18.1841252Z cudagraph partition due to non gpu ops 2025-12-04T08:58:18.1841544Z cudagraph partition due to non gpu ops 2025-12-04T08:58:18.1841802Z cudagraph partition due to non gpu ops 2025-12-04T08:58:18.1842094Z cudagraph partition due to non gpu ops 2025-12-04T08:58:18.1842382Z cudagraph partition due to non gpu ops 2025-12-04T08:58:18.1842647Z cudagraph partition due to non gpu ops 2025-12-04T08:58:18.1842936Z cudagraph partition due to non gpu ops 2025-12-04T08:58:18.1843223Z cudagraph partition due to non gpu ops 2025-12-04T08:58:18.1843541Z cudagraph partition due to non gpu ops 2025-12-04T08:58:18.1843950Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:18.1844382Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:18.1844820Z res = mod(**inputs) 2025-12-04T08:58:18.1845269Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T08:58:18.1845768Z outputs = self.roberta( 2025-12-04T08:58:18.1846287Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 798, in forward 2025-12-04T08:58:18.1846790Z embedding_output = self.embeddings( 2025-12-04T08:58:18.1847318Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 87, in forward 2025-12-04T08:58:18.1847974Z position_ids = create_position_ids_from_input_ids(input_ids, self.padding_idx, past_key_values_length) 2025-12-04T08:58:18.1848670Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 1549, in create_position_ids_from_input_ids 2025-12-04T08:58:18.1849390Z incremental_indices = (torch.cumsum(mask, dim=1).type_as(mask) + past_key_values_length) * mask 2025-12-04T08:58:18.1849694Z 2025-12-04T08:58:18.1849843Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:18.1850498Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:18.1851063Z res = mod(**inputs) 2025-12-04T08:58:18.1851602Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T08:58:18.1852065Z outputs = self.roberta( 2025-12-04T08:58:18.1852598Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 798, in forward 2025-12-04T08:58:18.1853105Z embedding_output = self.embeddings( 2025-12-04T08:58:18.1853636Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 87, in forward 2025-12-04T08:58:18.1854275Z position_ids = create_position_ids_from_input_ids(input_ids, self.padding_idx, past_key_values_length) 2025-12-04T08:58:18.1854993Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 1549, in create_position_ids_from_input_ids 2025-12-04T08:58:18.1855748Z incremental_indices = (torch.cumsum(mask, dim=1).type_as(mask) + past_key_values_length) * mask 2025-12-04T08:58:18.1856077Z 2025-12-04T08:58:18.1856280Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:18.1856868Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:18.1857370Z res = mod(**inputs) 2025-12-04T08:58:18.1857867Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T08:58:18.1858372Z outputs = self.roberta( 2025-12-04T08:58:18.1858860Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T08:58:18.1859392Z encoder_outputs = self.encoder( 2025-12-04T08:58:18.1859920Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T08:58:18.1860427Z layer_outputs = layer_module( 2025-12-04T08:58:18.1860869Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:18.1861407Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:18.1861930Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.1862460Z return func(*args, **kwargs) 2025-12-04T08:58:18.1862959Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T08:58:18.1863512Z self_attention_outputs = self.attention( 2025-12-04T08:58:18.1864022Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.1864480Z return func(*args, **kwargs) 2025-12-04T08:58:18.1865145Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 440, in forward 2025-12-04T08:58:18.1865677Z self_outputs = self.self( 2025-12-04T08:58:18.1866210Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.1866658Z return func(*args, **kwargs) 2025-12-04T08:58:18.1867182Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 313, in forward 2025-12-04T08:58:18.1867871Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-12-04T08:58:18.1868188Z 2025-12-04T08:58:18.1868354Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:18.1868794Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:18.1869303Z res = mod(**inputs) 2025-12-04T08:58:18.1869865Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T08:58:18.1870387Z outputs = self.roberta( 2025-12-04T08:58:18.1870817Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T08:58:18.1871349Z encoder_outputs = self.encoder( 2025-12-04T08:58:18.1871826Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T08:58:18.1872262Z layer_outputs = layer_module( 2025-12-04T08:58:18.1872818Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:18.1873278Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:18.1873817Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.1874295Z return func(*args, **kwargs) 2025-12-04T08:58:18.1874816Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T08:58:18.1875337Z self_attention_outputs = self.attention( 2025-12-04T08:58:18.1875889Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.1876359Z return func(*args, **kwargs) 2025-12-04T08:58:18.1876873Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 440, in forward 2025-12-04T08:58:18.1877451Z self_outputs = self.self( 2025-12-04T08:58:18.1877944Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.1890108Z return func(*args, **kwargs) 2025-12-04T08:58:18.1890820Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 337, in forward 2025-12-04T08:58:18.1891306Z self.key(current_states) 2025-12-04T08:58:18.1891450Z 2025-12-04T08:58:18.1891590Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:18.1892008Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:18.1892382Z res = mod(**inputs) 2025-12-04T08:58:18.1892811Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T08:58:18.1893257Z outputs = self.roberta( 2025-12-04T08:58:18.1893677Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T08:58:18.1894124Z encoder_outputs = self.encoder( 2025-12-04T08:58:18.1894563Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T08:58:18.1894997Z layer_outputs = layer_module( 2025-12-04T08:58:18.1895397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:18.1895898Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:18.1896344Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.1896764Z return func(*args, **kwargs) 2025-12-04T08:58:18.1897198Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T08:58:18.1897650Z self_attention_outputs = self.attention( 2025-12-04T08:58:18.1898083Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.1898493Z return func(*args, **kwargs) 2025-12-04T08:58:18.1898927Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 440, in forward 2025-12-04T08:58:18.1899438Z self_outputs = self.self( 2025-12-04T08:58:18.1899836Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.1900249Z return func(*args, **kwargs) 2025-12-04T08:58:18.1900678Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 342, in forward 2025-12-04T08:58:18.1901119Z self.value(current_states) 2025-12-04T08:58:18.1901254Z 2025-12-04T08:58:18.1901350Z cudagraph partition due to non gpu ops 2025-12-04T08:58:18.1901633Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:18.1902052Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:18.1902409Z res = mod(**inputs) 2025-12-04T08:58:18.1902861Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T08:58:18.1903295Z outputs = self.roberta( 2025-12-04T08:58:18.1903717Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T08:58:18.1904210Z encoder_outputs = self.encoder( 2025-12-04T08:58:18.1904614Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T08:58:18.1905023Z layer_outputs = layer_module( 2025-12-04T08:58:18.1905381Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:18.1905759Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:18.1906171Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.1906583Z return func(*args, **kwargs) 2025-12-04T08:58:18.1906993Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T08:58:18.1907440Z self_attention_outputs = self.attention( 2025-12-04T08:58:18.1907862Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.1908269Z return func(*args, **kwargs) 2025-12-04T08:58:18.1908680Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 440, in forward 2025-12-04T08:58:18.1909085Z self_outputs = self.self( 2025-12-04T08:58:18.1909459Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.1909968Z return func(*args, **kwargs) 2025-12-04T08:58:18.1910403Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 363, in forward 2025-12-04T08:58:18.1910910Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T08:58:18.1911114Z 2025-12-04T08:58:18.1911241Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:18.1911659Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:18.1912022Z res = mod(**inputs) 2025-12-04T08:58:18.1912435Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T08:58:18.1912862Z outputs = self.roberta( 2025-12-04T08:58:18.1913280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T08:58:18.1913734Z encoder_outputs = self.encoder( 2025-12-04T08:58:18.1914161Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T08:58:18.1914588Z layer_outputs = layer_module( 2025-12-04T08:58:18.1914995Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:18.1915421Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:18.1915845Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.1916250Z return func(*args, **kwargs) 2025-12-04T08:58:18.1916666Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T08:58:18.1917110Z self_attention_outputs = self.attention( 2025-12-04T08:58:18.1917525Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.1917929Z return func(*args, **kwargs) 2025-12-04T08:58:18.1918415Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 449, in forward 2025-12-04T08:58:18.1918953Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T08:58:18.1919630Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 387, in forward 2025-12-04T08:58:18.1920100Z hidden_states = self.dense(hidden_states) 2025-12-04T08:58:18.1920261Z 2025-12-04T08:58:18.1920391Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:18.1921052Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:18.1921411Z res = mod(**inputs) 2025-12-04T08:58:18.1921822Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T08:58:18.1922255Z outputs = self.roberta( 2025-12-04T08:58:18.1922662Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T08:58:18.1923107Z encoder_outputs = self.encoder( 2025-12-04T08:58:18.1923538Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T08:58:18.1923977Z layer_outputs = layer_module( 2025-12-04T08:58:18.1924356Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:18.1924757Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:18.1925177Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.1925581Z return func(*args, **kwargs) 2025-12-04T08:58:18.1925998Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 543, in forward 2025-12-04T08:58:18.1926448Z layer_output = apply_chunking_to_forward( 2025-12-04T08:58:18.1926894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:58:18.1927327Z return forward_fn(*input_tensors) 2025-12-04T08:58:18.1927906Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 551, in feed_forward_chunk 2025-12-04T08:58:18.1928437Z intermediate_output = self.intermediate(attention_output) 2025-12-04T08:58:18.1928920Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 465, in forward 2025-12-04T08:58:18.1929374Z hidden_states = self.dense(hidden_states) 2025-12-04T08:58:18.1929530Z 2025-12-04T08:58:18.1929656Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:18.1930064Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:18.1930421Z res = mod(**inputs) 2025-12-04T08:58:18.1930874Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T08:58:18.1931332Z outputs = self.roberta( 2025-12-04T08:58:18.1931746Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T08:58:18.1932185Z encoder_outputs = self.encoder( 2025-12-04T08:58:18.1932617Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T08:58:18.1933047Z layer_outputs = layer_module( 2025-12-04T08:58:18.1933419Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:18.1933824Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:18.1934245Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.1934675Z return func(*args, **kwargs) 2025-12-04T08:58:18.1935094Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 543, in forward 2025-12-04T08:58:18.1935542Z layer_output = apply_chunking_to_forward( 2025-12-04T08:58:18.1935979Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:58:18.1936401Z return forward_fn(*input_tensors) 2025-12-04T08:58:18.1936865Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 551, in feed_forward_chunk 2025-12-04T08:58:18.1937374Z intermediate_output = self.intermediate(attention_output) 2025-12-04T08:58:18.1937830Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-12-04T08:58:18.1938269Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:58:18.1938669Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T08:58:18.1939031Z return self.act(input) 2025-12-04T08:58:18.1939146Z 2025-12-04T08:58:18.1939257Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:18.1939629Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:18.1939958Z res = mod(**inputs) 2025-12-04T08:58:18.1940335Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T08:58:18.1940729Z outputs = self.roberta( 2025-12-04T08:58:18.1941112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T08:58:18.1941517Z encoder_outputs = self.encoder( 2025-12-04T08:58:18.1941920Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T08:58:18.1942320Z layer_outputs = layer_module( 2025-12-04T08:58:18.1942697Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:18.1943072Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:18.1943463Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.1943856Z return func(*args, **kwargs) 2025-12-04T08:58:18.1944259Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 543, in forward 2025-12-04T08:58:18.1944686Z layer_output = apply_chunking_to_forward( 2025-12-04T08:58:18.1945103Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:58:18.1945520Z return forward_fn(*input_tensors) 2025-12-04T08:58:18.1945983Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 552, in feed_forward_chunk 2025-12-04T08:58:18.1946500Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T08:58:18.1946962Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 479, in forward 2025-12-04T08:58:18.1947383Z hidden_states = self.dense(hidden_states) 2025-12-04T08:58:18.1947522Z 2025-12-04T08:58:18.1947635Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:18.1948002Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:18.1948334Z res = mod(**inputs) 2025-12-04T08:58:18.1948716Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T08:58:18.1949144Z outputs = self.roberta( 2025-12-04T08:58:18.1949516Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T08:58:18.1949911Z encoder_outputs = self.encoder( 2025-12-04T08:58:18.1950302Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T08:58:18.1950689Z layer_outputs = layer_module( 2025-12-04T08:58:18.1951037Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:18.1951402Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:18.1951783Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.1952152Z return func(*args, **kwargs) 2025-12-04T08:58:18.1952535Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T08:58:18.1952945Z self_attention_outputs = self.attention( 2025-12-04T08:58:18.1953334Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.1953707Z return func(*args, **kwargs) 2025-12-04T08:58:18.1954093Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 440, in forward 2025-12-04T08:58:18.1954491Z self_outputs = self.self( 2025-12-04T08:58:18.1954850Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.1955225Z return func(*args, **kwargs) 2025-12-04T08:58:18.1955611Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 313, in forward 2025-12-04T08:58:18.1956151Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-12-04T08:58:18.1956419Z 2025-12-04T08:58:18.1956526Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:18.1956892Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:18.1957235Z res = mod(**inputs) 2025-12-04T08:58:18.1957612Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T08:58:18.1957999Z outputs = self.roberta( 2025-12-04T08:58:18.1958537Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T08:58:18.1958999Z encoder_outputs = self.encoder( 2025-12-04T08:58:18.1959438Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T08:58:18.1959873Z layer_outputs = layer_module( 2025-12-04T08:58:18.1960283Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:18.1960672Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:18.1961053Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.1961430Z return func(*args, **kwargs) 2025-12-04T08:58:18.1961827Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T08:58:18.1962244Z self_attention_outputs = self.attention( 2025-12-04T08:58:18.1962636Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.1963010Z return func(*args, **kwargs) 2025-12-04T08:58:18.1963395Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 440, in forward 2025-12-04T08:58:18.1963808Z self_outputs = self.self( 2025-12-04T08:58:18.1964229Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.1964616Z return func(*args, **kwargs) 2025-12-04T08:58:18.1965018Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 337, in forward 2025-12-04T08:58:18.1965415Z self.key(current_states) 2025-12-04T08:58:18.1965536Z 2025-12-04T08:58:18.1965643Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:18.1966008Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:18.1966334Z res = mod(**inputs) 2025-12-04T08:58:18.1966702Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T08:58:18.1967100Z outputs = self.roberta( 2025-12-04T08:58:18.1967484Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T08:58:18.1967885Z encoder_outputs = self.encoder( 2025-12-04T08:58:18.1968288Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T08:58:18.1968690Z layer_outputs = layer_module( 2025-12-04T08:58:18.1969044Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:18.1969409Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:18.1969798Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.1970181Z return func(*args, **kwargs) 2025-12-04T08:58:18.1970569Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T08:58:18.1970986Z self_attention_outputs = self.attention( 2025-12-04T08:58:18.1971369Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.1971737Z return func(*args, **kwargs) 2025-12-04T08:58:18.1972129Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 440, in forward 2025-12-04T08:58:18.1972520Z self_outputs = self.self( 2025-12-04T08:58:18.1972873Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.1973239Z return func(*args, **kwargs) 2025-12-04T08:58:18.1973606Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 342, in forward 2025-12-04T08:58:18.1973991Z self.value(current_states) 2025-12-04T08:58:18.1974108Z 2025-12-04T08:58:18.1974198Z cudagraph partition due to non gpu ops 2025-12-04T08:58:18.1974426Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:18.1974802Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:18.1975143Z res = mod(**inputs) 2025-12-04T08:58:18.1975507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T08:58:18.1975882Z outputs = self.roberta( 2025-12-04T08:58:18.1976247Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T08:58:18.1976636Z encoder_outputs = self.encoder( 2025-12-04T08:58:18.1977006Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T08:58:18.1977393Z layer_outputs = layer_module( 2025-12-04T08:58:18.1977732Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:18.1978122Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:18.1978494Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.1978867Z return func(*args, **kwargs) 2025-12-04T08:58:18.1979255Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T08:58:18.1979670Z self_attention_outputs = self.attention( 2025-12-04T08:58:18.1980050Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.1980434Z return func(*args, **kwargs) 2025-12-04T08:58:18.1980812Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 440, in forward 2025-12-04T08:58:18.1981195Z self_outputs = self.self( 2025-12-04T08:58:18.1981550Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.1981916Z return func(*args, **kwargs) 2025-12-04T08:58:18.1982298Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 363, in forward 2025-12-04T08:58:18.1982737Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T08:58:18.1982924Z 2025-12-04T08:58:18.1983026Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:18.1983381Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:18.1983689Z res = mod(**inputs) 2025-12-04T08:58:18.1984056Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T08:58:18.1984457Z outputs = self.roberta( 2025-12-04T08:58:18.1984832Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T08:58:18.1985214Z encoder_outputs = self.encoder( 2025-12-04T08:58:18.1985659Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T08:58:18.1986074Z layer_outputs = layer_module( 2025-12-04T08:58:18.1986425Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:18.1986783Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:18.1987169Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.1987535Z return func(*args, **kwargs) 2025-12-04T08:58:18.1987900Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T08:58:18.1988309Z self_attention_outputs = self.attention( 2025-12-04T08:58:18.1988711Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.1989101Z return func(*args, **kwargs) 2025-12-04T08:58:18.1989478Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 449, in forward 2025-12-04T08:58:18.1989928Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T08:58:18.1990375Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 387, in forward 2025-12-04T08:58:18.1990785Z hidden_states = self.dense(hidden_states) 2025-12-04T08:58:18.1990926Z 2025-12-04T08:58:18.1991334Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:18.1991705Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:18.1992062Z res = mod(**inputs) 2025-12-04T08:58:18.1992447Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T08:58:18.1992845Z outputs = self.roberta( 2025-12-04T08:58:18.1993226Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T08:58:18.1993625Z encoder_outputs = self.encoder( 2025-12-04T08:58:18.1994013Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T08:58:18.1994420Z layer_outputs = layer_module( 2025-12-04T08:58:18.1994777Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:18.1995148Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:18.1995537Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.1995926Z return func(*args, **kwargs) 2025-12-04T08:58:18.1996327Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 543, in forward 2025-12-04T08:58:18.1996764Z layer_output = apply_chunking_to_forward( 2025-12-04T08:58:18.1997208Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:58:18.1997641Z return forward_fn(*input_tensors) 2025-12-04T08:58:18.1998188Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 551, in feed_forward_chunk 2025-12-04T08:58:18.1998728Z intermediate_output = self.intermediate(attention_output) 2025-12-04T08:58:18.1999229Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 465, in forward 2025-12-04T08:58:18.1999688Z hidden_states = self.dense(hidden_states) 2025-12-04T08:58:18.1999838Z 2025-12-04T08:58:18.1999963Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:18.2000355Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:18.2000721Z res = mod(**inputs) 2025-12-04T08:58:18.2001113Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T08:58:18.2001512Z outputs = self.roberta( 2025-12-04T08:58:18.2001903Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T08:58:18.2002321Z encoder_outputs = self.encoder( 2025-12-04T08:58:18.2002745Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T08:58:18.2003170Z layer_outputs = layer_module( 2025-12-04T08:58:18.2003570Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:18.2003988Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:18.2004400Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2004807Z return func(*args, **kwargs) 2025-12-04T08:58:18.2005273Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 543, in forward 2025-12-04T08:58:18.2005711Z layer_output = apply_chunking_to_forward( 2025-12-04T08:58:18.2006140Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:58:18.2006571Z return forward_fn(*input_tensors) 2025-12-04T08:58:18.2007042Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 551, in feed_forward_chunk 2025-12-04T08:58:18.2007577Z intermediate_output = self.intermediate(attention_output) 2025-12-04T08:58:18.2008061Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-12-04T08:58:18.2008538Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:58:18.2008963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T08:58:18.2009340Z return self.act(input) 2025-12-04T08:58:18.2009471Z 2025-12-04T08:58:18.2009586Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:18.2009988Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:18.2010345Z res = mod(**inputs) 2025-12-04T08:58:18.2010745Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T08:58:18.2011179Z outputs = self.roberta( 2025-12-04T08:58:18.2011599Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T08:58:18.2012029Z encoder_outputs = self.encoder( 2025-12-04T08:58:18.2012465Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T08:58:18.2012897Z layer_outputs = layer_module( 2025-12-04T08:58:18.2013281Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:18.2013677Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:18.2014099Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2014492Z return func(*args, **kwargs) 2025-12-04T08:58:18.2014892Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 543, in forward 2025-12-04T08:58:18.2015325Z layer_output = apply_chunking_to_forward( 2025-12-04T08:58:18.2015793Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:58:18.2016227Z return forward_fn(*input_tensors) 2025-12-04T08:58:18.2016686Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 552, in feed_forward_chunk 2025-12-04T08:58:18.2017220Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T08:58:18.2017690Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 479, in forward 2025-12-04T08:58:18.2018117Z hidden_states = self.dense(hidden_states) 2025-12-04T08:58:18.2018260Z 2025-12-04T08:58:18.2018369Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:18.2018746Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:18.2019120Z res = mod(**inputs) 2025-12-04T08:58:18.2019519Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T08:58:18.2019921Z outputs = self.roberta( 2025-12-04T08:58:18.2020345Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T08:58:18.2020961Z encoder_outputs = self.encoder( 2025-12-04T08:58:18.2021384Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T08:58:18.2021814Z layer_outputs = layer_module( 2025-12-04T08:58:18.2022183Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:18.2022552Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:18.2022991Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2023378Z return func(*args, **kwargs) 2025-12-04T08:58:18.2023780Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T08:58:18.2024189Z self_attention_outputs = self.attention( 2025-12-04T08:58:18.2024585Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2024971Z return func(*args, **kwargs) 2025-12-04T08:58:18.2025364Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 440, in forward 2025-12-04T08:58:18.2025764Z self_outputs = self.self( 2025-12-04T08:58:18.2026141Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2026524Z return func(*args, **kwargs) 2025-12-04T08:58:18.2026917Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 313, in forward 2025-12-04T08:58:18.2027463Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-12-04T08:58:18.2027739Z 2025-12-04T08:58:18.2027844Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:18.2028212Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:18.2028538Z res = mod(**inputs) 2025-12-04T08:58:18.2028918Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T08:58:18.2029319Z outputs = self.roberta( 2025-12-04T08:58:18.2029706Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T08:58:18.2030103Z encoder_outputs = self.encoder( 2025-12-04T08:58:18.2030519Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T08:58:18.2030965Z layer_outputs = layer_module( 2025-12-04T08:58:18.2031328Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:18.2031701Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:18.2032100Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2032493Z return func(*args, **kwargs) 2025-12-04T08:58:18.2032892Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T08:58:18.2033322Z self_attention_outputs = self.attention( 2025-12-04T08:58:18.2033729Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2034178Z return func(*args, **kwargs) 2025-12-04T08:58:18.2034569Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 440, in forward 2025-12-04T08:58:18.2034976Z self_outputs = self.self( 2025-12-04T08:58:18.2035355Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2035753Z return func(*args, **kwargs) 2025-12-04T08:58:18.2036171Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 337, in forward 2025-12-04T08:58:18.2036599Z self.key(current_states) 2025-12-04T08:58:18.2036723Z 2025-12-04T08:58:18.2036845Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:18.2037232Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:18.2037610Z res = mod(**inputs) 2025-12-04T08:58:18.2038017Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T08:58:18.2038524Z outputs = self.roberta( 2025-12-04T08:58:18.2038969Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T08:58:18.2039428Z encoder_outputs = self.encoder( 2025-12-04T08:58:18.2039879Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T08:58:18.2040307Z layer_outputs = layer_module( 2025-12-04T08:58:18.2040691Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:18.2041092Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:18.2041518Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2041927Z return func(*args, **kwargs) 2025-12-04T08:58:18.2042349Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T08:58:18.2042792Z self_attention_outputs = self.attention( 2025-12-04T08:58:18.2043206Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2043611Z return func(*args, **kwargs) 2025-12-04T08:58:18.2044027Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 440, in forward 2025-12-04T08:58:18.2044458Z self_outputs = self.self( 2025-12-04T08:58:18.2044850Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2045261Z return func(*args, **kwargs) 2025-12-04T08:58:18.2045681Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 342, in forward 2025-12-04T08:58:18.2046113Z self.value(current_states) 2025-12-04T08:58:18.2046251Z 2025-12-04T08:58:18.2046357Z cudagraph partition due to non gpu ops 2025-12-04T08:58:18.2046596Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:18.2046948Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:18.2047260Z res = mod(**inputs) 2025-12-04T08:58:18.2047623Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T08:58:18.2048006Z outputs = self.roberta( 2025-12-04T08:58:18.2048365Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T08:58:18.2048757Z encoder_outputs = self.encoder( 2025-12-04T08:58:18.2049162Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T08:58:18.2049568Z layer_outputs = layer_module( 2025-12-04T08:58:18.2049902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:18.2050263Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:18.2050639Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2051002Z return func(*args, **kwargs) 2025-12-04T08:58:18.2051364Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T08:58:18.2051764Z self_attention_outputs = self.attention( 2025-12-04T08:58:18.2052140Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2052522Z return func(*args, **kwargs) 2025-12-04T08:58:18.2052899Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 440, in forward 2025-12-04T08:58:18.2053288Z self_outputs = self.self( 2025-12-04T08:58:18.2053652Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2054010Z return func(*args, **kwargs) 2025-12-04T08:58:18.2054383Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 363, in forward 2025-12-04T08:58:18.2054831Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T08:58:18.2055009Z 2025-12-04T08:58:18.2055119Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:18.2055466Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:18.2055785Z res = mod(**inputs) 2025-12-04T08:58:18.2056151Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T08:58:18.2056529Z outputs = self.roberta( 2025-12-04T08:58:18.2056896Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T08:58:18.2057283Z encoder_outputs = self.encoder( 2025-12-04T08:58:18.2057666Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T08:58:18.2058044Z layer_outputs = layer_module( 2025-12-04T08:58:18.2058383Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:18.2058746Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:18.2059124Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2059499Z return func(*args, **kwargs) 2025-12-04T08:58:18.2059904Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T08:58:18.2060315Z self_attention_outputs = self.attention( 2025-12-04T08:58:18.2060698Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2061120Z return func(*args, **kwargs) 2025-12-04T08:58:18.2061519Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 449, in forward 2025-12-04T08:58:18.2061990Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T08:58:18.2062447Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 387, in forward 2025-12-04T08:58:18.2062870Z hidden_states = self.dense(hidden_states) 2025-12-04T08:58:18.2063015Z 2025-12-04T08:58:18.2063142Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:18.2063510Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:18.2063825Z res = mod(**inputs) 2025-12-04T08:58:18.2064200Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T08:58:18.2064608Z outputs = self.roberta( 2025-12-04T08:58:18.2064991Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T08:58:18.2065411Z encoder_outputs = self.encoder( 2025-12-04T08:58:18.2065813Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T08:58:18.2066236Z layer_outputs = layer_module( 2025-12-04T08:58:18.2066595Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:18.2066972Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:18.2067369Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2067750Z return func(*args, **kwargs) 2025-12-04T08:58:18.2068150Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 543, in forward 2025-12-04T08:58:18.2068576Z layer_output = apply_chunking_to_forward( 2025-12-04T08:58:18.2068993Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:58:18.2069395Z return forward_fn(*input_tensors) 2025-12-04T08:58:18.2069840Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 551, in feed_forward_chunk 2025-12-04T08:58:18.2070334Z intermediate_output = self.intermediate(attention_output) 2025-12-04T08:58:18.2070785Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 465, in forward 2025-12-04T08:58:18.2071207Z hidden_states = self.dense(hidden_states) 2025-12-04T08:58:18.2071359Z 2025-12-04T08:58:18.2071468Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:18.2071843Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:18.2072174Z res = mod(**inputs) 2025-12-04T08:58:18.2072556Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T08:58:18.2072963Z outputs = self.roberta( 2025-12-04T08:58:18.2073354Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T08:58:18.2073753Z encoder_outputs = self.encoder( 2025-12-04T08:58:18.2074157Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T08:58:18.2074585Z layer_outputs = layer_module( 2025-12-04T08:58:18.2074937Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:18.2075312Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:18.2075706Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2076095Z return func(*args, **kwargs) 2025-12-04T08:58:18.2076507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 543, in forward 2025-12-04T08:58:18.2076947Z layer_output = apply_chunking_to_forward( 2025-12-04T08:58:18.2077405Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:58:18.2077845Z return forward_fn(*input_tensors) 2025-12-04T08:58:18.2078382Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 551, in feed_forward_chunk 2025-12-04T08:58:18.2078903Z intermediate_output = self.intermediate(attention_output) 2025-12-04T08:58:18.2079381Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-12-04T08:58:18.2079849Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:58:18.2080270Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T08:58:18.2080649Z return self.act(input) 2025-12-04T08:58:18.2080772Z 2025-12-04T08:58:18.2080894Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:18.2081307Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:18.2081666Z res = mod(**inputs) 2025-12-04T08:58:18.2082073Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T08:58:18.2082496Z outputs = self.roberta( 2025-12-04T08:58:18.2082910Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T08:58:18.2083342Z encoder_outputs = self.encoder( 2025-12-04T08:58:18.2083839Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T08:58:18.2084265Z layer_outputs = layer_module( 2025-12-04T08:58:18.2084646Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:18.2085045Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:18.2085467Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2085867Z return func(*args, **kwargs) 2025-12-04T08:58:18.2086287Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 543, in forward 2025-12-04T08:58:18.2086734Z layer_output = apply_chunking_to_forward( 2025-12-04T08:58:18.2087167Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:58:18.2087597Z return forward_fn(*input_tensors) 2025-12-04T08:58:18.2088067Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 552, in feed_forward_chunk 2025-12-04T08:58:18.2088599Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T08:58:18.2089090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 479, in forward 2025-12-04T08:58:18.2089521Z hidden_states = self.dense(hidden_states) 2025-12-04T08:58:18.2089662Z 2025-12-04T08:58:18.2089796Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:18.2090173Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:18.2090501Z res = mod(**inputs) 2025-12-04T08:58:18.2090886Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T08:58:18.2091294Z outputs = self.roberta( 2025-12-04T08:58:18.2091702Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T08:58:18.2092101Z encoder_outputs = self.encoder( 2025-12-04T08:58:18.2092508Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T08:58:18.2092952Z layer_outputs = layer_module( 2025-12-04T08:58:18.2093327Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:18.2093710Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:18.2094107Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2094487Z return func(*args, **kwargs) 2025-12-04T08:58:18.2094892Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T08:58:18.2095298Z self_attention_outputs = self.attention( 2025-12-04T08:58:18.2095686Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2096059Z return func(*args, **kwargs) 2025-12-04T08:58:18.2096468Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 440, in forward 2025-12-04T08:58:18.2096890Z self_outputs = self.self( 2025-12-04T08:58:18.2097284Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2097680Z return func(*args, **kwargs) 2025-12-04T08:58:18.2098097Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 313, in forward 2025-12-04T08:58:18.2098678Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-12-04T08:58:18.2098965Z 2025-12-04T08:58:18.2099079Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:18.2099475Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:18.2099829Z res = mod(**inputs) 2025-12-04T08:58:18.2100235Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T08:58:18.2100654Z outputs = self.roberta( 2025-12-04T08:58:18.2101037Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T08:58:18.2101442Z encoder_outputs = self.encoder( 2025-12-04T08:58:18.2101837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T08:58:18.2102256Z layer_outputs = layer_module( 2025-12-04T08:58:18.2102631Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:18.2103027Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:18.2103435Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2103840Z return func(*args, **kwargs) 2025-12-04T08:58:18.2104247Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T08:58:18.2104664Z self_attention_outputs = self.attention( 2025-12-04T08:58:18.2105071Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2105458Z return func(*args, **kwargs) 2025-12-04T08:58:18.2105851Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 440, in forward 2025-12-04T08:58:18.2106248Z self_outputs = self.self( 2025-12-04T08:58:18.2106624Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2107004Z return func(*args, **kwargs) 2025-12-04T08:58:18.2107401Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 337, in forward 2025-12-04T08:58:18.2107821Z self.key(current_states) 2025-12-04T08:58:18.2108003Z 2025-12-04T08:58:18.2108108Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:18.2108490Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:18.2108826Z res = mod(**inputs) 2025-12-04T08:58:18.2109219Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T08:58:18.2109630Z outputs = self.roberta( 2025-12-04T08:58:18.2110042Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T08:58:18.2110455Z encoder_outputs = self.encoder( 2025-12-04T08:58:18.2110889Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T08:58:18.2111341Z layer_outputs = layer_module( 2025-12-04T08:58:18.2111720Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:18.2112116Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:18.2112533Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2112945Z return func(*args, **kwargs) 2025-12-04T08:58:18.2113360Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T08:58:18.2113781Z self_attention_outputs = self.attention( 2025-12-04T08:58:18.2114173Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2114557Z return func(*args, **kwargs) 2025-12-04T08:58:18.2114945Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 440, in forward 2025-12-04T08:58:18.2115358Z self_outputs = self.self( 2025-12-04T08:58:18.2115736Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2116112Z return func(*args, **kwargs) 2025-12-04T08:58:18.2116502Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 342, in forward 2025-12-04T08:58:18.2116930Z self.value(current_states) 2025-12-04T08:58:18.2117058Z 2025-12-04T08:58:18.2117156Z cudagraph partition due to non gpu ops 2025-12-04T08:58:18.2117412Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:18.2117805Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:18.2118234Z res = mod(**inputs) 2025-12-04T08:58:18.2118663Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T08:58:18.2119118Z outputs = self.roberta( 2025-12-04T08:58:18.2119587Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T08:58:18.2120053Z encoder_outputs = self.encoder( 2025-12-04T08:58:18.2120506Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T08:58:18.2121082Z layer_outputs = layer_module( 2025-12-04T08:58:18.2121452Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:18.2121827Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:18.2122216Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2122609Z return func(*args, **kwargs) 2025-12-04T08:58:18.2123046Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T08:58:18.2123483Z self_attention_outputs = self.attention( 2025-12-04T08:58:18.2123882Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2124264Z return func(*args, **kwargs) 2025-12-04T08:58:18.2124659Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 440, in forward 2025-12-04T08:58:18.2125061Z self_outputs = self.self( 2025-12-04T08:58:18.2125438Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2125825Z return func(*args, **kwargs) 2025-12-04T08:58:18.2126216Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 363, in forward 2025-12-04T08:58:18.2126730Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T08:58:18.2126932Z 2025-12-04T08:58:18.2127040Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:18.2127421Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:18.2127748Z res = mod(**inputs) 2025-12-04T08:58:18.2128135Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T08:58:18.2128544Z outputs = self.roberta( 2025-12-04T08:58:18.2128934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T08:58:18.2129337Z encoder_outputs = self.encoder( 2025-12-04T08:58:18.2129736Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T08:58:18.2130147Z layer_outputs = layer_module( 2025-12-04T08:58:18.2130501Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:18.2130888Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:18.2131275Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2131663Z return func(*args, **kwargs) 2025-12-04T08:58:18.2132052Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T08:58:18.2132470Z self_attention_outputs = self.attention( 2025-12-04T08:58:18.2132867Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2133248Z return func(*args, **kwargs) 2025-12-04T08:58:18.2133635Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 449, in forward 2025-12-04T08:58:18.2134102Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T08:58:18.2134590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 387, in forward 2025-12-04T08:58:18.2135025Z hidden_states = self.dense(hidden_states) 2025-12-04T08:58:18.2135185Z 2025-12-04T08:58:18.2135298Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:18.2135692Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:18.2136046Z res = mod(**inputs) 2025-12-04T08:58:18.2136441Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T08:58:18.2136870Z outputs = self.roberta( 2025-12-04T08:58:18.2137280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T08:58:18.2137747Z encoder_outputs = self.encoder( 2025-12-04T08:58:18.2138194Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T08:58:18.2138600Z layer_outputs = layer_module( 2025-12-04T08:58:18.2138958Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:18.2139327Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:18.2139715Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2140119Z return func(*args, **kwargs) 2025-12-04T08:58:18.2140533Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 543, in forward 2025-12-04T08:58:18.2140968Z layer_output = apply_chunking_to_forward( 2025-12-04T08:58:18.2141422Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:58:18.2141830Z return forward_fn(*input_tensors) 2025-12-04T08:58:18.2142263Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 551, in feed_forward_chunk 2025-12-04T08:58:18.2142754Z intermediate_output = self.intermediate(attention_output) 2025-12-04T08:58:18.2143224Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 465, in forward 2025-12-04T08:58:18.2143666Z hidden_states = self.dense(hidden_states) 2025-12-04T08:58:18.2143807Z 2025-12-04T08:58:18.2143916Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:18.2144297Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:18.2144646Z res = mod(**inputs) 2025-12-04T08:58:18.2145055Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T08:58:18.2145479Z outputs = self.roberta( 2025-12-04T08:58:18.2145895Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T08:58:18.2146333Z encoder_outputs = self.encoder( 2025-12-04T08:58:18.2146765Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T08:58:18.2147198Z layer_outputs = layer_module( 2025-12-04T08:58:18.2147570Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:18.2147947Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:18.2148335Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2148724Z return func(*args, **kwargs) 2025-12-04T08:58:18.2149117Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 543, in forward 2025-12-04T08:58:18.2149530Z layer_output = apply_chunking_to_forward( 2025-12-04T08:58:18.2149985Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:58:18.2150400Z return forward_fn(*input_tensors) 2025-12-04T08:58:18.2150842Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 551, in feed_forward_chunk 2025-12-04T08:58:18.2151320Z intermediate_output = self.intermediate(attention_output) 2025-12-04T08:58:18.2151771Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-12-04T08:58:18.2152222Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:58:18.2152641Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T08:58:18.2153008Z return self.act(input) 2025-12-04T08:58:18.2153132Z 2025-12-04T08:58:18.2153243Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:18.2153622Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:18.2153954Z res = mod(**inputs) 2025-12-04T08:58:18.2154360Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T08:58:18.2154788Z outputs = self.roberta( 2025-12-04T08:58:18.2155197Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T08:58:18.2155621Z encoder_outputs = self.encoder( 2025-12-04T08:58:18.2156050Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T08:58:18.2156501Z layer_outputs = layer_module( 2025-12-04T08:58:18.2156882Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:18.2157291Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:18.2157719Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2158207Z return func(*args, **kwargs) 2025-12-04T08:58:18.2158643Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 543, in forward 2025-12-04T08:58:18.2159111Z layer_output = apply_chunking_to_forward( 2025-12-04T08:58:18.2159561Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:58:18.2159994Z return forward_fn(*input_tensors) 2025-12-04T08:58:18.2160426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 552, in feed_forward_chunk 2025-12-04T08:58:18.2160928Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T08:58:18.2161392Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 479, in forward 2025-12-04T08:58:18.2161825Z hidden_states = self.dense(hidden_states) 2025-12-04T08:58:18.2161966Z 2025-12-04T08:58:18.2162074Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:18.2162444Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:18.2162780Z res = mod(**inputs) 2025-12-04T08:58:18.2163152Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T08:58:18.2163558Z outputs = self.roberta( 2025-12-04T08:58:18.2163947Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T08:58:18.2164355Z encoder_outputs = self.encoder( 2025-12-04T08:58:18.2164775Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T08:58:18.2165188Z layer_outputs = layer_module( 2025-12-04T08:58:18.2165548Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:18.2165923Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:18.2166312Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2166702Z return func(*args, **kwargs) 2025-12-04T08:58:18.2167102Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T08:58:18.2167519Z self_attention_outputs = self.attention( 2025-12-04T08:58:18.2167976Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2168361Z return func(*args, **kwargs) 2025-12-04T08:58:18.2168743Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 440, in forward 2025-12-04T08:58:18.2169141Z self_outputs = self.self( 2025-12-04T08:58:18.2169509Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2169886Z return func(*args, **kwargs) 2025-12-04T08:58:18.2170264Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 313, in forward 2025-12-04T08:58:18.2170800Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-12-04T08:58:18.2171086Z 2025-12-04T08:58:18.2171194Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:18.2171565Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:18.2171888Z res = mod(**inputs) 2025-12-04T08:58:18.2172259Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T08:58:18.2172654Z outputs = self.roberta( 2025-12-04T08:58:18.2173028Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T08:58:18.2173430Z encoder_outputs = self.encoder( 2025-12-04T08:58:18.2173818Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T08:58:18.2174212Z layer_outputs = layer_module( 2025-12-04T08:58:18.2174559Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:18.2174935Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:18.2175319Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2175692Z return func(*args, **kwargs) 2025-12-04T08:58:18.2176066Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T08:58:18.2176473Z self_attention_outputs = self.attention( 2025-12-04T08:58:18.2176858Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2177223Z return func(*args, **kwargs) 2025-12-04T08:58:18.2177606Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 440, in forward 2025-12-04T08:58:18.2178001Z self_outputs = self.self( 2025-12-04T08:58:18.2178374Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2178741Z return func(*args, **kwargs) 2025-12-04T08:58:18.2179145Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 337, in forward 2025-12-04T08:58:18.2179542Z self.key(current_states) 2025-12-04T08:58:18.2179656Z 2025-12-04T08:58:18.2179760Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:18.2180130Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:18.2180452Z res = mod(**inputs) 2025-12-04T08:58:18.2180820Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T08:58:18.2181210Z outputs = self.roberta( 2025-12-04T08:58:18.2181619Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T08:58:18.2182043Z encoder_outputs = self.encoder( 2025-12-04T08:58:18.2182444Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T08:58:18.2182842Z layer_outputs = layer_module( 2025-12-04T08:58:18.2183201Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:18.2183578Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:18.2183968Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2184348Z return func(*args, **kwargs) 2025-12-04T08:58:18.2184743Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T08:58:18.2185183Z self_attention_outputs = self.attention( 2025-12-04T08:58:18.2185576Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2185963Z return func(*args, **kwargs) 2025-12-04T08:58:18.2186359Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 440, in forward 2025-12-04T08:58:18.2186760Z self_outputs = self.self( 2025-12-04T08:58:18.2187136Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2187521Z return func(*args, **kwargs) 2025-12-04T08:58:18.2187922Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 342, in forward 2025-12-04T08:58:18.2188345Z self.value(current_states) 2025-12-04T08:58:18.2188481Z 2025-12-04T08:58:18.2188573Z cudagraph partition due to non gpu ops 2025-12-04T08:58:18.2188837Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:18.2189225Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:18.2189583Z res = mod(**inputs) 2025-12-04T08:58:18.2189985Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T08:58:18.2190417Z outputs = self.roberta( 2025-12-04T08:58:18.2190819Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T08:58:18.2191255Z encoder_outputs = self.encoder( 2025-12-04T08:58:18.2191685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T08:58:18.2192117Z layer_outputs = layer_module( 2025-12-04T08:58:18.2192489Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:18.2192888Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:18.2193300Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2193723Z return func(*args, **kwargs) 2025-12-04T08:58:18.2194148Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T08:58:18.2194593Z self_attention_outputs = self.attention( 2025-12-04T08:58:18.2195017Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2195420Z return func(*args, **kwargs) 2025-12-04T08:58:18.2195843Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 440, in forward 2025-12-04T08:58:18.2196276Z self_outputs = self.self( 2025-12-04T08:58:18.2196695Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2197115Z return func(*args, **kwargs) 2025-12-04T08:58:18.2197535Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 363, in forward 2025-12-04T08:58:18.2198050Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T08:58:18.2198357Z 2025-12-04T08:58:18.2198480Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:18.2198891Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:18.2199256Z res = mod(**inputs) 2025-12-04T08:58:18.2199730Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T08:58:18.2200134Z outputs = self.roberta( 2025-12-04T08:58:18.2200569Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T08:58:18.2201028Z encoder_outputs = self.encoder( 2025-12-04T08:58:18.2201535Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T08:58:18.2201990Z layer_outputs = layer_module( 2025-12-04T08:58:18.2202386Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:18.2202804Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:18.2203226Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2203658Z return func(*args, **kwargs) 2025-12-04T08:58:18.2204091Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T08:58:18.2204551Z self_attention_outputs = self.attention( 2025-12-04T08:58:18.2204994Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2205415Z return func(*args, **kwargs) 2025-12-04T08:58:18.2205862Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 449, in forward 2025-12-04T08:58:18.2206362Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T08:58:18.2206882Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 387, in forward 2025-12-04T08:58:18.2207346Z hidden_states = self.dense(hidden_states) 2025-12-04T08:58:18.2207502Z 2025-12-04T08:58:18.2207627Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:18.2208025Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:18.2208391Z res = mod(**inputs) 2025-12-04T08:58:18.2208807Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T08:58:18.2209260Z outputs = self.roberta( 2025-12-04T08:58:18.2209708Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T08:58:18.2210159Z encoder_outputs = self.encoder( 2025-12-04T08:58:18.2210596Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T08:58:18.2211032Z layer_outputs = layer_module( 2025-12-04T08:58:18.2211422Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:18.2211835Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:18.2212272Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2212673Z return func(*args, **kwargs) 2025-12-04T08:58:18.2213114Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 543, in forward 2025-12-04T08:58:18.2213586Z layer_output = apply_chunking_to_forward( 2025-12-04T08:58:18.2214024Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:58:18.2214458Z return forward_fn(*input_tensors) 2025-12-04T08:58:18.2214930Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 551, in feed_forward_chunk 2025-12-04T08:58:18.2215452Z intermediate_output = self.intermediate(attention_output) 2025-12-04T08:58:18.2215933Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 465, in forward 2025-12-04T08:58:18.2216402Z hidden_states = self.dense(hidden_states) 2025-12-04T08:58:18.2216552Z 2025-12-04T08:58:18.2216675Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:18.2217068Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:18.2217425Z res = mod(**inputs) 2025-12-04T08:58:18.2217828Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T08:58:18.2218254Z outputs = self.roberta( 2025-12-04T08:58:18.2218658Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T08:58:18.2219091Z encoder_outputs = self.encoder( 2025-12-04T08:58:18.2219516Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T08:58:18.2219936Z layer_outputs = layer_module( 2025-12-04T08:58:18.2220290Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:18.2220668Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:18.2221247Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2221644Z return func(*args, **kwargs) 2025-12-04T08:58:18.2222045Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 543, in forward 2025-12-04T08:58:18.2222466Z layer_output = apply_chunking_to_forward( 2025-12-04T08:58:18.2222880Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:58:18.2223283Z return forward_fn(*input_tensors) 2025-12-04T08:58:18.2223728Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 551, in feed_forward_chunk 2025-12-04T08:58:18.2224223Z intermediate_output = self.intermediate(attention_output) 2025-12-04T08:58:18.2224683Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-12-04T08:58:18.2225183Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:58:18.2225582Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T08:58:18.2225949Z return self.act(input) 2025-12-04T08:58:18.2226069Z 2025-12-04T08:58:18.2226185Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:18.2226578Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:18.2226928Z res = mod(**inputs) 2025-12-04T08:58:18.2227345Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T08:58:18.2227768Z outputs = self.roberta( 2025-12-04T08:58:18.2228202Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T08:58:18.2228637Z encoder_outputs = self.encoder( 2025-12-04T08:58:18.2229036Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T08:58:18.2229442Z layer_outputs = layer_module( 2025-12-04T08:58:18.2229806Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:18.2230184Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:18.2230593Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2231004Z return func(*args, **kwargs) 2025-12-04T08:58:18.2231426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 543, in forward 2025-12-04T08:58:18.2231903Z layer_output = apply_chunking_to_forward( 2025-12-04T08:58:18.2232345Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:58:18.2232790Z return forward_fn(*input_tensors) 2025-12-04T08:58:18.2233237Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 552, in feed_forward_chunk 2025-12-04T08:58:18.2233742Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T08:58:18.2234218Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 479, in forward 2025-12-04T08:58:18.2234640Z hidden_states = self.dense(hidden_states) 2025-12-04T08:58:18.2234784Z 2025-12-04T08:58:18.2234900Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:18.2235276Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:18.2235616Z res = mod(**inputs) 2025-12-04T08:58:18.2236006Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T08:58:18.2236419Z outputs = self.roberta( 2025-12-04T08:58:18.2236806Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T08:58:18.2237221Z encoder_outputs = self.encoder( 2025-12-04T08:58:18.2237628Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T08:58:18.2238036Z layer_outputs = layer_module( 2025-12-04T08:58:18.2238461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:18.2238845Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:18.2239250Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2239654Z return func(*args, **kwargs) 2025-12-04T08:58:18.2240095Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T08:58:18.2240527Z self_attention_outputs = self.attention( 2025-12-04T08:58:18.2240939Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2241349Z return func(*args, **kwargs) 2025-12-04T08:58:18.2241770Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 440, in forward 2025-12-04T08:58:18.2242200Z self_outputs = self.self( 2025-12-04T08:58:18.2242590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2242996Z return func(*args, **kwargs) 2025-12-04T08:58:18.2243435Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 313, in forward 2025-12-04T08:58:18.2244040Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-12-04T08:58:18.2244328Z 2025-12-04T08:58:18.2244442Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:18.2244838Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:18.2245192Z res = mod(**inputs) 2025-12-04T08:58:18.2245590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T08:58:18.2246021Z outputs = self.roberta( 2025-12-04T08:58:18.2246428Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T08:58:18.2246888Z encoder_outputs = self.encoder( 2025-12-04T08:58:18.2247313Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T08:58:18.2247756Z layer_outputs = layer_module( 2025-12-04T08:58:18.2248145Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:18.2248541Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:18.2248965Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2249362Z return func(*args, **kwargs) 2025-12-04T08:58:18.2249767Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T08:58:18.2250190Z self_attention_outputs = self.attention( 2025-12-04T08:58:18.2250603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2251029Z return func(*args, **kwargs) 2025-12-04T08:58:18.2251464Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 440, in forward 2025-12-04T08:58:18.2251905Z self_outputs = self.self( 2025-12-04T08:58:18.2252314Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2252733Z return func(*args, **kwargs) 2025-12-04T08:58:18.2253150Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 337, in forward 2025-12-04T08:58:18.2253589Z self.key(current_states) 2025-12-04T08:58:18.2253724Z 2025-12-04T08:58:18.2253841Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:18.2254243Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:18.2254599Z res = mod(**inputs) 2025-12-04T08:58:18.2255011Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T08:58:18.2255477Z outputs = self.roberta( 2025-12-04T08:58:18.2255859Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T08:58:18.2256269Z encoder_outputs = self.encoder( 2025-12-04T08:58:18.2256667Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T08:58:18.2257073Z layer_outputs = layer_module( 2025-12-04T08:58:18.2257443Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:18.2257839Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:18.2258254Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2258678Z return func(*args, **kwargs) 2025-12-04T08:58:18.2259118Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T08:58:18.2259562Z self_attention_outputs = self.attention( 2025-12-04T08:58:18.2259980Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2260379Z return func(*args, **kwargs) 2025-12-04T08:58:18.2260798Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 440, in forward 2025-12-04T08:58:18.2261219Z self_outputs = self.self( 2025-12-04T08:58:18.2261606Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2262025Z return func(*args, **kwargs) 2025-12-04T08:58:18.2262449Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 342, in forward 2025-12-04T08:58:18.2262887Z self.value(current_states) 2025-12-04T08:58:18.2263017Z 2025-12-04T08:58:18.2263117Z cudagraph partition due to non gpu ops 2025-12-04T08:58:18.2263388Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:18.2263765Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:18.2264096Z res = mod(**inputs) 2025-12-04T08:58:18.2264475Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T08:58:18.2264908Z outputs = self.roberta( 2025-12-04T08:58:18.2265321Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T08:58:18.2265765Z encoder_outputs = self.encoder( 2025-12-04T08:58:18.2266190Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T08:58:18.2266627Z layer_outputs = layer_module( 2025-12-04T08:58:18.2267004Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:18.2267394Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:18.2267814Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2268226Z return func(*args, **kwargs) 2025-12-04T08:58:18.2268645Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T08:58:18.2269083Z self_attention_outputs = self.attention( 2025-12-04T08:58:18.2269359Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2269437Z return func(*args, **kwargs) 2025-12-04T08:58:18.2269728Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 440, in forward 2025-12-04T08:58:18.2269839Z self_outputs = self.self( 2025-12-04T08:58:18.2270106Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2270188Z return func(*args, **kwargs) 2025-12-04T08:58:18.2270477Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 363, in forward 2025-12-04T08:58:18.2270618Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T08:58:18.2270622Z 2025-12-04T08:58:18.2270743Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:18.2270972Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:18.2271051Z res = mod(**inputs) 2025-12-04T08:58:18.2271361Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T08:58:18.2271456Z outputs = self.roberta( 2025-12-04T08:58:18.2271755Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T08:58:18.2271834Z encoder_outputs = self.encoder( 2025-12-04T08:58:18.2272128Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T08:58:18.2272213Z layer_outputs = layer_module( 2025-12-04T08:58:18.2272461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:18.2272552Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:18.2272823Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2272920Z return func(*args, **kwargs) 2025-12-04T08:58:18.2273216Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T08:58:18.2273307Z self_attention_outputs = self.attention( 2025-12-04T08:58:18.2273574Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2273654Z return func(*args, **kwargs) 2025-12-04T08:58:18.2273943Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 449, in forward 2025-12-04T08:58:18.2274088Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T08:58:18.2274378Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 387, in forward 2025-12-04T08:58:18.2274469Z hidden_states = self.dense(hidden_states) 2025-12-04T08:58:18.2274473Z 2025-12-04T08:58:18.2274595Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:18.2274822Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:18.2274900Z res = mod(**inputs) 2025-12-04T08:58:18.2275211Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T08:58:18.2275286Z outputs = self.roberta( 2025-12-04T08:58:18.2275592Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T08:58:18.2275674Z encoder_outputs = self.encoder( 2025-12-04T08:58:18.2275995Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T08:58:18.2276081Z layer_outputs = layer_module( 2025-12-04T08:58:18.2276337Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:18.2276432Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:18.2276728Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2276807Z return func(*args, **kwargs) 2025-12-04T08:58:18.2277110Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 543, in forward 2025-12-04T08:58:18.2277204Z layer_output = apply_chunking_to_forward( 2025-12-04T08:58:18.2277504Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:58:18.2277591Z return forward_fn(*input_tensors) 2025-12-04T08:58:18.2277924Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 551, in feed_forward_chunk 2025-12-04T08:58:18.2278089Z intermediate_output = self.intermediate(attention_output) 2025-12-04T08:58:18.2278483Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 465, in forward 2025-12-04T08:58:18.2278581Z hidden_states = self.dense(hidden_states) 2025-12-04T08:58:18.2278593Z 2025-12-04T08:58:18.2278710Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:18.2278935Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:18.2279015Z res = mod(**inputs) 2025-12-04T08:58:18.2279317Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T08:58:18.2279393Z outputs = self.roberta( 2025-12-04T08:58:18.2279686Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T08:58:18.2279791Z encoder_outputs = self.encoder( 2025-12-04T08:58:18.2280082Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T08:58:18.2280165Z layer_outputs = layer_module( 2025-12-04T08:58:18.2280409Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:18.2280503Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:18.2280766Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2280842Z return func(*args, **kwargs) 2025-12-04T08:58:18.2281133Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 543, in forward 2025-12-04T08:58:18.2281224Z layer_output = apply_chunking_to_forward( 2025-12-04T08:58:18.2281513Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:58:18.2281601Z return forward_fn(*input_tensors) 2025-12-04T08:58:18.2281923Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 551, in feed_forward_chunk 2025-12-04T08:58:18.2282064Z intermediate_output = self.intermediate(attention_output) 2025-12-04T08:58:18.2282345Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-12-04T08:58:18.2282473Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:58:18.2282706Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T08:58:18.2282783Z return self.act(input) 2025-12-04T08:58:18.2282787Z 2025-12-04T08:58:18.2282904Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:18.2283122Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:18.2283194Z res = mod(**inputs) 2025-12-04T08:58:18.2283508Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T08:58:18.2283588Z outputs = self.roberta( 2025-12-04T08:58:18.2283881Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T08:58:18.2283961Z encoder_outputs = self.encoder( 2025-12-04T08:58:18.2284245Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T08:58:18.2284329Z layer_outputs = layer_module( 2025-12-04T08:58:18.2284573Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:18.2284658Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:18.2284951Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2285043Z return func(*args, **kwargs) 2025-12-04T08:58:18.2285342Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 543, in forward 2025-12-04T08:58:18.2285431Z layer_output = apply_chunking_to_forward( 2025-12-04T08:58:18.2285716Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:58:18.2285807Z return forward_fn(*input_tensors) 2025-12-04T08:58:18.2286132Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 552, in feed_forward_chunk 2025-12-04T08:58:18.2286285Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T08:58:18.2286588Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 479, in forward 2025-12-04T08:58:18.2286678Z hidden_states = self.dense(hidden_states) 2025-12-04T08:58:18.2286683Z 2025-12-04T08:58:18.2286802Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:18.2287020Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:18.2287090Z res = mod(**inputs) 2025-12-04T08:58:18.2287387Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T08:58:18.2287461Z outputs = self.roberta( 2025-12-04T08:58:18.2287757Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T08:58:18.2287834Z encoder_outputs = self.encoder( 2025-12-04T08:58:18.2288118Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T08:58:18.2288204Z layer_outputs = layer_module( 2025-12-04T08:58:18.2288449Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:18.2288543Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:18.2288810Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2288884Z return func(*args, **kwargs) 2025-12-04T08:58:18.2289176Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T08:58:18.2289265Z self_attention_outputs = self.attention( 2025-12-04T08:58:18.2289534Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2289617Z return func(*args, **kwargs) 2025-12-04T08:58:18.2289903Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 440, in forward 2025-12-04T08:58:18.2289992Z self_outputs = self.self( 2025-12-04T08:58:18.2290326Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2290401Z return func(*args, **kwargs) 2025-12-04T08:58:18.2290696Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 313, in forward 2025-12-04T08:58:18.2290925Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-12-04T08:58:18.2290929Z 2025-12-04T08:58:18.2291046Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:18.2291263Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:18.2291332Z res = mod(**inputs) 2025-12-04T08:58:18.2291660Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T08:58:18.2291751Z outputs = self.roberta( 2025-12-04T08:58:18.2292036Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T08:58:18.2292124Z encoder_outputs = self.encoder( 2025-12-04T08:58:18.2292416Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T08:58:18.2292496Z layer_outputs = layer_module( 2025-12-04T08:58:18.2292725Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:18.2292804Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:18.2293061Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2293151Z return func(*args, **kwargs) 2025-12-04T08:58:18.2293427Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T08:58:18.2293521Z self_attention_outputs = self.attention( 2025-12-04T08:58:18.2293778Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2293856Z return func(*args, **kwargs) 2025-12-04T08:58:18.2294132Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 440, in forward 2025-12-04T08:58:18.2294206Z self_outputs = self.self( 2025-12-04T08:58:18.2294470Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2294540Z return func(*args, **kwargs) 2025-12-04T08:58:18.2294822Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 337, in forward 2025-12-04T08:58:18.2294899Z self.key(current_states) 2025-12-04T08:58:18.2294904Z 2025-12-04T08:58:18.2295009Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:18.2295228Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:18.2295294Z res = mod(**inputs) 2025-12-04T08:58:18.2295569Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T08:58:18.2295647Z outputs = self.roberta( 2025-12-04T08:58:18.2295920Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T08:58:18.2296002Z encoder_outputs = self.encoder( 2025-12-04T08:58:18.2296296Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T08:58:18.2296375Z layer_outputs = layer_module( 2025-12-04T08:58:18.2296634Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:18.2296721Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:18.2297018Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2297095Z return func(*args, **kwargs) 2025-12-04T08:58:18.2297384Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T08:58:18.2297480Z self_attention_outputs = self.attention( 2025-12-04T08:58:18.2297745Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2297818Z return func(*args, **kwargs) 2025-12-04T08:58:18.2298138Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 440, in forward 2025-12-04T08:58:18.2298232Z self_outputs = self.self( 2025-12-04T08:58:18.2298512Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2298584Z return func(*args, **kwargs) 2025-12-04T08:58:18.2298860Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 342, in forward 2025-12-04T08:58:18.2298945Z self.value(current_states) 2025-12-04T08:58:18.2298949Z 2025-12-04T08:58:18.2299035Z cudagraph partition due to non gpu ops 2025-12-04T08:58:18.2299142Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:18.2299361Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:18.2299430Z res = mod(**inputs) 2025-12-04T08:58:18.2299734Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T08:58:18.2299850Z outputs = self.roberta( 2025-12-04T08:58:18.2300130Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T08:58:18.2300216Z encoder_outputs = self.encoder( 2025-12-04T08:58:18.2300495Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T08:58:18.2300578Z layer_outputs = layer_module( 2025-12-04T08:58:18.2300818Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:18.2300899Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:18.2301164Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2301236Z return func(*args, **kwargs) 2025-12-04T08:58:18.2301519Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T08:58:18.2301615Z self_attention_outputs = self.attention( 2025-12-04T08:58:18.2301875Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2301954Z return func(*args, **kwargs) 2025-12-04T08:58:18.2302240Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 440, in forward 2025-12-04T08:58:18.2302313Z self_outputs = self.self( 2025-12-04T08:58:18.2302578Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2302650Z return func(*args, **kwargs) 2025-12-04T08:58:18.2302951Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 363, in forward 2025-12-04T08:58:18.2303099Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T08:58:18.2303106Z 2025-12-04T08:58:18.2303215Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:18.2303459Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:18.2303528Z res = mod(**inputs) 2025-12-04T08:58:18.2303804Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T08:58:18.2303884Z outputs = self.roberta( 2025-12-04T08:58:18.2304161Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T08:58:18.2304245Z encoder_outputs = self.encoder( 2025-12-04T08:58:18.2304544Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T08:58:18.2304621Z layer_outputs = layer_module( 2025-12-04T08:58:18.2304921Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:18.2305020Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:18.2305279Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2305359Z return func(*args, **kwargs) 2025-12-04T08:58:18.2305639Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T08:58:18.2305732Z self_attention_outputs = self.attention( 2025-12-04T08:58:18.2305988Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2306058Z return func(*args, **kwargs) 2025-12-04T08:58:18.2306340Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 449, in forward 2025-12-04T08:58:18.2306497Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T08:58:18.2306795Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 387, in forward 2025-12-04T08:58:18.2306883Z hidden_states = self.dense(hidden_states) 2025-12-04T08:58:18.2306887Z 2025-12-04T08:58:18.2306997Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:18.2307219Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:18.2307289Z res = mod(**inputs) 2025-12-04T08:58:18.2307574Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T08:58:18.2307657Z outputs = self.roberta( 2025-12-04T08:58:18.2307939Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T08:58:18.2308027Z encoder_outputs = self.encoder( 2025-12-04T08:58:18.2308313Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T08:58:18.2308393Z layer_outputs = layer_module( 2025-12-04T08:58:18.2308642Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:18.2308727Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:18.2308993Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2309074Z return func(*args, **kwargs) 2025-12-04T08:58:18.2309355Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 543, in forward 2025-12-04T08:58:18.2309449Z layer_output = apply_chunking_to_forward( 2025-12-04T08:58:18.2309733Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:58:18.2309818Z return forward_fn(*input_tensors) 2025-12-04T08:58:18.2310163Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 551, in feed_forward_chunk 2025-12-04T08:58:18.2310296Z intermediate_output = self.intermediate(attention_output) 2025-12-04T08:58:18.2310586Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 465, in forward 2025-12-04T08:58:18.2310671Z hidden_states = self.dense(hidden_states) 2025-12-04T08:58:18.2310674Z 2025-12-04T08:58:18.2310780Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:18.2310991Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:18.2311056Z res = mod(**inputs) 2025-12-04T08:58:18.2311346Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T08:58:18.2311440Z outputs = self.roberta( 2025-12-04T08:58:18.2311711Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T08:58:18.2311794Z encoder_outputs = self.encoder( 2025-12-04T08:58:18.2312062Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T08:58:18.2312135Z layer_outputs = layer_module( 2025-12-04T08:58:18.2312371Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:18.2312450Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:18.2312709Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2312798Z return func(*args, **kwargs) 2025-12-04T08:58:18.2313073Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 543, in forward 2025-12-04T08:58:18.2313167Z layer_output = apply_chunking_to_forward( 2025-12-04T08:58:18.2313432Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:58:18.2313510Z return forward_fn(*input_tensors) 2025-12-04T08:58:18.2313822Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 551, in feed_forward_chunk 2025-12-04T08:58:18.2313944Z intermediate_output = self.intermediate(attention_output) 2025-12-04T08:58:18.2314218Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-12-04T08:58:18.2314334Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:58:18.2314554Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T08:58:18.2314634Z return self.act(input) 2025-12-04T08:58:18.2314638Z 2025-12-04T08:58:18.2314744Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:18.2314963Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:18.2315032Z res = mod(**inputs) 2025-12-04T08:58:18.2315315Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T08:58:18.2315396Z outputs = self.roberta( 2025-12-04T08:58:18.2315681Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T08:58:18.2315758Z encoder_outputs = self.encoder( 2025-12-04T08:58:18.2316047Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T08:58:18.2316129Z layer_outputs = layer_module( 2025-12-04T08:58:18.2316376Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:18.2316487Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:18.2316754Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2316836Z return func(*args, **kwargs) 2025-12-04T08:58:18.2317117Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 543, in forward 2025-12-04T08:58:18.2317206Z layer_output = apply_chunking_to_forward( 2025-12-04T08:58:18.2317495Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:58:18.2317578Z return forward_fn(*input_tensors) 2025-12-04T08:58:18.2317928Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 552, in feed_forward_chunk 2025-12-04T08:58:18.2318096Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T08:58:18.2318465Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 479, in forward 2025-12-04T08:58:18.2318572Z hidden_states = self.dense(hidden_states) 2025-12-04T08:58:18.2318577Z 2025-12-04T08:58:18.2318691Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:18.2318918Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:18.2318991Z res = mod(**inputs) 2025-12-04T08:58:18.2319282Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T08:58:18.2319390Z outputs = self.roberta( 2025-12-04T08:58:18.2319673Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T08:58:18.2319757Z encoder_outputs = self.encoder( 2025-12-04T08:58:18.2320029Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T08:58:18.2320103Z layer_outputs = layer_module( 2025-12-04T08:58:18.2320338Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:18.2320419Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:18.2320668Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2320897Z return func(*args, **kwargs) 2025-12-04T08:58:18.2321173Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T08:58:18.2321267Z self_attention_outputs = self.attention( 2025-12-04T08:58:18.2321521Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2321594Z return func(*args, **kwargs) 2025-12-04T08:58:18.2321874Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 440, in forward 2025-12-04T08:58:18.2321950Z self_outputs = self.self( 2025-12-04T08:58:18.2322201Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2322280Z return func(*args, **kwargs) 2025-12-04T08:58:18.2322549Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 313, in forward 2025-12-04T08:58:18.2322772Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-12-04T08:58:18.2322777Z 2025-12-04T08:58:18.2322885Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:18.2323096Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:18.2323213Z res = mod(**inputs) 2025-12-04T08:58:18.2323486Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T08:58:18.2323563Z outputs = self.roberta( 2025-12-04T08:58:18.2323833Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T08:58:18.2323906Z encoder_outputs = self.encoder( 2025-12-04T08:58:18.2324186Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T08:58:18.2324259Z layer_outputs = layer_module( 2025-12-04T08:58:18.2324491Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:18.2324601Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:18.2324876Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2324955Z return func(*args, **kwargs) 2025-12-04T08:58:18.2325223Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T08:58:18.2325307Z self_attention_outputs = self.attention( 2025-12-04T08:58:18.2325566Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2325637Z return func(*args, **kwargs) 2025-12-04T08:58:18.2325911Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 440, in forward 2025-12-04T08:58:18.2326016Z self_outputs = self.self( 2025-12-04T08:58:18.2326284Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2326367Z return func(*args, **kwargs) 2025-12-04T08:58:18.2326653Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 337, in forward 2025-12-04T08:58:18.2326729Z self.key(current_states) 2025-12-04T08:58:18.2326733Z 2025-12-04T08:58:18.2326852Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:18.2327068Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:18.2327145Z res = mod(**inputs) 2025-12-04T08:58:18.2327431Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T08:58:18.2327501Z outputs = self.roberta( 2025-12-04T08:58:18.2327780Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T08:58:18.2327857Z encoder_outputs = self.encoder( 2025-12-04T08:58:18.2328128Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T08:58:18.2328208Z layer_outputs = layer_module( 2025-12-04T08:58:18.2328445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:18.2328536Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:18.2328800Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2328874Z return func(*args, **kwargs) 2025-12-04T08:58:18.2329168Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T08:58:18.2329260Z self_attention_outputs = self.attention( 2025-12-04T08:58:18.2329533Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2329608Z return func(*args, **kwargs) 2025-12-04T08:58:18.2329916Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 440, in forward 2025-12-04T08:58:18.2330004Z self_outputs = self.self( 2025-12-04T08:58:18.2330267Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2330339Z return func(*args, **kwargs) 2025-12-04T08:58:18.2330630Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 342, in forward 2025-12-04T08:58:18.2330709Z self.value(current_states) 2025-12-04T08:58:18.2330713Z 2025-12-04T08:58:18.2330811Z cudagraph partition due to non gpu ops 2025-12-04T08:58:18.2330926Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:18.2331158Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:18.2331253Z res = mod(**inputs) 2025-12-04T08:58:18.2331541Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T08:58:18.2331614Z outputs = self.roberta( 2025-12-04T08:58:18.2331906Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T08:58:18.2331984Z encoder_outputs = self.encoder( 2025-12-04T08:58:18.2332277Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T08:58:18.2332363Z layer_outputs = layer_module( 2025-12-04T08:58:18.2332604Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:18.2332715Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:18.2332981Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2333064Z return func(*args, **kwargs) 2025-12-04T08:58:18.2333348Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T08:58:18.2333436Z self_attention_outputs = self.attention( 2025-12-04T08:58:18.2333705Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2333778Z return func(*args, **kwargs) 2025-12-04T08:58:18.2334061Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 440, in forward 2025-12-04T08:58:18.2334145Z self_outputs = self.self( 2025-12-04T08:58:18.2334411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2334492Z return func(*args, **kwargs) 2025-12-04T08:58:18.2334778Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 363, in forward 2025-12-04T08:58:18.2334922Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T08:58:18.2334926Z 2025-12-04T08:58:18.2335041Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:18.2335254Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:18.2335333Z res = mod(**inputs) 2025-12-04T08:58:18.2335618Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T08:58:18.2335693Z outputs = self.roberta( 2025-12-04T08:58:18.2335980Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T08:58:18.2336062Z encoder_outputs = self.encoder( 2025-12-04T08:58:18.2336368Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T08:58:18.2336456Z layer_outputs = layer_module( 2025-12-04T08:58:18.2336698Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:18.2336791Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:18.2337068Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2337142Z return func(*args, **kwargs) 2025-12-04T08:58:18.2337437Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T08:58:18.2337537Z self_attention_outputs = self.attention( 2025-12-04T08:58:18.2337797Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2337890Z return func(*args, **kwargs) 2025-12-04T08:58:18.2338152Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 449, in forward 2025-12-04T08:58:18.2338290Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T08:58:18.2338550Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 387, in forward 2025-12-04T08:58:18.2338634Z hidden_states = self.dense(hidden_states) 2025-12-04T08:58:18.2338637Z 2025-12-04T08:58:18.2338747Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:18.2338945Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:18.2339047Z res = mod(**inputs) 2025-12-04T08:58:18.2339312Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T08:58:18.2339383Z outputs = self.roberta( 2025-12-04T08:58:18.2339653Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T08:58:18.2339725Z encoder_outputs = self.encoder( 2025-12-04T08:58:18.2339986Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T08:58:18.2340067Z layer_outputs = layer_module( 2025-12-04T08:58:18.2340290Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:18.2340374Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:18.2340619Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2340689Z return func(*args, **kwargs) 2025-12-04T08:58:18.2340963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 543, in forward 2025-12-04T08:58:18.2341049Z layer_output = apply_chunking_to_forward( 2025-12-04T08:58:18.2341318Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:58:18.2341395Z return forward_fn(*input_tensors) 2025-12-04T08:58:18.2341693Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 551, in feed_forward_chunk 2025-12-04T08:58:18.2341818Z intermediate_output = self.intermediate(attention_output) 2025-12-04T08:58:18.2342080Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 465, in forward 2025-12-04T08:58:18.2342163Z hidden_states = self.dense(hidden_states) 2025-12-04T08:58:18.2342166Z 2025-12-04T08:58:18.2342277Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:18.2342476Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:18.2342565Z res = mod(**inputs) 2025-12-04T08:58:18.2342827Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T08:58:18.2342895Z outputs = self.roberta( 2025-12-04T08:58:18.2343163Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T08:58:18.2343236Z encoder_outputs = self.encoder( 2025-12-04T08:58:18.2343503Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T08:58:18.2343574Z layer_outputs = layer_module( 2025-12-04T08:58:18.2343802Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:18.2343903Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:18.2344166Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2344235Z return func(*args, **kwargs) 2025-12-04T08:58:18.2344502Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 543, in forward 2025-12-04T08:58:18.2344584Z layer_output = apply_chunking_to_forward( 2025-12-04T08:58:18.2344849Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:58:18.2344926Z return forward_fn(*input_tensors) 2025-12-04T08:58:18.2345220Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 551, in feed_forward_chunk 2025-12-04T08:58:18.2345367Z intermediate_output = self.intermediate(attention_output) 2025-12-04T08:58:18.2345630Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-12-04T08:58:18.2345746Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:58:18.2345970Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T08:58:18.2346042Z return self.act(input) 2025-12-04T08:58:18.2346046Z 2025-12-04T08:58:18.2346154Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:18.2346353Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:18.2346417Z res = mod(**inputs) 2025-12-04T08:58:18.2346686Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T08:58:18.2346757Z outputs = self.roberta( 2025-12-04T08:58:18.2347025Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T08:58:18.2347098Z encoder_outputs = self.encoder( 2025-12-04T08:58:18.2347359Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T08:58:18.2347436Z layer_outputs = layer_module( 2025-12-04T08:58:18.2347658Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:18.2347734Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:18.2347986Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2348054Z return func(*args, **kwargs) 2025-12-04T08:58:18.2348320Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 543, in forward 2025-12-04T08:58:18.2348404Z layer_output = apply_chunking_to_forward( 2025-12-04T08:58:18.2348665Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:58:18.2348765Z return forward_fn(*input_tensors) 2025-12-04T08:58:18.2349065Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 552, in feed_forward_chunk 2025-12-04T08:58:18.2349204Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T08:58:18.2349475Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 479, in forward 2025-12-04T08:58:18.2349556Z hidden_states = self.dense(hidden_states) 2025-12-04T08:58:18.2349560Z 2025-12-04T08:58:18.2349670Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:18.2349872Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:18.2349953Z res = mod(**inputs) 2025-12-04T08:58:18.2350243Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T08:58:18.2350314Z outputs = self.roberta( 2025-12-04T08:58:18.2350586Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T08:58:18.2350659Z encoder_outputs = self.encoder( 2025-12-04T08:58:18.2350922Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T08:58:18.2351000Z layer_outputs = layer_module( 2025-12-04T08:58:18.2351221Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:18.2351307Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:18.2351578Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2351650Z return func(*args, **kwargs) 2025-12-04T08:58:18.2351920Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T08:58:18.2352003Z self_attention_outputs = self.attention( 2025-12-04T08:58:18.2352245Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2352323Z return func(*args, **kwargs) 2025-12-04T08:58:18.2352585Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 440, in forward 2025-12-04T08:58:18.2352664Z self_outputs = self.self( 2025-12-04T08:58:18.2352907Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2352979Z return func(*args, **kwargs) 2025-12-04T08:58:18.2353248Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 313, in forward 2025-12-04T08:58:18.2353457Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-12-04T08:58:18.2353460Z 2025-12-04T08:58:18.2353571Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:18.2353774Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:18.2353840Z res = mod(**inputs) 2025-12-04T08:58:18.2354120Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T08:58:18.2354192Z outputs = self.roberta( 2025-12-04T08:58:18.2354476Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T08:58:18.2354565Z encoder_outputs = self.encoder( 2025-12-04T08:58:18.2354851Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T08:58:18.2354936Z layer_outputs = layer_module( 2025-12-04T08:58:18.2355195Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:18.2355280Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:18.2355549Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2355618Z return func(*args, **kwargs) 2025-12-04T08:58:18.2355887Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T08:58:18.2355984Z self_attention_outputs = self.attention( 2025-12-04T08:58:18.2356253Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2356350Z return func(*args, **kwargs) 2025-12-04T08:58:18.2356702Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 440, in forward 2025-12-04T08:58:18.2356779Z self_outputs = self.self( 2025-12-04T08:58:18.2357049Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2357124Z return func(*args, **kwargs) 2025-12-04T08:58:18.2357418Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 337, in forward 2025-12-04T08:58:18.2357494Z self.key(current_states) 2025-12-04T08:58:18.2357497Z 2025-12-04T08:58:18.2357608Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:18.2357829Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:18.2357919Z res = mod(**inputs) 2025-12-04T08:58:18.2358286Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T08:58:18.2358378Z outputs = self.roberta( 2025-12-04T08:58:18.2358672Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T08:58:18.2358763Z encoder_outputs = self.encoder( 2025-12-04T08:58:18.2359059Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T08:58:18.2359140Z layer_outputs = layer_module( 2025-12-04T08:58:18.2359400Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:18.2359487Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:18.2359772Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2359869Z return func(*args, **kwargs) 2025-12-04T08:58:18.2360161Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T08:58:18.2360260Z self_attention_outputs = self.attention( 2025-12-04T08:58:18.2360526Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2360601Z return func(*args, **kwargs) 2025-12-04T08:58:18.2360895Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 440, in forward 2025-12-04T08:58:18.2360974Z self_outputs = self.self( 2025-12-04T08:58:18.2361243Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2361321Z return func(*args, **kwargs) 2025-12-04T08:58:18.2361608Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 342, in forward 2025-12-04T08:58:18.2361696Z self.value(current_states) 2025-12-04T08:58:18.2361699Z 2025-12-04T08:58:18.2361810Z cudagraph partition due to non gpu ops 2025-12-04T08:58:18.2361926Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:18.2362149Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:18.2362218Z res = mod(**inputs) 2025-12-04T08:58:18.2362510Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T08:58:18.2362584Z outputs = self.roberta( 2025-12-04T08:58:18.2362876Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T08:58:18.2362962Z encoder_outputs = self.encoder( 2025-12-04T08:58:18.2363265Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T08:58:18.2363393Z layer_outputs = layer_module( 2025-12-04T08:58:18.2363642Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:18.2363726Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:18.2363994Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2364069Z return func(*args, **kwargs) 2025-12-04T08:58:18.2364358Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T08:58:18.2364456Z self_attention_outputs = self.attention( 2025-12-04T08:58:18.2364715Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2364812Z return func(*args, **kwargs) 2025-12-04T08:58:18.2365104Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 440, in forward 2025-12-04T08:58:18.2365183Z self_outputs = self.self( 2025-12-04T08:58:18.2365457Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2365530Z return func(*args, **kwargs) 2025-12-04T08:58:18.2365815Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 363, in forward 2025-12-04T08:58:18.2365966Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T08:58:18.2365970Z 2025-12-04T08:58:18.2366078Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:18.2366303Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:18.2366375Z res = mod(**inputs) 2025-12-04T08:58:18.2366664Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T08:58:18.2366749Z outputs = self.roberta( 2025-12-04T08:58:18.2367037Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T08:58:18.2367124Z encoder_outputs = self.encoder( 2025-12-04T08:58:18.2367409Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T08:58:18.2367486Z layer_outputs = layer_module( 2025-12-04T08:58:18.2367732Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:18.2367817Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:18.2368083Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2368168Z return func(*args, **kwargs) 2025-12-04T08:58:18.2368455Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T08:58:18.2368569Z self_attention_outputs = self.attention( 2025-12-04T08:58:18.2368835Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2368908Z return func(*args, **kwargs) 2025-12-04T08:58:18.2369199Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 449, in forward 2025-12-04T08:58:18.2369339Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T08:58:18.2369626Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 387, in forward 2025-12-04T08:58:18.2369725Z hidden_states = self.dense(hidden_states) 2025-12-04T08:58:18.2369729Z 2025-12-04T08:58:18.2369858Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:18.2370102Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:18.2370174Z res = mod(**inputs) 2025-12-04T08:58:18.2370457Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T08:58:18.2370539Z outputs = self.roberta( 2025-12-04T08:58:18.2370824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T08:58:18.2370909Z encoder_outputs = self.encoder( 2025-12-04T08:58:18.2371201Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T08:58:18.2371277Z layer_outputs = layer_module( 2025-12-04T08:58:18.2371543Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:18.2371628Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:18.2371892Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2371972Z return func(*args, **kwargs) 2025-12-04T08:58:18.2372265Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 543, in forward 2025-12-04T08:58:18.2372362Z layer_output = apply_chunking_to_forward( 2025-12-04T08:58:18.2372644Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:58:18.2372725Z return forward_fn(*input_tensors) 2025-12-04T08:58:18.2373053Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 551, in feed_forward_chunk 2025-12-04T08:58:18.2373187Z intermediate_output = self.intermediate(attention_output) 2025-12-04T08:58:18.2373480Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 465, in forward 2025-12-04T08:58:18.2373570Z hidden_states = self.dense(hidden_states) 2025-12-04T08:58:18.2373574Z 2025-12-04T08:58:18.2373684Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:18.2373910Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:18.2373978Z res = mod(**inputs) 2025-12-04T08:58:18.2374263Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T08:58:18.2374343Z outputs = self.roberta( 2025-12-04T08:58:18.2374627Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T08:58:18.2374715Z encoder_outputs = self.encoder( 2025-12-04T08:58:18.2375000Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T08:58:18.2375079Z layer_outputs = layer_module( 2025-12-04T08:58:18.2375344Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:18.2375429Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:18.2375715Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2375790Z return func(*args, **kwargs) 2025-12-04T08:58:18.2376115Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 543, in forward 2025-12-04T08:58:18.2376205Z layer_output = apply_chunking_to_forward( 2025-12-04T08:58:18.2376474Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:58:18.2376568Z return forward_fn(*input_tensors) 2025-12-04T08:58:18.2376905Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 551, in feed_forward_chunk 2025-12-04T08:58:18.2377029Z intermediate_output = self.intermediate(attention_output) 2025-12-04T08:58:18.2377305Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-12-04T08:58:18.2377422Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:58:18.2377644Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T08:58:18.2377724Z return self.act(input) 2025-12-04T08:58:18.2377728Z 2025-12-04T08:58:18.2377838Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:18.2378080Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:18.2378154Z res = mod(**inputs) 2025-12-04T08:58:18.2378442Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T08:58:18.2378526Z outputs = self.roberta( 2025-12-04T08:58:18.2378809Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T08:58:18.2378889Z encoder_outputs = self.encoder( 2025-12-04T08:58:18.2379180Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T08:58:18.2379258Z layer_outputs = layer_module( 2025-12-04T08:58:18.2379509Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:18.2379594Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:18.2379866Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2379950Z return func(*args, **kwargs) 2025-12-04T08:58:18.2380234Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 543, in forward 2025-12-04T08:58:18.2380325Z layer_output = apply_chunking_to_forward( 2025-12-04T08:58:18.2380611Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:58:18.2380693Z return forward_fn(*input_tensors) 2025-12-04T08:58:18.2381022Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 552, in feed_forward_chunk 2025-12-04T08:58:18.2381164Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T08:58:18.2381447Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 479, in forward 2025-12-04T08:58:18.2381548Z hidden_states = self.dense(hidden_states) 2025-12-04T08:58:18.2381553Z 2025-12-04T08:58:18.2381664Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:18.2381905Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:18.2381976Z res = mod(**inputs) 2025-12-04T08:58:18.2382259Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T08:58:18.2382341Z outputs = self.roberta( 2025-12-04T08:58:18.2382625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T08:58:18.2382705Z encoder_outputs = self.encoder( 2025-12-04T08:58:18.2383002Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T08:58:18.2383080Z layer_outputs = layer_module( 2025-12-04T08:58:18.2383350Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:18.2383457Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:18.2383725Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2383810Z return func(*args, **kwargs) 2025-12-04T08:58:18.2384093Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T08:58:18.2384190Z self_attention_outputs = self.attention( 2025-12-04T08:58:18.2384455Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2384530Z return func(*args, **kwargs) 2025-12-04T08:58:18.2384841Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 440, in forward 2025-12-04T08:58:18.2384920Z self_outputs = self.self( 2025-12-04T08:58:18.2385184Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2385267Z return func(*args, **kwargs) 2025-12-04T08:58:18.2385548Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 313, in forward 2025-12-04T08:58:18.2385783Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-12-04T08:58:18.2385787Z 2025-12-04T08:58:18.2385898Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:18.2386113Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:18.2386200Z res = mod(**inputs) 2025-12-04T08:58:18.2386471Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T08:58:18.2386548Z outputs = self.roberta( 2025-12-04T08:58:18.2386815Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T08:58:18.2386888Z encoder_outputs = self.encoder( 2025-12-04T08:58:18.2387161Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T08:58:18.2387233Z layer_outputs = layer_module( 2025-12-04T08:58:18.2387468Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:18.2387561Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:18.2387823Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2387905Z return func(*args, **kwargs) 2025-12-04T08:58:18.2388187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T08:58:18.2388276Z self_attention_outputs = self.attention( 2025-12-04T08:58:18.2388561Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2388634Z return func(*args, **kwargs) 2025-12-04T08:58:18.2391661Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 440, in forward 2025-12-04T08:58:18.2391750Z self_outputs = self.self( 2025-12-04T08:58:18.2392024Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2392097Z return func(*args, **kwargs) 2025-12-04T08:58:18.2392390Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 337, in forward 2025-12-04T08:58:18.2392492Z self.key(current_states) 2025-12-04T08:58:18.2392497Z 2025-12-04T08:58:18.2392620Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:18.2392840Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:18.2392912Z res = mod(**inputs) 2025-12-04T08:58:18.2393213Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T08:58:18.2393322Z outputs = self.roberta( 2025-12-04T08:58:18.2393606Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T08:58:18.2393693Z encoder_outputs = self.encoder( 2025-12-04T08:58:18.2393975Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T08:58:18.2394081Z layer_outputs = layer_module( 2025-12-04T08:58:18.2394332Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:18.2394418Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:18.2394700Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2394774Z return func(*args, **kwargs) 2025-12-04T08:58:18.2395065Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T08:58:18.2395164Z self_attention_outputs = self.attention( 2025-12-04T08:58:18.2395435Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2395516Z return func(*args, **kwargs) 2025-12-04T08:58:18.2395806Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 440, in forward 2025-12-04T08:58:18.2395888Z self_outputs = self.self( 2025-12-04T08:58:18.2396166Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2396240Z return func(*args, **kwargs) 2025-12-04T08:58:18.2396531Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 342, in forward 2025-12-04T08:58:18.2396619Z self.value(current_states) 2025-12-04T08:58:18.2396623Z 2025-12-04T08:58:18.2396712Z cudagraph partition due to non gpu ops 2025-12-04T08:58:18.2396832Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:18.2397052Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:18.2397124Z res = mod(**inputs) 2025-12-04T08:58:18.2397425Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T08:58:18.2397503Z outputs = self.roberta( 2025-12-04T08:58:18.2397798Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T08:58:18.2397905Z encoder_outputs = self.encoder( 2025-12-04T08:58:18.2398420Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T08:58:18.2398591Z layer_outputs = layer_module( 2025-12-04T08:58:18.2398852Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:18.2398941Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:18.2399235Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2399323Z return func(*args, **kwargs) 2025-12-04T08:58:18.2399636Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T08:58:18.2399727Z self_attention_outputs = self.attention( 2025-12-04T08:58:18.2399995Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2400079Z return func(*args, **kwargs) 2025-12-04T08:58:18.2400389Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 440, in forward 2025-12-04T08:58:18.2400468Z self_outputs = self.self( 2025-12-04T08:58:18.2400742Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2400816Z return func(*args, **kwargs) 2025-12-04T08:58:18.2401113Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 363, in forward 2025-12-04T08:58:18.2401278Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T08:58:18.2401284Z 2025-12-04T08:58:18.2401396Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:18.2401628Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:18.2401698Z res = mod(**inputs) 2025-12-04T08:58:18.2402009Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T08:58:18.2402083Z outputs = self.roberta( 2025-12-04T08:58:18.2402372Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T08:58:18.2402459Z encoder_outputs = self.encoder( 2025-12-04T08:58:18.2402748Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T08:58:18.2402823Z layer_outputs = layer_module( 2025-12-04T08:58:18.2403062Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:18.2403142Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:18.2403405Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2403477Z return func(*args, **kwargs) 2025-12-04T08:58:18.2403748Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T08:58:18.2403842Z self_attention_outputs = self.attention( 2025-12-04T08:58:18.2404095Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2404165Z return func(*args, **kwargs) 2025-12-04T08:58:18.2404446Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 449, in forward 2025-12-04T08:58:18.2404581Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T08:58:18.2404878Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 387, in forward 2025-12-04T08:58:18.2404966Z hidden_states = self.dense(hidden_states) 2025-12-04T08:58:18.2404969Z 2025-12-04T08:58:18.2405073Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:18.2405321Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:18.2405386Z res = mod(**inputs) 2025-12-04T08:58:18.2405662Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T08:58:18.2405732Z outputs = self.roberta( 2025-12-04T08:58:18.2406001Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T08:58:18.2406085Z encoder_outputs = self.encoder( 2025-12-04T08:58:18.2406370Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T08:58:18.2406447Z layer_outputs = layer_module( 2025-12-04T08:58:18.2406681Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:18.2406760Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:18.2407013Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2407083Z return func(*args, **kwargs) 2025-12-04T08:58:18.2407349Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 543, in forward 2025-12-04T08:58:18.2407441Z layer_output = apply_chunking_to_forward( 2025-12-04T08:58:18.2407727Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:58:18.2407811Z return forward_fn(*input_tensors) 2025-12-04T08:58:18.2408126Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 551, in feed_forward_chunk 2025-12-04T08:58:18.2408258Z intermediate_output = self.intermediate(attention_output) 2025-12-04T08:58:18.2408557Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 465, in forward 2025-12-04T08:58:18.2408645Z hidden_states = self.dense(hidden_states) 2025-12-04T08:58:18.2408649Z 2025-12-04T08:58:18.2408768Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:18.2408983Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:18.2409052Z res = mod(**inputs) 2025-12-04T08:58:18.2409344Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T08:58:18.2409417Z outputs = self.roberta( 2025-12-04T08:58:18.2409700Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T08:58:18.2409789Z encoder_outputs = self.encoder( 2025-12-04T08:58:18.2410076Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T08:58:18.2410159Z layer_outputs = layer_module( 2025-12-04T08:58:18.2410387Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:18.2410466Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:18.2410719Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2410792Z return func(*args, **kwargs) 2025-12-04T08:58:18.2411059Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 543, in forward 2025-12-04T08:58:18.2411153Z layer_output = apply_chunking_to_forward( 2025-12-04T08:58:18.2411479Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:58:18.2411566Z return forward_fn(*input_tensors) 2025-12-04T08:58:18.2411896Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 551, in feed_forward_chunk 2025-12-04T08:58:18.2412020Z intermediate_output = self.intermediate(attention_output) 2025-12-04T08:58:18.2412298Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-12-04T08:58:18.2412413Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:58:18.2412661Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T08:58:18.2412735Z return self.act(input) 2025-12-04T08:58:18.2412739Z 2025-12-04T08:58:18.2412844Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:18.2413061Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:18.2413127Z res = mod(**inputs) 2025-12-04T08:58:18.2413396Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T08:58:18.2413473Z outputs = self.roberta( 2025-12-04T08:58:18.2413740Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T08:58:18.2413823Z encoder_outputs = self.encoder( 2025-12-04T08:58:18.2414091Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T08:58:18.2414187Z layer_outputs = layer_module( 2025-12-04T08:58:18.2414423Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:18.2414505Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:18.2414757Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2414838Z return func(*args, **kwargs) 2025-12-04T08:58:18.2415104Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 543, in forward 2025-12-04T08:58:18.2415197Z layer_output = apply_chunking_to_forward( 2025-12-04T08:58:18.2415465Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:58:18.2415543Z return forward_fn(*input_tensors) 2025-12-04T08:58:18.2415860Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 552, in feed_forward_chunk 2025-12-04T08:58:18.2415994Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T08:58:18.2416270Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 479, in forward 2025-12-04T08:58:18.2416354Z hidden_states = self.dense(hidden_states) 2025-12-04T08:58:18.2416358Z 2025-12-04T08:58:18.2416463Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:18.2416676Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:18.2416742Z res = mod(**inputs) 2025-12-04T08:58:18.2417014Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T08:58:18.2417084Z outputs = self.roberta( 2025-12-04T08:58:18.2417356Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T08:58:18.2417439Z encoder_outputs = self.encoder( 2025-12-04T08:58:18.2417731Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T08:58:18.2417806Z layer_outputs = layer_module( 2025-12-04T08:58:18.2418041Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:18.2418143Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:18.2418401Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2418473Z return func(*args, **kwargs) 2025-12-04T08:58:18.2418741Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T08:58:18.2418834Z self_attention_outputs = self.attention( 2025-12-04T08:58:18.2419103Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2419176Z return func(*args, **kwargs) 2025-12-04T08:58:18.2419458Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 440, in forward 2025-12-04T08:58:18.2419530Z self_outputs = self.self( 2025-12-04T08:58:18.2419791Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2419862Z return func(*args, **kwargs) 2025-12-04T08:58:18.2420131Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 313, in forward 2025-12-04T08:58:18.2420355Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-12-04T08:58:18.2420377Z 2025-12-04T08:58:18.2420489Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:18.2421047Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:18.2421136Z res = mod(**inputs) 2025-12-04T08:58:18.2421410Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T08:58:18.2421492Z outputs = self.roberta( 2025-12-04T08:58:18.2421759Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T08:58:18.2421834Z encoder_outputs = self.encoder( 2025-12-04T08:58:18.2422107Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T08:58:18.2422179Z layer_outputs = layer_module( 2025-12-04T08:58:18.2422416Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:18.2422497Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:18.2422746Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2422826Z return func(*args, **kwargs) 2025-12-04T08:58:18.2423092Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T08:58:18.2423186Z self_attention_outputs = self.attention( 2025-12-04T08:58:18.2423432Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2423503Z return func(*args, **kwargs) 2025-12-04T08:58:18.2423780Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 440, in forward 2025-12-04T08:58:18.2423857Z self_outputs = self.self( 2025-12-04T08:58:18.2424107Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2424187Z return func(*args, **kwargs) 2025-12-04T08:58:18.2424496Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 337, in forward 2025-12-04T08:58:18.2424576Z self.key(current_states) 2025-12-04T08:58:18.2424580Z 2025-12-04T08:58:18.2424683Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:18.2424919Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:18.2424991Z res = mod(**inputs) 2025-12-04T08:58:18.2425258Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T08:58:18.2425330Z outputs = self.roberta( 2025-12-04T08:58:18.2425606Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T08:58:18.2425704Z encoder_outputs = self.encoder( 2025-12-04T08:58:18.2425983Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T08:58:18.2426057Z layer_outputs = layer_module( 2025-12-04T08:58:18.2426287Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:18.2426375Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:18.2426625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2426703Z return func(*args, **kwargs) 2025-12-04T08:58:18.2426968Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T08:58:18.2427086Z self_attention_outputs = self.attention( 2025-12-04T08:58:18.2427338Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2427406Z return func(*args, **kwargs) 2025-12-04T08:58:18.2427668Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 440, in forward 2025-12-04T08:58:18.2427749Z self_outputs = self.self( 2025-12-04T08:58:18.2427999Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2428078Z return func(*args, **kwargs) 2025-12-04T08:58:18.2428349Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 342, in forward 2025-12-04T08:58:18.2428422Z self.value(current_states) 2025-12-04T08:58:18.2428426Z 2025-12-04T08:58:18.2428518Z cudagraph partition due to non gpu ops 2025-12-04T08:58:18.2428625Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:18.2428831Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:18.2428906Z res = mod(**inputs) 2025-12-04T08:58:18.2429183Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T08:58:18.2429260Z outputs = self.roberta( 2025-12-04T08:58:18.2429532Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T08:58:18.2429606Z encoder_outputs = self.encoder( 2025-12-04T08:58:18.2429875Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T08:58:18.2429945Z layer_outputs = layer_module( 2025-12-04T08:58:18.2430176Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:18.2430256Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:18.2430501Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2430595Z return func(*args, **kwargs) 2025-12-04T08:58:18.2430865Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T08:58:18.2430948Z self_attention_outputs = self.attention( 2025-12-04T08:58:18.2431225Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2431295Z return func(*args, **kwargs) 2025-12-04T08:58:18.2431578Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 440, in forward 2025-12-04T08:58:18.2431649Z self_outputs = self.self( 2025-12-04T08:58:18.2431894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2431986Z return func(*args, **kwargs) 2025-12-04T08:58:18.2432250Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 363, in forward 2025-12-04T08:58:18.2432384Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T08:58:18.2432397Z 2025-12-04T08:58:18.2432500Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:18.2432704Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:18.2432777Z res = mod(**inputs) 2025-12-04T08:58:18.2433045Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T08:58:18.2433115Z outputs = self.roberta( 2025-12-04T08:58:18.2433390Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T08:58:18.2433485Z encoder_outputs = self.encoder( 2025-12-04T08:58:18.2433763Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T08:58:18.2433835Z layer_outputs = layer_module( 2025-12-04T08:58:18.2434061Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:18.2434150Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:18.2434405Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2434478Z return func(*args, **kwargs) 2025-12-04T08:58:18.2434769Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T08:58:18.2434858Z self_attention_outputs = self.attention( 2025-12-04T08:58:18.2435132Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2435207Z return func(*args, **kwargs) 2025-12-04T08:58:18.2435491Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 449, in forward 2025-12-04T08:58:18.2435639Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T08:58:18.2435925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 387, in forward 2025-12-04T08:58:18.2436026Z hidden_states = self.dense(hidden_states) 2025-12-04T08:58:18.2436030Z 2025-12-04T08:58:18.2436142Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:18.2436357Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:18.2436437Z res = mod(**inputs) 2025-12-04T08:58:18.2436725Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T08:58:18.2436799Z outputs = self.roberta( 2025-12-04T08:58:18.2437106Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T08:58:18.2437187Z encoder_outputs = self.encoder( 2025-12-04T08:58:18.2437480Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T08:58:18.2437576Z layer_outputs = layer_module( 2025-12-04T08:58:18.2437817Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:18.2437908Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:18.2438243Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2438326Z return func(*args, **kwargs) 2025-12-04T08:58:18.2438636Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 543, in forward 2025-12-04T08:58:18.2438730Z layer_output = apply_chunking_to_forward( 2025-12-04T08:58:18.2439031Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:58:18.2439117Z return forward_fn(*input_tensors) 2025-12-04T08:58:18.2439459Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 551, in feed_forward_chunk 2025-12-04T08:58:18.2439598Z intermediate_output = self.intermediate(attention_output) 2025-12-04T08:58:18.2439884Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 465, in forward 2025-12-04T08:58:18.2439980Z hidden_states = self.dense(hidden_states) 2025-12-04T08:58:18.2440015Z 2025-12-04T08:58:18.2440130Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:18.2440346Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:18.2440426Z res = mod(**inputs) 2025-12-04T08:58:18.2440722Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T08:58:18.2440799Z outputs = self.roberta( 2025-12-04T08:58:18.2441069Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T08:58:18.2441145Z encoder_outputs = self.encoder( 2025-12-04T08:58:18.2441422Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T08:58:18.2441494Z layer_outputs = layer_module( 2025-12-04T08:58:18.2441724Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:18.2441813Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:18.2442062Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2442140Z return func(*args, **kwargs) 2025-12-04T08:58:18.2442406Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 543, in forward 2025-12-04T08:58:18.2442492Z layer_output = apply_chunking_to_forward( 2025-12-04T08:58:18.2442765Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:58:18.2442843Z return forward_fn(*input_tensors) 2025-12-04T08:58:18.2443145Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 551, in feed_forward_chunk 2025-12-04T08:58:18.2443277Z intermediate_output = self.intermediate(attention_output) 2025-12-04T08:58:18.2443548Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-12-04T08:58:18.2443691Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:58:18.2443916Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T08:58:18.2443988Z return self.act(input) 2025-12-04T08:58:18.2444011Z 2025-12-04T08:58:18.2444126Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:18.2444332Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:18.2444403Z res = mod(**inputs) 2025-12-04T08:58:18.2444673Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T08:58:18.2444745Z outputs = self.roberta( 2025-12-04T08:58:18.2445036Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T08:58:18.2445113Z encoder_outputs = self.encoder( 2025-12-04T08:58:18.2445380Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T08:58:18.2445461Z layer_outputs = layer_module( 2025-12-04T08:58:18.2445696Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:18.2445786Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:18.2446037Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2446108Z return func(*args, **kwargs) 2025-12-04T08:58:18.2446384Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 543, in forward 2025-12-04T08:58:18.2446490Z layer_output = apply_chunking_to_forward( 2025-12-04T08:58:18.2446765Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:58:18.2446845Z return forward_fn(*input_tensors) 2025-12-04T08:58:18.2447146Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 552, in feed_forward_chunk 2025-12-04T08:58:18.2447290Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T08:58:18.2447560Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 479, in forward 2025-12-04T08:58:18.2447642Z hidden_states = self.dense(hidden_states) 2025-12-04T08:58:18.2447654Z 2025-12-04T08:58:18.2447758Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:18.2447961Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:18.2448037Z res = mod(**inputs) 2025-12-04T08:58:18.2448306Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T08:58:18.2448378Z outputs = self.roberta( 2025-12-04T08:58:18.2448652Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T08:58:18.2448726Z encoder_outputs = self.encoder( 2025-12-04T08:58:18.2449000Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T08:58:18.2449072Z layer_outputs = layer_module( 2025-12-04T08:58:18.2449298Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:18.2449383Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:18.2449637Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2449708Z return func(*args, **kwargs) 2025-12-04T08:58:18.2449996Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T08:58:18.2450083Z self_attention_outputs = self.attention( 2025-12-04T08:58:18.2450340Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2450434Z return func(*args, **kwargs) 2025-12-04T08:58:18.2450705Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 440, in forward 2025-12-04T08:58:18.2450787Z self_outputs = self.self( 2025-12-04T08:58:18.2451038Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2451109Z return func(*args, **kwargs) 2025-12-04T08:58:18.2451401Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 313, in forward 2025-12-04T08:58:18.2451623Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-12-04T08:58:18.2451626Z 2025-12-04T08:58:18.2451739Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:18.2451946Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:18.2452013Z res = mod(**inputs) 2025-12-04T08:58:18.2452290Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T08:58:18.2452360Z outputs = self.roberta( 2025-12-04T08:58:18.2452635Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T08:58:18.2452729Z encoder_outputs = self.encoder( 2025-12-04T08:58:18.2453001Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T08:58:18.2453082Z layer_outputs = layer_module( 2025-12-04T08:58:18.2453315Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:18.2453395Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:18.2453656Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2453727Z return func(*args, **kwargs) 2025-12-04T08:58:18.2454037Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T08:58:18.2454121Z self_attention_outputs = self.attention( 2025-12-04T08:58:18.2454372Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2454452Z return func(*args, **kwargs) 2025-12-04T08:58:18.2454721Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 440, in forward 2025-12-04T08:58:18.2454801Z self_outputs = self.self( 2025-12-04T08:58:18.2455051Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2455122Z return func(*args, **kwargs) 2025-12-04T08:58:18.2455396Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 337, in forward 2025-12-04T08:58:18.2455468Z self.key(current_states) 2025-12-04T08:58:18.2455474Z 2025-12-04T08:58:18.2455578Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:18.2455790Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:18.2455857Z res = mod(**inputs) 2025-12-04T08:58:18.2456134Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T08:58:18.2456204Z outputs = self.roberta( 2025-12-04T08:58:18.2456486Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T08:58:18.2456568Z encoder_outputs = self.encoder( 2025-12-04T08:58:18.2456854Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T08:58:18.2456927Z layer_outputs = layer_module( 2025-12-04T08:58:18.2457163Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:18.2457243Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:18.2457500Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2457593Z return func(*args, **kwargs) 2025-12-04T08:58:18.2457863Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T08:58:18.2457956Z self_attention_outputs = self.attention( 2025-12-04T08:58:18.2458210Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2458286Z return func(*args, **kwargs) 2025-12-04T08:58:18.2458548Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 440, in forward 2025-12-04T08:58:18.2458619Z self_outputs = self.self( 2025-12-04T08:58:18.2458874Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2458943Z return func(*args, **kwargs) 2025-12-04T08:58:18.2459234Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 342, in forward 2025-12-04T08:58:18.2459327Z self.value(current_states) 2025-12-04T08:58:18.2459334Z 2025-12-04T08:58:18.2459419Z cudagraph partition due to non gpu ops 2025-12-04T08:58:18.2459528Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:18.2459727Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:18.2459794Z res = mod(**inputs) 2025-12-04T08:58:18.2460063Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T08:58:18.2460132Z outputs = self.roberta( 2025-12-04T08:58:18.2460394Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T08:58:18.2460474Z encoder_outputs = self.encoder( 2025-12-04T08:58:18.2460738Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T08:58:18.2460817Z layer_outputs = layer_module( 2025-12-04T08:58:18.2461040Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:18.2461119Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:18.2461373Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2461443Z return func(*args, **kwargs) 2025-12-04T08:58:18.2461711Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T08:58:18.2461793Z self_attention_outputs = self.attention( 2025-12-04T08:58:18.2462035Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2462112Z return func(*args, **kwargs) 2025-12-04T08:58:18.2462375Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 440, in forward 2025-12-04T08:58:18.2462444Z self_outputs = self.self( 2025-12-04T08:58:18.2462712Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2462782Z return func(*args, **kwargs) 2025-12-04T08:58:18.2463067Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 363, in forward 2025-12-04T08:58:18.2463199Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-12-04T08:58:18.2463202Z 2025-12-04T08:58:18.2463305Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:18.2463509Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:18.2463576Z res = mod(**inputs) 2025-12-04T08:58:18.2463870Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T08:58:18.2463939Z outputs = self.roberta( 2025-12-04T08:58:18.2464212Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T08:58:18.2464292Z encoder_outputs = self.encoder( 2025-12-04T08:58:18.2464553Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T08:58:18.2464623Z layer_outputs = layer_module( 2025-12-04T08:58:18.2464856Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:18.2464933Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:18.2465183Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2465273Z return func(*args, **kwargs) 2025-12-04T08:58:18.2465534Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 513, in forward 2025-12-04T08:58:18.2465627Z self_attention_outputs = self.attention( 2025-12-04T08:58:18.2465868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2465936Z return func(*args, **kwargs) 2025-12-04T08:58:18.2466202Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 449, in forward 2025-12-04T08:58:18.2466331Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T08:58:18.2466605Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 387, in forward 2025-12-04T08:58:18.2466690Z hidden_states = self.dense(hidden_states) 2025-12-04T08:58:18.2466694Z 2025-12-04T08:58:18.2466800Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:18.2467013Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:18.2467090Z res = mod(**inputs) 2025-12-04T08:58:18.2467358Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T08:58:18.2467428Z outputs = self.roberta( 2025-12-04T08:58:18.2467689Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T08:58:18.2467770Z encoder_outputs = self.encoder( 2025-12-04T08:58:18.2468029Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T08:58:18.2468101Z layer_outputs = layer_module( 2025-12-04T08:58:18.2468337Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:18.2468419Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:18.2468692Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2468764Z return func(*args, **kwargs) 2025-12-04T08:58:18.2469033Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 543, in forward 2025-12-04T08:58:18.2469143Z layer_output = apply_chunking_to_forward( 2025-12-04T08:58:18.2469408Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:58:18.2469494Z return forward_fn(*input_tensors) 2025-12-04T08:58:18.2469799Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 551, in feed_forward_chunk 2025-12-04T08:58:18.2469923Z intermediate_output = self.intermediate(attention_output) 2025-12-04T08:58:18.2470224Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 465, in forward 2025-12-04T08:58:18.2470310Z hidden_states = self.dense(hidden_states) 2025-12-04T08:58:18.2470314Z 2025-12-04T08:58:18.2470418Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:18.2470629Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:18.2470697Z res = mod(**inputs) 2025-12-04T08:58:18.2470968Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T08:58:18.2471039Z outputs = self.roberta( 2025-12-04T08:58:18.2471303Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T08:58:18.2471405Z encoder_outputs = self.encoder( 2025-12-04T08:58:18.2471676Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T08:58:18.2471755Z layer_outputs = layer_module( 2025-12-04T08:58:18.2471994Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:18.2472072Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:18.2472326Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2472395Z return func(*args, **kwargs) 2025-12-04T08:58:18.2472657Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 543, in forward 2025-12-04T08:58:18.2472745Z layer_output = apply_chunking_to_forward( 2025-12-04T08:58:18.2473006Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:58:18.2473091Z return forward_fn(*input_tensors) 2025-12-04T08:58:18.2473396Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 551, in feed_forward_chunk 2025-12-04T08:58:18.2473519Z intermediate_output = self.intermediate(attention_output) 2025-12-04T08:58:18.2473799Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-12-04T08:58:18.2473915Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T08:58:18.2474145Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T08:58:18.2474217Z return self.act(input) 2025-12-04T08:58:18.2474221Z 2025-12-04T08:58:18.2474328Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:18.2474543Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:18.2474613Z res = mod(**inputs) 2025-12-04T08:58:18.2474894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward 2025-12-04T08:58:18.2474993Z outputs = self.roberta( 2025-12-04T08:58:18.2475280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 862, in forward 2025-12-04T08:58:18.2475386Z encoder_outputs = self.encoder( 2025-12-04T08:58:18.2475670Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 606, in forward 2025-12-04T08:58:18.2475747Z layer_outputs = layer_module( 2025-12-04T08:58:18.2476000Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:18.2476086Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:18.2476382Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:18.2476462Z return func(*args, **kwargs) 2025-12-04T08:58:18.2476732Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 543, in forward 2025-12-04T08:58:18.2476824Z layer_output = apply_chunking_to_forward( 2025-12-04T08:58:18.2477092Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T08:58:18.2477171Z return forward_fn(*input_tensors) 2025-12-04T08:58:18.2477483Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 552, in feed_forward_chunk 2025-12-04T08:58:18.2477617Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T08:58:18.2477896Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 479, in forward 2025-12-04T08:58:18.2478001Z hidden_states = self.dense(hidden_states) 2025-12-04T08:58:18.2478005Z 2025-12-04T08:58:18.2478192Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:18.2478441Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:18.2478514Z res = mod(**inputs) 2025-12-04T08:58:18.2478808Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 988, in forward 2025-12-04T08:58:18.2478930Z prediction_scores = self.lm_head(sequence_output) 2025-12-04T08:58:18.2479235Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 1121, in forward 2025-12-04T08:58:18.2479322Z x = self.dense(features) 2025-12-04T08:58:18.2479327Z 2025-12-04T08:58:18.2479451Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:18.2479670Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:18.2479753Z res = mod(**inputs) 2025-12-04T08:58:18.2480044Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 988, in forward 2025-12-04T08:58:18.2480160Z prediction_scores = self.lm_head(sequence_output) 2025-12-04T08:58:18.2480456Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 1126, in forward 2025-12-04T08:58:18.2480530Z x = self.decoder(x) 2025-12-04T08:58:18.2480534Z 2025-12-04T08:58:18.2480651Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:18.2480871Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:18.2480942Z res = mod(**inputs) 2025-12-04T08:58:18.2481235Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 994, in forward 2025-12-04T08:58:18.2481318Z lm_loss = self.loss_function( 2025-12-04T08:58:18.2481594Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/loss/loss_utils.py", line 67, in ForCausalLMLoss 2025-12-04T08:58:18.2481813Z loss = fixed_cross_entropy(logits, shift_labels, num_items_in_batch, ignore_index, **kwargs) 2025-12-04T08:58:18.2482093Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/loss/loss_utils.py", line 36, in fixed_cross_entropy 2025-12-04T08:58:18.2482334Z loss = nn.functional.cross_entropy(source, target, ignore_index=ignore_index, reduction=reduction) 2025-12-04T08:58:18.2482339Z 2025-12-04T08:58:29.1256191Z Compilation time (from dynamo_timed): 17.13489116 2025-12-04T08:58:29.1401950Z pass 2025-12-04T08:58:29.1402495Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T08:58:29.1409700Z TIMING: _recursive_pre_grad_passes:0.00675 _recursive_joint_graph_passes:0.66949 _recursive_post_grad_passes:0.06803 async_compile.wait:0.84263 code_gen:9.79628 inductor_compile:11.11266 backend_compile:14.24845 gc:0.00138 entire_frame_compile:17.13489 total_wall_time:17.13489 2025-12-04T08:58:29.1410971Z STATS: call_* op count: 303 | FakeTensorMode.__torch_dispatch__:7229 | FakeTensor.__torch_dispatch__:4402 | ProxyTorchDispatchMode.__torch_dispatch__:1992 2025-12-04T08:58:29.1411524Z Dynamo produced 1 graphs covering 303 ops with 0 graph breaks (0 unique) 2025-12-04T08:58:31.5566947Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/__init__.py:65: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-12-04T08:58:31.5567774Z import pynvml # type: ignore[import] 2025-12-04T08:58:35.0355166Z 2025-12-04T08:58:36.3118536Z loading model: 0it [00:00, ?it/s] 2025-12-04T08:58:36.3119292Z loading model: 0it [00:01, ?it/s] 2025-12-04T08:58:36.3129545Z cpu eval T5ForConditionalGeneration 2025-12-04T08:58:38.9990234Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T08:58:40.0268588Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T08:58:41.1362709Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T08:58:49.6747484Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:49.6748008Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:49.6748410Z res = mod(**inputs) 2025-12-04T08:58:49.6748826Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T08:58:49.6749217Z decoder_outputs = self.decoder( 2025-12-04T08:58:49.6749642Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T08:58:49.6750027Z layer_outputs = layer_module( 2025-12-04T08:58:49.6750395Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:49.6750768Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:49.6751159Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.6751550Z return func(*args, **kwargs) 2025-12-04T08:58:49.6751917Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T08:58:49.6752308Z self_attention_outputs = self.layer[0]( 2025-12-04T08:58:49.6752696Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.6753080Z return func(*args, **kwargs) 2025-12-04T08:58:49.6753451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 603, in forward 2025-12-04T08:58:49.6753847Z attention_output = self.SelfAttention( 2025-12-04T08:58:49.6754574Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.6754993Z return func(*args, **kwargs) 2025-12-04T08:58:49.6755395Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 549, in forward 2025-12-04T08:58:49.6755905Z position_bias = position_bias + causal_mask 2025-12-04T08:58:49.6756067Z 2025-12-04T08:58:49.6756196Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:49.6756595Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:49.6756954Z res = mod(**inputs) 2025-12-04T08:58:49.6757409Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1727, in forward 2025-12-04T08:58:49.6757829Z encoder_outputs = self.encoder( 2025-12-04T08:58:49.6758575Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T08:58:49.6759013Z layer_outputs = layer_module( 2025-12-04T08:58:49.6759415Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:49.6759831Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:49.6760249Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.6760637Z return func(*args, **kwargs) 2025-12-04T08:58:49.6761011Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T08:58:49.6761456Z self_attention_outputs = self.layer[0]( 2025-12-04T08:58:49.6761865Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.6762312Z return func(*args, **kwargs) 2025-12-04T08:58:49.6762667Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 603, in forward 2025-12-04T08:58:49.6763078Z attention_output = self.SelfAttention( 2025-12-04T08:58:49.6763476Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.6763862Z return func(*args, **kwargs) 2025-12-04T08:58:49.6764226Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 492, in forward 2025-12-04T08:58:49.6764642Z query_states = self.q(hidden_states) 2025-12-04T08:58:49.6764803Z 2025-12-04T08:58:49.6764919Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:49.6765296Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:49.6765654Z res = mod(**inputs) 2025-12-04T08:58:49.6766011Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1727, in forward 2025-12-04T08:58:49.6766421Z encoder_outputs = self.encoder( 2025-12-04T08:58:49.6766825Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T08:58:49.6767212Z layer_outputs = layer_module( 2025-12-04T08:58:49.6767574Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:49.6767945Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:49.6768338Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.6768725Z return func(*args, **kwargs) 2025-12-04T08:58:49.6769123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T08:58:49.6769508Z self_attention_outputs = self.layer[0]( 2025-12-04T08:58:49.6769925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.6770315Z return func(*args, **kwargs) 2025-12-04T08:58:49.6770678Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 603, in forward 2025-12-04T08:58:49.6771101Z attention_output = self.SelfAttention( 2025-12-04T08:58:49.6771500Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.6771889Z return func(*args, **kwargs) 2025-12-04T08:58:49.6772250Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 513, in forward 2025-12-04T08:58:49.6772645Z key_states = self.k(current_states) 2025-12-04T08:58:49.6772782Z 2025-12-04T08:58:49.6772919Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:49.6773294Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:49.6773634Z res = mod(**inputs) 2025-12-04T08:58:49.6773994Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1727, in forward 2025-12-04T08:58:49.6774383Z encoder_outputs = self.encoder( 2025-12-04T08:58:49.6774760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T08:58:49.6775146Z layer_outputs = layer_module( 2025-12-04T08:58:49.6775511Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:49.6775884Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:49.6776296Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.6776683Z return func(*args, **kwargs) 2025-12-04T08:58:49.6777055Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T08:58:49.6777440Z self_attention_outputs = self.layer[0]( 2025-12-04T08:58:49.6777838Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.6778226Z return func(*args, **kwargs) 2025-12-04T08:58:49.6778600Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 603, in forward 2025-12-04T08:58:49.6778988Z attention_output = self.SelfAttention( 2025-12-04T08:58:49.6779386Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.6779765Z return func(*args, **kwargs) 2025-12-04T08:58:49.6780120Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 514, in forward 2025-12-04T08:58:49.6780497Z value_states = self.v(current_states) 2025-12-04T08:58:49.6780641Z 2025-12-04T08:58:49.6780725Z cudagraph partition due to non gpu ops 2025-12-04T08:58:49.6780946Z cudagraph partition due to non gpu ops 2025-12-04T08:58:49.6781180Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:49.6781545Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:49.6781873Z res = mod(**inputs) 2025-12-04T08:58:49.6782216Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1727, in forward 2025-12-04T08:58:49.6782593Z encoder_outputs = self.encoder( 2025-12-04T08:58:49.6782962Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T08:58:49.6783338Z layer_outputs = layer_module( 2025-12-04T08:58:49.6783682Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:49.6784068Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:49.6784461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.6784893Z return func(*args, **kwargs) 2025-12-04T08:58:49.6785261Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T08:58:49.6785649Z self_attention_outputs = self.layer[0]( 2025-12-04T08:58:49.6786050Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.6786416Z return func(*args, **kwargs) 2025-12-04T08:58:49.6786791Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 603, in forward 2025-12-04T08:58:49.6787183Z attention_output = self.SelfAttention( 2025-12-04T08:58:49.6787581Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.6787958Z return func(*args, **kwargs) 2025-12-04T08:58:49.6788326Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 572, in forward 2025-12-04T08:58:49.6788715Z attn_output = self.o(attn_output) 2025-12-04T08:58:49.6788850Z 2025-12-04T08:58:49.6788958Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:49.6789334Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:49.6789663Z res = mod(**inputs) 2025-12-04T08:58:49.6790021Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1727, in forward 2025-12-04T08:58:49.6790411Z encoder_outputs = self.encoder( 2025-12-04T08:58:49.6790795Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T08:58:49.6791198Z layer_outputs = layer_module( 2025-12-04T08:58:49.6791567Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:49.6791963Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:49.6792381Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.6792796Z return func(*args, **kwargs) 2025-12-04T08:58:49.6793179Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 737, in forward 2025-12-04T08:58:49.6793612Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T08:58:49.6794037Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-12-04T08:58:49.6794459Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T08:58:49.6794882Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward 2025-12-04T08:58:49.6795267Z hidden_states = self.wi(hidden_states) 2025-12-04T08:58:49.6795404Z 2025-12-04T08:58:49.6795520Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:49.6795887Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:49.6796220Z res = mod(**inputs) 2025-12-04T08:58:49.6796576Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1727, in forward 2025-12-04T08:58:49.6796971Z encoder_outputs = self.encoder( 2025-12-04T08:58:49.6797364Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T08:58:49.6797774Z layer_outputs = layer_module( 2025-12-04T08:58:49.6798255Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:49.6798710Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:49.6799145Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.6799598Z return func(*args, **kwargs) 2025-12-04T08:58:49.6799969Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 737, in forward 2025-12-04T08:58:49.6800367Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T08:58:49.6800766Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-12-04T08:58:49.6801200Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T08:58:49.6801654Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 289, in forward 2025-12-04T08:58:49.6802055Z hidden_states = self.act(hidden_states) 2025-12-04T08:58:49.6802202Z 2025-12-04T08:58:49.6802312Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:49.6802687Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:49.6803015Z res = mod(**inputs) 2025-12-04T08:58:49.6803381Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1727, in forward 2025-12-04T08:58:49.6803769Z encoder_outputs = self.encoder( 2025-12-04T08:58:49.6804152Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T08:58:49.6804531Z layer_outputs = layer_module( 2025-12-04T08:58:49.6804891Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:49.6805295Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:49.6805683Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.6806133Z return func(*args, **kwargs) 2025-12-04T08:58:49.6806502Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 737, in forward 2025-12-04T08:58:49.6806904Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T08:58:49.6807296Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-12-04T08:58:49.6807764Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T08:58:49.6808182Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 297, in forward 2025-12-04T08:58:49.6808572Z hidden_states = self.wo(hidden_states) 2025-12-04T08:58:49.6808710Z 2025-12-04T08:58:49.6808817Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:49.6809183Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:49.6809515Z res = mod(**inputs) 2025-12-04T08:58:49.6809894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1727, in forward 2025-12-04T08:58:49.6810337Z encoder_outputs = self.encoder( 2025-12-04T08:58:49.6810715Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T08:58:49.6811101Z layer_outputs = layer_module( 2025-12-04T08:58:49.6811452Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:49.6811831Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:49.6812216Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.6812586Z return func(*args, **kwargs) 2025-12-04T08:58:49.6812969Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T08:58:49.6813361Z self_attention_outputs = self.layer[0]( 2025-12-04T08:58:49.6813759Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.6814160Z return func(*args, **kwargs) 2025-12-04T08:58:49.6814530Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 603, in forward 2025-12-04T08:58:49.6814925Z attention_output = self.SelfAttention( 2025-12-04T08:58:49.6815317Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.6815706Z return func(*args, **kwargs) 2025-12-04T08:58:49.6816093Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 492, in forward 2025-12-04T08:58:49.6816488Z query_states = self.q(hidden_states) 2025-12-04T08:58:49.6816636Z 2025-12-04T08:58:49.6816753Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:49.6817157Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:49.6817496Z res = mod(**inputs) 2025-12-04T08:58:49.6817854Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1727, in forward 2025-12-04T08:58:49.6818233Z encoder_outputs = self.encoder( 2025-12-04T08:58:49.6818610Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T08:58:49.6818999Z layer_outputs = layer_module( 2025-12-04T08:58:49.6819361Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:49.6819737Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:49.6820134Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.6820520Z return func(*args, **kwargs) 2025-12-04T08:58:49.6821141Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T08:58:49.6821550Z self_attention_outputs = self.layer[0]( 2025-12-04T08:58:49.6821939Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.6822305Z return func(*args, **kwargs) 2025-12-04T08:58:49.6822670Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 603, in forward 2025-12-04T08:58:49.6823065Z attention_output = self.SelfAttention( 2025-12-04T08:58:49.6823464Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.6823849Z return func(*args, **kwargs) 2025-12-04T08:58:49.6824224Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 513, in forward 2025-12-04T08:58:49.6824621Z key_states = self.k(current_states) 2025-12-04T08:58:49.6824755Z 2025-12-04T08:58:49.6824869Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:49.6825230Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:49.6825557Z res = mod(**inputs) 2025-12-04T08:58:49.6825907Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1727, in forward 2025-12-04T08:58:49.6826272Z encoder_outputs = self.encoder( 2025-12-04T08:58:49.6826641Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T08:58:49.6827017Z layer_outputs = layer_module( 2025-12-04T08:58:49.6827364Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:49.6827782Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:49.6828163Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.6828594Z return func(*args, **kwargs) 2025-12-04T08:58:49.6828954Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T08:58:49.6829349Z self_attention_outputs = self.layer[0]( 2025-12-04T08:58:49.6829743Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.6830124Z return func(*args, **kwargs) 2025-12-04T08:58:49.6830543Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 603, in forward 2025-12-04T08:58:49.6830984Z attention_output = self.SelfAttention( 2025-12-04T08:58:49.6831377Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.6831757Z return func(*args, **kwargs) 2025-12-04T08:58:49.6832123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 514, in forward 2025-12-04T08:58:49.6832514Z value_states = self.v(current_states) 2025-12-04T08:58:49.6832651Z 2025-12-04T08:58:49.6832743Z cudagraph partition due to non gpu ops 2025-12-04T08:58:49.6832960Z cudagraph partition due to non gpu ops 2025-12-04T08:58:49.6833209Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:49.6833582Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:49.6833937Z res = mod(**inputs) 2025-12-04T08:58:49.6834310Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1727, in forward 2025-12-04T08:58:49.6834718Z encoder_outputs = self.encoder( 2025-12-04T08:58:49.6835119Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T08:58:49.6835517Z layer_outputs = layer_module( 2025-12-04T08:58:49.6835897Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:49.6836296Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:49.6836705Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.6837126Z return func(*args, **kwargs) 2025-12-04T08:58:49.6837515Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T08:58:49.6837942Z self_attention_outputs = self.layer[0]( 2025-12-04T08:58:49.6838430Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.6838855Z return func(*args, **kwargs) 2025-12-04T08:58:49.6839253Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 603, in forward 2025-12-04T08:58:49.6839646Z attention_output = self.SelfAttention( 2025-12-04T08:58:49.6840035Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.6840422Z return func(*args, **kwargs) 2025-12-04T08:58:49.6840791Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 572, in forward 2025-12-04T08:58:49.6841179Z attn_output = self.o(attn_output) 2025-12-04T08:58:49.6841314Z 2025-12-04T08:58:49.6841421Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:49.6841795Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:49.6842129Z res = mod(**inputs) 2025-12-04T08:58:49.6842494Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1727, in forward 2025-12-04T08:58:49.6842881Z encoder_outputs = self.encoder( 2025-12-04T08:58:49.6843259Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T08:58:49.6843659Z layer_outputs = layer_module( 2025-12-04T08:58:49.6844010Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:49.6844388Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:49.6844780Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.6845162Z return func(*args, **kwargs) 2025-12-04T08:58:49.6845566Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 737, in forward 2025-12-04T08:58:49.6845972Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T08:58:49.6846426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-12-04T08:58:49.6846834Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T08:58:49.6847237Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward 2025-12-04T08:58:49.6847609Z hidden_states = self.wi(hidden_states) 2025-12-04T08:58:49.6847740Z 2025-12-04T08:58:49.6847848Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:49.6848201Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:49.6848538Z res = mod(**inputs) 2025-12-04T08:58:49.6848880Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1727, in forward 2025-12-04T08:58:49.6849240Z encoder_outputs = self.encoder( 2025-12-04T08:58:49.6849606Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T08:58:49.6849969Z layer_outputs = layer_module( 2025-12-04T08:58:49.6850321Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:49.6850692Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:49.6851068Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.6851436Z return func(*args, **kwargs) 2025-12-04T08:58:49.6851778Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 737, in forward 2025-12-04T08:58:49.6852172Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T08:58:49.6852567Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-12-04T08:58:49.6852972Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T08:58:49.6853364Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 289, in forward 2025-12-04T08:58:49.6853735Z hidden_states = self.act(hidden_states) 2025-12-04T08:58:49.6853869Z 2025-12-04T08:58:49.6853981Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:49.6854328Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:49.6854639Z res = mod(**inputs) 2025-12-04T08:58:49.6854975Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1727, in forward 2025-12-04T08:58:49.6855340Z encoder_outputs = self.encoder( 2025-12-04T08:58:49.6855694Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T08:58:49.6856056Z layer_outputs = layer_module( 2025-12-04T08:58:49.6856411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:49.6856777Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:49.6857191Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.6857568Z return func(*args, **kwargs) 2025-12-04T08:58:49.6857929Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 737, in forward 2025-12-04T08:58:49.6858318Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T08:58:49.6858712Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-12-04T08:58:49.6859152Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T08:58:49.6859564Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 297, in forward 2025-12-04T08:58:49.6859938Z hidden_states = self.wo(hidden_states) 2025-12-04T08:58:49.6860082Z 2025-12-04T08:58:49.6860188Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:49.6860566Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:49.6860905Z res = mod(**inputs) 2025-12-04T08:58:49.6861256Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1727, in forward 2025-12-04T08:58:49.6861647Z encoder_outputs = self.encoder( 2025-12-04T08:58:49.6862029Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T08:58:49.6862431Z layer_outputs = layer_module( 2025-12-04T08:58:49.6862796Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:49.6863179Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:49.6863575Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.6864012Z return func(*args, **kwargs) 2025-12-04T08:58:49.6864388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T08:58:49.6864788Z self_attention_outputs = self.layer[0]( 2025-12-04T08:58:49.6865180Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.6865573Z return func(*args, **kwargs) 2025-12-04T08:58:49.6865958Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 603, in forward 2025-12-04T08:58:49.6866381Z attention_output = self.SelfAttention( 2025-12-04T08:58:49.6866800Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.6867212Z return func(*args, **kwargs) 2025-12-04T08:58:49.6867607Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 492, in forward 2025-12-04T08:58:49.6868013Z query_states = self.q(hidden_states) 2025-12-04T08:58:49.6868159Z 2025-12-04T08:58:49.6868267Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:49.6868645Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:49.6868982Z res = mod(**inputs) 2025-12-04T08:58:49.6869333Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1727, in forward 2025-12-04T08:58:49.6869729Z encoder_outputs = self.encoder( 2025-12-04T08:58:49.6870136Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T08:58:49.6870561Z layer_outputs = layer_module( 2025-12-04T08:58:49.6870941Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:49.6871343Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:49.6871777Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.6872174Z return func(*args, **kwargs) 2025-12-04T08:58:49.6872565Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T08:58:49.6872979Z self_attention_outputs = self.layer[0]( 2025-12-04T08:58:49.6873394Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.6873808Z return func(*args, **kwargs) 2025-12-04T08:58:49.6874200Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 603, in forward 2025-12-04T08:58:49.6874614Z attention_output = self.SelfAttention( 2025-12-04T08:58:49.6875023Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.6875431Z return func(*args, **kwargs) 2025-12-04T08:58:49.6875817Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 513, in forward 2025-12-04T08:58:49.6876234Z key_states = self.k(current_states) 2025-12-04T08:58:49.6876381Z 2025-12-04T08:58:49.6876495Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:49.6876889Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:49.6877266Z res = mod(**inputs) 2025-12-04T08:58:49.6877637Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1727, in forward 2025-12-04T08:58:49.6878043Z encoder_outputs = self.encoder( 2025-12-04T08:58:49.6878533Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T08:58:49.6878952Z layer_outputs = layer_module( 2025-12-04T08:58:49.6879337Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:49.6879748Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:49.6880174Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.6880601Z return func(*args, **kwargs) 2025-12-04T08:58:49.6880981Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T08:58:49.6881725Z self_attention_outputs = self.layer[0]( 2025-12-04T08:58:49.6882144Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.6882555Z return func(*args, **kwargs) 2025-12-04T08:58:49.6882943Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 603, in forward 2025-12-04T08:58:49.6883358Z attention_output = self.SelfAttention( 2025-12-04T08:58:49.6883773Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.6884165Z return func(*args, **kwargs) 2025-12-04T08:58:49.6884567Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 514, in forward 2025-12-04T08:58:49.6885004Z value_states = self.v(current_states) 2025-12-04T08:58:49.6885152Z 2025-12-04T08:58:49.6885247Z cudagraph partition due to non gpu ops 2025-12-04T08:58:49.6885477Z cudagraph partition due to non gpu ops 2025-12-04T08:58:49.6885740Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:49.6886151Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:49.6886504Z res = mod(**inputs) 2025-12-04T08:58:49.6886896Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1727, in forward 2025-12-04T08:58:49.6887330Z encoder_outputs = self.encoder( 2025-12-04T08:58:49.6887728Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T08:58:49.6888140Z layer_outputs = layer_module( 2025-12-04T08:58:49.6888521Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:49.6888935Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:49.6889337Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.6889724Z return func(*args, **kwargs) 2025-12-04T08:58:49.6890090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T08:58:49.6890481Z self_attention_outputs = self.layer[0]( 2025-12-04T08:58:49.6890876Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.6891262Z return func(*args, **kwargs) 2025-12-04T08:58:49.6891633Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 603, in forward 2025-12-04T08:58:49.6892020Z attention_output = self.SelfAttention( 2025-12-04T08:58:49.6892416Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.6892835Z return func(*args, **kwargs) 2025-12-04T08:58:49.6893211Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 572, in forward 2025-12-04T08:58:49.6893621Z attn_output = self.o(attn_output) 2025-12-04T08:58:49.6893771Z 2025-12-04T08:58:49.6893889Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:49.6894304Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:49.6894660Z res = mod(**inputs) 2025-12-04T08:58:49.6895043Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1727, in forward 2025-12-04T08:58:49.6895459Z encoder_outputs = self.encoder( 2025-12-04T08:58:49.6895843Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T08:58:49.6896229Z layer_outputs = layer_module( 2025-12-04T08:58:49.6896597Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:49.6896980Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:49.6897384Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.6897769Z return func(*args, **kwargs) 2025-12-04T08:58:49.6898165Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 737, in forward 2025-12-04T08:58:49.6898601Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T08:58:49.6899001Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-12-04T08:58:49.6899461Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T08:58:49.6899925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward 2025-12-04T08:58:49.6900345Z hidden_states = self.wi(hidden_states) 2025-12-04T08:58:49.6900494Z 2025-12-04T08:58:49.6900610Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:49.6901033Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:49.6901370Z res = mod(**inputs) 2025-12-04T08:58:49.6901722Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1727, in forward 2025-12-04T08:58:49.6902123Z encoder_outputs = self.encoder( 2025-12-04T08:58:49.6902501Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T08:58:49.6902883Z layer_outputs = layer_module( 2025-12-04T08:58:49.6903237Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:49.6903615Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:49.6904019Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.6904407Z return func(*args, **kwargs) 2025-12-04T08:58:49.6904787Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 737, in forward 2025-12-04T08:58:49.6905211Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T08:58:49.6905637Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-12-04T08:58:49.6906091Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T08:58:49.6906510Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 289, in forward 2025-12-04T08:58:49.6906897Z hidden_states = self.act(hidden_states) 2025-12-04T08:58:49.6907055Z 2025-12-04T08:58:49.6907170Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:49.6907538Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:49.6907868Z res = mod(**inputs) 2025-12-04T08:58:49.6908224Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1727, in forward 2025-12-04T08:58:49.6908614Z encoder_outputs = self.encoder( 2025-12-04T08:58:49.6909015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T08:58:49.6909434Z layer_outputs = layer_module( 2025-12-04T08:58:49.6909814Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:49.6910206Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:49.6910622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.6911031Z return func(*args, **kwargs) 2025-12-04T08:58:49.6911421Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 737, in forward 2025-12-04T08:58:49.6911830Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T08:58:49.6912233Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-12-04T08:58:49.6912658Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T08:58:49.6913094Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 297, in forward 2025-12-04T08:58:49.6913504Z hidden_states = self.wo(hidden_states) 2025-12-04T08:58:49.6913662Z 2025-12-04T08:58:49.6913778Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:49.6914170Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:49.6914518Z res = mod(**inputs) 2025-12-04T08:58:49.6914899Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1727, in forward 2025-12-04T08:58:49.6915304Z encoder_outputs = self.encoder( 2025-12-04T08:58:49.6915737Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T08:58:49.6916142Z layer_outputs = layer_module( 2025-12-04T08:58:49.6916524Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:49.6916938Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:49.6917346Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.6917774Z return func(*args, **kwargs) 2025-12-04T08:58:49.6918238Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T08:58:49.6918668Z self_attention_outputs = self.layer[0]( 2025-12-04T08:58:49.6919107Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.6919520Z return func(*args, **kwargs) 2025-12-04T08:58:49.6919915Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 603, in forward 2025-12-04T08:58:49.6920350Z attention_output = self.SelfAttention( 2025-12-04T08:58:49.6920972Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.6921405Z return func(*args, **kwargs) 2025-12-04T08:58:49.6921796Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 492, in forward 2025-12-04T08:58:49.6922224Z query_states = self.q(hidden_states) 2025-12-04T08:58:49.6922434Z 2025-12-04T08:58:49.6922551Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:49.6922948Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:49.6923295Z res = mod(**inputs) 2025-12-04T08:58:49.6923675Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1727, in forward 2025-12-04T08:58:49.6924082Z encoder_outputs = self.encoder( 2025-12-04T08:58:49.6924485Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T08:58:49.6924890Z layer_outputs = layer_module( 2025-12-04T08:58:49.6925277Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:49.6925657Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:49.6926041Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.6926431Z return func(*args, **kwargs) 2025-12-04T08:58:49.6926800Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T08:58:49.6927188Z self_attention_outputs = self.layer[0]( 2025-12-04T08:58:49.6927574Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.6927959Z return func(*args, **kwargs) 2025-12-04T08:58:49.6928327Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 603, in forward 2025-12-04T08:58:49.6928720Z attention_output = self.SelfAttention( 2025-12-04T08:58:49.6929105Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.6929484Z return func(*args, **kwargs) 2025-12-04T08:58:49.6929850Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 513, in forward 2025-12-04T08:58:49.6930230Z key_states = self.k(current_states) 2025-12-04T08:58:49.6930374Z 2025-12-04T08:58:49.6930480Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:49.6930891Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:49.6931247Z res = mod(**inputs) 2025-12-04T08:58:49.6931621Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1727, in forward 2025-12-04T08:58:49.6932060Z encoder_outputs = self.encoder( 2025-12-04T08:58:49.6932442Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T08:58:49.6932818Z layer_outputs = layer_module( 2025-12-04T08:58:49.6933179Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:49.6933559Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:49.6933974Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.6934358Z return func(*args, **kwargs) 2025-12-04T08:58:49.6934727Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T08:58:49.6935119Z self_attention_outputs = self.layer[0]( 2025-12-04T08:58:49.6935520Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.6935900Z return func(*args, **kwargs) 2025-12-04T08:58:49.6936271Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 603, in forward 2025-12-04T08:58:49.6936659Z attention_output = self.SelfAttention( 2025-12-04T08:58:49.6937044Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.6937448Z return func(*args, **kwargs) 2025-12-04T08:58:49.6937819Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 514, in forward 2025-12-04T08:58:49.6938209Z value_states = self.v(current_states) 2025-12-04T08:58:49.6938347Z 2025-12-04T08:58:49.6938431Z cudagraph partition due to non gpu ops 2025-12-04T08:58:49.6938654Z cudagraph partition due to non gpu ops 2025-12-04T08:58:49.6938901Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:49.6939263Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:49.6939596Z res = mod(**inputs) 2025-12-04T08:58:49.6939951Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1727, in forward 2025-12-04T08:58:49.6940335Z encoder_outputs = self.encoder( 2025-12-04T08:58:49.6940710Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T08:58:49.6941092Z layer_outputs = layer_module( 2025-12-04T08:58:49.6941461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:49.6941818Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:49.6942195Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.6942574Z return func(*args, **kwargs) 2025-12-04T08:58:49.6942922Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T08:58:49.6943280Z self_attention_outputs = self.layer[0]( 2025-12-04T08:58:49.6943650Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.6944012Z return func(*args, **kwargs) 2025-12-04T08:58:49.6944351Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 603, in forward 2025-12-04T08:58:49.6944722Z attention_output = self.SelfAttention( 2025-12-04T08:58:49.6945121Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.6945489Z return func(*args, **kwargs) 2025-12-04T08:58:49.6945838Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 572, in forward 2025-12-04T08:58:49.6946229Z attn_output = self.o(attn_output) 2025-12-04T08:58:49.6946359Z 2025-12-04T08:58:49.6946471Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:49.6946835Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:49.6947146Z res = mod(**inputs) 2025-12-04T08:58:49.6947483Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1727, in forward 2025-12-04T08:58:49.6947866Z encoder_outputs = self.encoder( 2025-12-04T08:58:49.6948219Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T08:58:49.6948580Z layer_outputs = layer_module( 2025-12-04T08:58:49.6948916Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:49.6949273Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:49.6949634Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.6949996Z return func(*args, **kwargs) 2025-12-04T08:58:49.6950344Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T08:58:49.6950725Z self_attention_outputs = self.layer[0]( 2025-12-04T08:58:49.6951103Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.6951471Z return func(*args, **kwargs) 2025-12-04T08:58:49.6951830Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 613, in forward 2025-12-04T08:58:49.6952255Z hidden_states = hidden_states + self.dropout(attention_output[0]) 2025-12-04T08:58:49.6952450Z 2025-12-04T08:58:49.6952564Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:49.6952915Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:49.6953231Z res = mod(**inputs) 2025-12-04T08:58:49.6953561Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1727, in forward 2025-12-04T08:58:49.6953934Z encoder_outputs = self.encoder( 2025-12-04T08:58:49.6954314Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T08:58:49.6954688Z layer_outputs = layer_module( 2025-12-04T08:58:49.6955049Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:49.6955429Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:49.6955810Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.6956180Z return func(*args, **kwargs) 2025-12-04T08:58:49.6956535Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 737, in forward 2025-12-04T08:58:49.6956927Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T08:58:49.6957308Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-12-04T08:58:49.6957726Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T08:58:49.6958201Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward 2025-12-04T08:58:49.6958589Z hidden_states = self.wi(hidden_states) 2025-12-04T08:58:49.6958742Z 2025-12-04T08:58:49.6958850Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:49.6959223Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:49.6959582Z res = mod(**inputs) 2025-12-04T08:58:49.6959949Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1727, in forward 2025-12-04T08:58:49.6960339Z encoder_outputs = self.encoder( 2025-12-04T08:58:49.6960731Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T08:58:49.6961129Z layer_outputs = layer_module( 2025-12-04T08:58:49.6961503Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:49.6961875Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:49.6962254Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.6962626Z return func(*args, **kwargs) 2025-12-04T08:58:49.6962986Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 737, in forward 2025-12-04T08:58:49.6963390Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T08:58:49.6963789Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-12-04T08:58:49.6964214Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T08:58:49.6964620Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 289, in forward 2025-12-04T08:58:49.6965029Z hidden_states = self.act(hidden_states) 2025-12-04T08:58:49.6965166Z 2025-12-04T08:58:49.6965279Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:49.6965635Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:49.6965962Z res = mod(**inputs) 2025-12-04T08:58:49.6966311Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1727, in forward 2025-12-04T08:58:49.6966689Z encoder_outputs = self.encoder( 2025-12-04T08:58:49.6967061Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T08:58:49.6967437Z layer_outputs = layer_module( 2025-12-04T08:58:49.6967791Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:49.6968157Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:49.6968560Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.6968947Z return func(*args, **kwargs) 2025-12-04T08:58:49.6969325Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 737, in forward 2025-12-04T08:58:49.6969712Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T08:58:49.6970101Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-12-04T08:58:49.6970522Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T08:58:49.6970926Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 297, in forward 2025-12-04T08:58:49.6971305Z hidden_states = self.wo(hidden_states) 2025-12-04T08:58:49.6971445Z 2025-12-04T08:58:49.6971552Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:49.6971914Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:49.6972235Z res = mod(**inputs) 2025-12-04T08:58:49.6972604Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1727, in forward 2025-12-04T08:58:49.6972987Z encoder_outputs = self.encoder( 2025-12-04T08:58:49.6973399Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T08:58:49.6973801Z layer_outputs = layer_module( 2025-12-04T08:58:49.6974165Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:49.6974545Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:49.6974933Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.6975383Z return func(*args, **kwargs) 2025-12-04T08:58:49.6975768Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T08:58:49.6976159Z self_attention_outputs = self.layer[0]( 2025-12-04T08:58:49.6976553Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.6976938Z return func(*args, **kwargs) 2025-12-04T08:58:49.6977307Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 603, in forward 2025-12-04T08:58:49.6977696Z attention_output = self.SelfAttention( 2025-12-04T08:58:49.6978095Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.6978479Z return func(*args, **kwargs) 2025-12-04T08:58:49.6978845Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 492, in forward 2025-12-04T08:58:49.6979254Z query_states = self.q(hidden_states) 2025-12-04T08:58:49.6979400Z 2025-12-04T08:58:49.6979509Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:49.6979884Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:49.6980218Z res = mod(**inputs) 2025-12-04T08:58:49.6980568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1727, in forward 2025-12-04T08:58:49.6980957Z encoder_outputs = self.encoder( 2025-12-04T08:58:49.6981336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T08:58:49.6981714Z layer_outputs = layer_module( 2025-12-04T08:58:49.6982077Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:49.6982456Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:49.6982848Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.6983224Z return func(*args, **kwargs) 2025-12-04T08:58:49.6983591Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T08:58:49.6983985Z self_attention_outputs = self.layer[0]( 2025-12-04T08:58:49.6984371Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.6984760Z return func(*args, **kwargs) 2025-12-04T08:58:49.6985131Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 603, in forward 2025-12-04T08:58:49.6985530Z attention_output = self.SelfAttention( 2025-12-04T08:58:49.6985925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.6986311Z return func(*args, **kwargs) 2025-12-04T08:58:49.6986686Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 513, in forward 2025-12-04T08:58:49.6987074Z key_states = self.k(current_states) 2025-12-04T08:58:49.6987214Z 2025-12-04T08:58:49.6987319Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:49.6987682Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:49.6988024Z res = mod(**inputs) 2025-12-04T08:58:49.6988362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1727, in forward 2025-12-04T08:58:49.6988740Z encoder_outputs = self.encoder( 2025-12-04T08:58:49.6989107Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T08:58:49.6989484Z layer_outputs = layer_module( 2025-12-04T08:58:49.6989841Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:49.6990209Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:49.6990595Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.6990960Z return func(*args, **kwargs) 2025-12-04T08:58:49.6991315Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T08:58:49.6991696Z self_attention_outputs = self.layer[0]( 2025-12-04T08:58:49.6992081Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.6992447Z return func(*args, **kwargs) 2025-12-04T08:58:49.6992802Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 603, in forward 2025-12-04T08:58:49.6993212Z attention_output = self.SelfAttention( 2025-12-04T08:58:49.6993602Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.6993994Z return func(*args, **kwargs) 2025-12-04T08:58:49.6994360Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 514, in forward 2025-12-04T08:58:49.6994749Z value_states = self.v(current_states) 2025-12-04T08:58:49.6994891Z 2025-12-04T08:58:49.6994975Z cudagraph partition due to non gpu ops 2025-12-04T08:58:49.6995197Z cudagraph partition due to non gpu ops 2025-12-04T08:58:49.6995451Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:49.6995836Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:49.6996193Z res = mod(**inputs) 2025-12-04T08:58:49.6996571Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1727, in forward 2025-12-04T08:58:49.6996983Z encoder_outputs = self.encoder( 2025-12-04T08:58:49.6997379Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T08:58:49.6997788Z layer_outputs = layer_module( 2025-12-04T08:58:49.6998236Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:49.6998654Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:49.6999096Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.6999503Z return func(*args, **kwargs) 2025-12-04T08:58:49.6999893Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T08:58:49.7000302Z self_attention_outputs = self.layer[0]( 2025-12-04T08:58:49.7000723Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7001101Z return func(*args, **kwargs) 2025-12-04T08:58:49.7001486Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 603, in forward 2025-12-04T08:58:49.7001869Z attention_output = self.SelfAttention( 2025-12-04T08:58:49.7002265Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7002658Z return func(*args, **kwargs) 2025-12-04T08:58:49.7003011Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 572, in forward 2025-12-04T08:58:49.7003402Z attn_output = self.o(attn_output) 2025-12-04T08:58:49.7003539Z 2025-12-04T08:58:49.7003645Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:49.7004003Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:49.7004342Z res = mod(**inputs) 2025-12-04T08:58:49.7004695Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1727, in forward 2025-12-04T08:58:49.7005075Z encoder_outputs = self.encoder( 2025-12-04T08:58:49.7005438Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T08:58:49.7005813Z layer_outputs = layer_module( 2025-12-04T08:58:49.7006162Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:49.7006528Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:49.7006910Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7007285Z return func(*args, **kwargs) 2025-12-04T08:58:49.7007659Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 737, in forward 2025-12-04T08:58:49.7008050Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T08:58:49.7008440Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-12-04T08:58:49.7008886Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T08:58:49.7009293Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward 2025-12-04T08:58:49.7009669Z hidden_states = self.wi(hidden_states) 2025-12-04T08:58:49.7009808Z 2025-12-04T08:58:49.7009910Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:49.7010257Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:49.7010573Z res = mod(**inputs) 2025-12-04T08:58:49.7010912Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1727, in forward 2025-12-04T08:58:49.7011293Z encoder_outputs = self.encoder( 2025-12-04T08:58:49.7011657Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T08:58:49.7012027Z layer_outputs = layer_module( 2025-12-04T08:58:49.7012366Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:49.7012723Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:49.7013089Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7013443Z return func(*args, **kwargs) 2025-12-04T08:58:49.7013792Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 737, in forward 2025-12-04T08:58:49.7014166Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T08:58:49.7014539Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-12-04T08:58:49.7014937Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T08:58:49.7015358Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 289, in forward 2025-12-04T08:58:49.7015728Z hidden_states = self.act(hidden_states) 2025-12-04T08:58:49.7015862Z 2025-12-04T08:58:49.7015996Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:49.7016353Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:49.7016674Z res = mod(**inputs) 2025-12-04T08:58:49.7017017Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1727, in forward 2025-12-04T08:58:49.7017383Z encoder_outputs = self.encoder( 2025-12-04T08:58:49.7017747Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T08:58:49.7018147Z layer_outputs = layer_module( 2025-12-04T08:58:49.7018485Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:49.7018847Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:49.7019229Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7019611Z return func(*args, **kwargs) 2025-12-04T08:58:49.7019953Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 737, in forward 2025-12-04T08:58:49.7020332Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T08:58:49.7020836Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-12-04T08:58:49.7021307Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T08:58:49.7021714Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 297, in forward 2025-12-04T08:58:49.7022093Z hidden_states = self.wo(hidden_states) 2025-12-04T08:58:49.7022229Z 2025-12-04T08:58:49.7022346Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:49.7022708Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:49.7023028Z res = mod(**inputs) 2025-12-04T08:58:49.7023373Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1727, in forward 2025-12-04T08:58:49.7023745Z encoder_outputs = self.encoder( 2025-12-04T08:58:49.7024101Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T08:58:49.7024469Z layer_outputs = layer_module( 2025-12-04T08:58:49.7024816Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:49.7025169Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:49.7025546Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7025913Z return func(*args, **kwargs) 2025-12-04T08:58:49.7026262Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T08:58:49.7026629Z self_attention_outputs = self.layer[0]( 2025-12-04T08:58:49.7027005Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7027373Z return func(*args, **kwargs) 2025-12-04T08:58:49.7027719Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 603, in forward 2025-12-04T08:58:49.7028095Z attention_output = self.SelfAttention( 2025-12-04T08:58:49.7028473Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7028837Z return func(*args, **kwargs) 2025-12-04T08:58:49.7029207Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 492, in forward 2025-12-04T08:58:49.7029580Z query_states = self.q(hidden_states) 2025-12-04T08:58:49.7029716Z 2025-12-04T08:58:49.7029852Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:49.7030223Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:49.7030549Z res = mod(**inputs) 2025-12-04T08:58:49.7030903Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1727, in forward 2025-12-04T08:58:49.7031290Z encoder_outputs = self.encoder( 2025-12-04T08:58:49.7031664Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T08:58:49.7032062Z layer_outputs = layer_module( 2025-12-04T08:58:49.7032424Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:49.7032826Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:49.7033240Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7033648Z return func(*args, **kwargs) 2025-12-04T08:58:49.7034035Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T08:58:49.7034447Z self_attention_outputs = self.layer[0]( 2025-12-04T08:58:49.7034874Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7035302Z return func(*args, **kwargs) 2025-12-04T08:58:49.7035741Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 603, in forward 2025-12-04T08:58:49.7036158Z attention_output = self.SelfAttention( 2025-12-04T08:58:49.7036588Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7037005Z return func(*args, **kwargs) 2025-12-04T08:58:49.7037405Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 513, in forward 2025-12-04T08:58:49.7037815Z key_states = self.k(current_states) 2025-12-04T08:58:49.7037968Z 2025-12-04T08:58:49.7038082Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:49.7038545Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:49.7038908Z res = mod(**inputs) 2025-12-04T08:58:49.7039310Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1727, in forward 2025-12-04T08:58:49.7039739Z encoder_outputs = self.encoder( 2025-12-04T08:58:49.7040167Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T08:58:49.7040579Z layer_outputs = layer_module( 2025-12-04T08:58:49.7040961Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:49.7041364Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:49.7041782Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7042200Z return func(*args, **kwargs) 2025-12-04T08:58:49.7042645Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T08:58:49.7043068Z self_attention_outputs = self.layer[0]( 2025-12-04T08:58:49.7043482Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7043887Z return func(*args, **kwargs) 2025-12-04T08:58:49.7044295Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 603, in forward 2025-12-04T08:58:49.7044713Z attention_output = self.SelfAttention( 2025-12-04T08:58:49.7045137Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7045561Z return func(*args, **kwargs) 2025-12-04T08:58:49.7045953Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 514, in forward 2025-12-04T08:58:49.7046427Z value_states = self.v(current_states) 2025-12-04T08:58:49.7046582Z 2025-12-04T08:58:49.7046669Z cudagraph partition due to non gpu ops 2025-12-04T08:58:49.7046913Z cudagraph partition due to non gpu ops 2025-12-04T08:58:49.7047154Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:49.7047540Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:49.7047878Z res = mod(**inputs) 2025-12-04T08:58:49.7048242Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T08:58:49.7048652Z decoder_outputs = self.decoder( 2025-12-04T08:58:49.7049041Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T08:58:49.7062502Z layer_outputs = layer_module( 2025-12-04T08:58:49.7063114Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:49.7063506Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:49.7063919Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7064416Z return func(*args, **kwargs) 2025-12-04T08:58:49.7064795Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T08:58:49.7065211Z self_attention_outputs = self.layer[0]( 2025-12-04T08:58:49.7065623Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7066017Z return func(*args, **kwargs) 2025-12-04T08:58:49.7066384Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 603, in forward 2025-12-04T08:58:49.7066788Z attention_output = self.SelfAttention( 2025-12-04T08:58:49.7067197Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7067577Z return func(*args, **kwargs) 2025-12-04T08:58:49.7067954Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 492, in forward 2025-12-04T08:58:49.7068343Z query_states = self.q(hidden_states) 2025-12-04T08:58:49.7068488Z 2025-12-04T08:58:49.7068614Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:49.7068991Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:49.7069342Z res = mod(**inputs) 2025-12-04T08:58:49.7069703Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T08:58:49.7070089Z decoder_outputs = self.decoder( 2025-12-04T08:58:49.7070456Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T08:58:49.7070841Z layer_outputs = layer_module( 2025-12-04T08:58:49.7071210Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:49.7071595Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:49.7072000Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7072435Z return func(*args, **kwargs) 2025-12-04T08:58:49.7072811Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T08:58:49.7073195Z self_attention_outputs = self.layer[0]( 2025-12-04T08:58:49.7073622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7073986Z return func(*args, **kwargs) 2025-12-04T08:58:49.7074331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 603, in forward 2025-12-04T08:58:49.7074719Z attention_output = self.SelfAttention( 2025-12-04T08:58:49.7075107Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7075506Z return func(*args, **kwargs) 2025-12-04T08:58:49.7075865Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 513, in forward 2025-12-04T08:58:49.7076253Z key_states = self.k(current_states) 2025-12-04T08:58:49.7076394Z 2025-12-04T08:58:49.7076519Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:49.7076906Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:49.7077237Z res = mod(**inputs) 2025-12-04T08:58:49.7077603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T08:58:49.7077991Z decoder_outputs = self.decoder( 2025-12-04T08:58:49.7078467Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T08:58:49.7078921Z layer_outputs = layer_module( 2025-12-04T08:58:49.7079320Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:49.7079705Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:49.7080093Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7080484Z return func(*args, **kwargs) 2025-12-04T08:58:49.7080860Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T08:58:49.7081243Z self_attention_outputs = self.layer[0]( 2025-12-04T08:58:49.7081644Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7082025Z return func(*args, **kwargs) 2025-12-04T08:58:49.7082406Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 603, in forward 2025-12-04T08:58:49.7082780Z attention_output = self.SelfAttention( 2025-12-04T08:58:49.7083167Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7083539Z return func(*args, **kwargs) 2025-12-04T08:58:49.7083885Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 514, in forward 2025-12-04T08:58:49.7084265Z value_states = self.v(current_states) 2025-12-04T08:58:49.7084408Z 2025-12-04T08:58:49.7084493Z cudagraph partition due to non gpu ops 2025-12-04T08:58:49.7084715Z cudagraph partition due to non gpu ops 2025-12-04T08:58:49.7084954Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:49.7085320Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:49.7085651Z res = mod(**inputs) 2025-12-04T08:58:49.7085997Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T08:58:49.7086377Z decoder_outputs = self.decoder( 2025-12-04T08:58:49.7086764Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T08:58:49.7087152Z layer_outputs = layer_module( 2025-12-04T08:58:49.7087497Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:49.7087887Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:49.7088274Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7088662Z return func(*args, **kwargs) 2025-12-04T08:58:49.7089015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T08:58:49.7089400Z self_attention_outputs = self.layer[0]( 2025-12-04T08:58:49.7089797Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7090175Z return func(*args, **kwargs) 2025-12-04T08:58:49.7090548Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 603, in forward 2025-12-04T08:58:49.7090947Z attention_output = self.SelfAttention( 2025-12-04T08:58:49.7091342Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7091724Z return func(*args, **kwargs) 2025-12-04T08:58:49.7092082Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 572, in forward 2025-12-04T08:58:49.7092458Z attn_output = self.o(attn_output) 2025-12-04T08:58:49.7092590Z 2025-12-04T08:58:49.7092714Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:49.7093083Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:49.7093414Z res = mod(**inputs) 2025-12-04T08:58:49.7093764Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T08:58:49.7094129Z decoder_outputs = self.decoder( 2025-12-04T08:58:49.7094499Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T08:58:49.7094876Z layer_outputs = layer_module( 2025-12-04T08:58:49.7095216Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:49.7095584Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:49.7095964Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7096349Z return func(*args, **kwargs) 2025-12-04T08:58:49.7096700Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 711, in forward 2025-12-04T08:58:49.7097085Z cross_attention_outputs = self.layer[1]( 2025-12-04T08:58:49.7097474Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7097849Z return func(*args, **kwargs) 2025-12-04T08:58:49.7098206Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 640, in forward 2025-12-04T08:58:49.7098598Z attention_output = self.EncDecAttention( 2025-12-04T08:58:49.7098983Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7099347Z return func(*args, **kwargs) 2025-12-04T08:58:49.7099703Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 492, in forward 2025-12-04T08:58:49.7100099Z query_states = self.q(hidden_states) 2025-12-04T08:58:49.7100231Z 2025-12-04T08:58:49.7100335Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:49.7100716Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:49.7101045Z res = mod(**inputs) 2025-12-04T08:58:49.7101392Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1727, in forward 2025-12-04T08:58:49.7101786Z encoder_outputs = self.encoder( 2025-12-04T08:58:49.7102157Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T08:58:49.7102574Z layer_outputs = layer_module( 2025-12-04T08:58:49.7102927Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:49.7103313Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:49.7103712Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7104091Z return func(*args, **kwargs) 2025-12-04T08:58:49.7104447Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T08:58:49.7104830Z self_attention_outputs = self.layer[0]( 2025-12-04T08:58:49.7105215Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7105593Z return func(*args, **kwargs) 2025-12-04T08:58:49.7105944Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 603, in forward 2025-12-04T08:58:49.7106327Z attention_output = self.SelfAttention( 2025-12-04T08:58:49.7106714Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7107100Z return func(*args, **kwargs) 2025-12-04T08:58:49.7107476Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 572, in forward 2025-12-04T08:58:49.7107874Z attn_output = self.o(attn_output) 2025-12-04T08:58:49.7108018Z 2025-12-04T08:58:49.7108144Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:49.7108544Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:49.7108908Z res = mod(**inputs) 2025-12-04T08:58:49.7109293Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1727, in forward 2025-12-04T08:58:49.7109674Z encoder_outputs = self.encoder( 2025-12-04T08:58:49.7110046Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T08:58:49.7110428Z layer_outputs = layer_module( 2025-12-04T08:58:49.7110815Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:49.7111211Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:49.7111630Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7112041Z return func(*args, **kwargs) 2025-12-04T08:58:49.7112425Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T08:58:49.7112846Z self_attention_outputs = self.layer[0]( 2025-12-04T08:58:49.7113270Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7113702Z return func(*args, **kwargs) 2025-12-04T08:58:49.7114087Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 613, in forward 2025-12-04T08:58:49.7114564Z hidden_states = hidden_states + self.dropout(attention_output[0]) 2025-12-04T08:58:49.7114773Z 2025-12-04T08:58:49.7114899Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:49.7115314Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:49.7115659Z res = mod(**inputs) 2025-12-04T08:58:49.7116036Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1727, in forward 2025-12-04T08:58:49.7116466Z encoder_outputs = self.encoder( 2025-12-04T08:58:49.7116885Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T08:58:49.7117312Z layer_outputs = layer_module( 2025-12-04T08:58:49.7117713Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:49.7118111Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:49.7118699Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7119125Z return func(*args, **kwargs) 2025-12-04T08:58:49.7119539Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 737, in forward 2025-12-04T08:58:49.7119950Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T08:58:49.7120365Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-12-04T08:58:49.7121018Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T08:58:49.7121454Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward 2025-12-04T08:58:49.7121841Z hidden_states = self.wi(hidden_states) 2025-12-04T08:58:49.7121998Z 2025-12-04T08:58:49.7122178Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:49.7122575Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:49.7122927Z res = mod(**inputs) 2025-12-04T08:58:49.7123309Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1727, in forward 2025-12-04T08:58:49.7123701Z encoder_outputs = self.encoder( 2025-12-04T08:58:49.7124083Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T08:58:49.7124465Z layer_outputs = layer_module( 2025-12-04T08:58:49.7124832Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:49.7125214Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:49.7125612Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7126000Z return func(*args, **kwargs) 2025-12-04T08:58:49.7126376Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 737, in forward 2025-12-04T08:58:49.7126781Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T08:58:49.7127179Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-12-04T08:58:49.7127611Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T08:58:49.7128031Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 289, in forward 2025-12-04T08:58:49.7128419Z hidden_states = self.act(hidden_states) 2025-12-04T08:58:49.7128559Z 2025-12-04T08:58:49.7128665Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:49.7129037Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:49.7129374Z res = mod(**inputs) 2025-12-04T08:58:49.7129725Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1727, in forward 2025-12-04T08:58:49.7130101Z encoder_outputs = self.encoder( 2025-12-04T08:58:49.7130518Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T08:58:49.7130901Z layer_outputs = layer_module( 2025-12-04T08:58:49.7131252Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:49.7131658Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:49.7132050Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7132438Z return func(*args, **kwargs) 2025-12-04T08:58:49.7132795Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 737, in forward 2025-12-04T08:58:49.7133200Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T08:58:49.7133626Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-12-04T08:58:49.7134066Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T08:58:49.7134480Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 297, in forward 2025-12-04T08:58:49.7134871Z hidden_states = self.wo(hidden_states) 2025-12-04T08:58:49.7135019Z 2025-12-04T08:58:49.7135128Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:49.7135498Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:49.7135822Z res = mod(**inputs) 2025-12-04T08:58:49.7136178Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T08:58:49.7136587Z decoder_outputs = self.decoder( 2025-12-04T08:58:49.7136961Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T08:58:49.7137349Z layer_outputs = layer_module( 2025-12-04T08:58:49.7137711Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:49.7138085Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:49.7138470Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7138856Z return func(*args, **kwargs) 2025-12-04T08:58:49.7139227Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 711, in forward 2025-12-04T08:58:49.7139612Z cross_attention_outputs = self.layer[1]( 2025-12-04T08:58:49.7140005Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7140389Z return func(*args, **kwargs) 2025-12-04T08:58:49.7140766Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 640, in forward 2025-12-04T08:58:49.7141147Z attention_output = self.EncDecAttention( 2025-12-04T08:58:49.7141542Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7141920Z return func(*args, **kwargs) 2025-12-04T08:58:49.7142287Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 513, in forward 2025-12-04T08:58:49.7142654Z key_states = self.k(current_states) 2025-12-04T08:58:49.7142793Z 2025-12-04T08:58:49.7142898Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:49.7143259Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:49.7143575Z res = mod(**inputs) 2025-12-04T08:58:49.7143919Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T08:58:49.7144293Z decoder_outputs = self.decoder( 2025-12-04T08:58:49.7144674Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T08:58:49.7145037Z layer_outputs = layer_module( 2025-12-04T08:58:49.7145384Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:49.7145766Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:49.7146138Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7146514Z return func(*args, **kwargs) 2025-12-04T08:58:49.7146878Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 711, in forward 2025-12-04T08:58:49.7147259Z cross_attention_outputs = self.layer[1]( 2025-12-04T08:58:49.7147650Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7148023Z return func(*args, **kwargs) 2025-12-04T08:58:49.7148381Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 640, in forward 2025-12-04T08:58:49.7148764Z attention_output = self.EncDecAttention( 2025-12-04T08:58:49.7149159Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7149551Z return func(*args, **kwargs) 2025-12-04T08:58:49.7149908Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 514, in forward 2025-12-04T08:58:49.7149990Z value_states = self.v(current_states) 2025-12-04T08:58:49.7150013Z 2025-12-04T08:58:49.7150095Z cudagraph partition due to non gpu ops 2025-12-04T08:58:49.7150185Z cudagraph partition due to non gpu ops 2025-12-04T08:58:49.7150289Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:49.7150499Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:49.7150565Z res = mod(**inputs) 2025-12-04T08:58:49.7150803Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T08:58:49.7150888Z decoder_outputs = self.decoder( 2025-12-04T08:58:49.7151128Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T08:58:49.7151200Z layer_outputs = layer_module( 2025-12-04T08:58:49.7151429Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:49.7151507Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:49.7151756Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7151825Z return func(*args, **kwargs) 2025-12-04T08:58:49.7152060Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 711, in forward 2025-12-04T08:58:49.7152147Z cross_attention_outputs = self.layer[1]( 2025-12-04T08:58:49.7152387Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7152457Z return func(*args, **kwargs) 2025-12-04T08:58:49.7152709Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 640, in forward 2025-12-04T08:58:49.7152793Z attention_output = self.EncDecAttention( 2025-12-04T08:58:49.7153039Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7153108Z return func(*args, **kwargs) 2025-12-04T08:58:49.7153344Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 572, in forward 2025-12-04T08:58:49.7153430Z attn_output = self.o(attn_output) 2025-12-04T08:58:49.7153449Z 2025-12-04T08:58:49.7153553Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:49.7153758Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:49.7153839Z res = mod(**inputs) 2025-12-04T08:58:49.7154075Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T08:58:49.7154155Z decoder_outputs = self.decoder( 2025-12-04T08:58:49.7154388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T08:58:49.7154459Z layer_outputs = layer_module( 2025-12-04T08:58:49.7154709Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:49.7154787Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:49.7155038Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7155107Z return func(*args, **kwargs) 2025-12-04T08:58:49.7155339Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 737, in forward 2025-12-04T08:58:49.7155439Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T08:58:49.7155671Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-12-04T08:58:49.7155792Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T08:58:49.7156024Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward 2025-12-04T08:58:49.7156122Z hidden_states = self.wi(hidden_states) 2025-12-04T08:58:49.7156127Z 2025-12-04T08:58:49.7156235Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:49.7156435Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:49.7156500Z res = mod(**inputs) 2025-12-04T08:58:49.7156752Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T08:58:49.7156827Z decoder_outputs = self.decoder( 2025-12-04T08:58:49.7157068Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T08:58:49.7157138Z layer_outputs = layer_module( 2025-12-04T08:58:49.7157360Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:49.7157447Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:49.7157688Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7157756Z return func(*args, **kwargs) 2025-12-04T08:58:49.7157997Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 737, in forward 2025-12-04T08:58:49.7158088Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T08:58:49.7158431Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-12-04T08:58:49.7158568Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T08:58:49.7158840Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 289, in forward 2025-12-04T08:58:49.7158938Z hidden_states = self.act(hidden_states) 2025-12-04T08:58:49.7158943Z 2025-12-04T08:58:49.7159058Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:49.7159292Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:49.7159364Z res = mod(**inputs) 2025-12-04T08:58:49.7159690Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T08:58:49.7159772Z decoder_outputs = self.decoder( 2025-12-04T08:58:49.7160009Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T08:58:49.7160099Z layer_outputs = layer_module( 2025-12-04T08:58:49.7160348Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:49.7160423Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:49.7160665Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7160734Z return func(*args, **kwargs) 2025-12-04T08:58:49.7160979Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 737, in forward 2025-12-04T08:58:49.7161076Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T08:58:49.7161319Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-12-04T08:58:49.7161437Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T08:58:49.7161671Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 297, in forward 2025-12-04T08:58:49.7161750Z hidden_states = self.wo(hidden_states) 2025-12-04T08:58:49.7161754Z 2025-12-04T08:58:49.7161863Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:49.7162061Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:49.7162150Z res = mod(**inputs) 2025-12-04T08:58:49.7162398Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T08:58:49.7162472Z decoder_outputs = self.decoder( 2025-12-04T08:58:49.7162719Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T08:58:49.7162791Z layer_outputs = layer_module( 2025-12-04T08:58:49.7163013Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:49.7163102Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:49.7163343Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7163413Z return func(*args, **kwargs) 2025-12-04T08:58:49.7163656Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T08:58:49.7163738Z self_attention_outputs = self.layer[0]( 2025-12-04T08:58:49.7163988Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7164056Z return func(*args, **kwargs) 2025-12-04T08:58:49.7164294Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 603, in forward 2025-12-04T08:58:49.7164386Z attention_output = self.SelfAttention( 2025-12-04T08:58:49.7164626Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7164701Z return func(*args, **kwargs) 2025-12-04T08:58:49.7164936Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 492, in forward 2025-12-04T08:58:49.7165013Z query_states = self.q(hidden_states) 2025-12-04T08:58:49.7165017Z 2025-12-04T08:58:49.7165128Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:49.7165332Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:49.7165398Z res = mod(**inputs) 2025-12-04T08:58:49.7165657Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T08:58:49.7165731Z decoder_outputs = self.decoder( 2025-12-04T08:58:49.7165979Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T08:58:49.7166066Z layer_outputs = layer_module( 2025-12-04T08:58:49.7166296Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:49.7166378Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:49.7166614Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7166682Z return func(*args, **kwargs) 2025-12-04T08:58:49.7166935Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T08:58:49.7167016Z self_attention_outputs = self.layer[0]( 2025-12-04T08:58:49.7167257Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7167324Z return func(*args, **kwargs) 2025-12-04T08:58:49.7167559Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 603, in forward 2025-12-04T08:58:49.7167651Z attention_output = self.SelfAttention( 2025-12-04T08:58:49.7167891Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7167965Z return func(*args, **kwargs) 2025-12-04T08:58:49.7168201Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 513, in forward 2025-12-04T08:58:49.7168296Z key_states = self.k(current_states) 2025-12-04T08:58:49.7168302Z 2025-12-04T08:58:49.7168412Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:49.7168614Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:49.7168677Z res = mod(**inputs) 2025-12-04T08:58:49.7168922Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T08:58:49.7168997Z decoder_outputs = self.decoder( 2025-12-04T08:58:49.7169239Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T08:58:49.7169310Z layer_outputs = layer_module( 2025-12-04T08:58:49.7169531Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:49.7169617Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:49.7169859Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7169927Z return func(*args, **kwargs) 2025-12-04T08:58:49.7170171Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T08:58:49.7170252Z self_attention_outputs = self.layer[0]( 2025-12-04T08:58:49.7170497Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7170567Z return func(*args, **kwargs) 2025-12-04T08:58:49.7170802Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 603, in forward 2025-12-04T08:58:49.7170892Z attention_output = self.SelfAttention( 2025-12-04T08:58:49.7171133Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7171208Z return func(*args, **kwargs) 2025-12-04T08:58:49.7171452Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 514, in forward 2025-12-04T08:58:49.7171543Z value_states = self.v(current_states) 2025-12-04T08:58:49.7171547Z 2025-12-04T08:58:49.7171634Z cudagraph partition due to non gpu ops 2025-12-04T08:58:49.7171712Z cudagraph partition due to non gpu ops 2025-12-04T08:58:49.7171814Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:49.7172032Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:49.7172095Z res = mod(**inputs) 2025-12-04T08:58:49.7172340Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T08:58:49.7172410Z decoder_outputs = self.decoder( 2025-12-04T08:58:49.7172638Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T08:58:49.7172728Z layer_outputs = layer_module( 2025-12-04T08:58:49.7172946Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:49.7173026Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:49.7173272Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7173342Z return func(*args, **kwargs) 2025-12-04T08:58:49.7173581Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T08:58:49.7173660Z self_attention_outputs = self.layer[0]( 2025-12-04T08:58:49.7173902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7173996Z return func(*args, **kwargs) 2025-12-04T08:58:49.7174232Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 603, in forward 2025-12-04T08:58:49.7174319Z attention_output = self.SelfAttention( 2025-12-04T08:58:49.7174557Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7174626Z return func(*args, **kwargs) 2025-12-04T08:58:49.7174867Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 572, in forward 2025-12-04T08:58:49.7174944Z attn_output = self.o(attn_output) 2025-12-04T08:58:49.7174948Z 2025-12-04T08:58:49.7175052Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:49.7175257Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:49.7175321Z res = mod(**inputs) 2025-12-04T08:58:49.7175564Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T08:58:49.7175639Z decoder_outputs = self.decoder( 2025-12-04T08:58:49.7175875Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T08:58:49.7175952Z layer_outputs = layer_module( 2025-12-04T08:58:49.7176174Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:49.7176253Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:49.7176498Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7176566Z return func(*args, **kwargs) 2025-12-04T08:58:49.7176808Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 711, in forward 2025-12-04T08:58:49.7176888Z cross_attention_outputs = self.layer[1]( 2025-12-04T08:58:49.7177131Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7177208Z return func(*args, **kwargs) 2025-12-04T08:58:49.7177463Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 640, in forward 2025-12-04T08:58:49.7177556Z attention_output = self.EncDecAttention( 2025-12-04T08:58:49.7177799Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7177927Z return func(*args, **kwargs) 2025-12-04T08:58:49.7178168Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 492, in forward 2025-12-04T08:58:49.7178245Z query_states = self.q(hidden_states) 2025-12-04T08:58:49.7178249Z 2025-12-04T08:58:49.7178352Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:49.7178557Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:49.7178636Z res = mod(**inputs) 2025-12-04T08:58:49.7178878Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T08:58:49.7178954Z decoder_outputs = self.decoder( 2025-12-04T08:58:49.7179192Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T08:58:49.7179272Z layer_outputs = layer_module( 2025-12-04T08:58:49.7179494Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:49.7179571Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:49.7179818Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7179888Z return func(*args, **kwargs) 2025-12-04T08:58:49.7180153Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 711, in forward 2025-12-04T08:58:49.7180233Z cross_attention_outputs = self.layer[1]( 2025-12-04T08:58:49.7180478Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7180557Z return func(*args, **kwargs) 2025-12-04T08:58:49.7180797Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 640, in forward 2025-12-04T08:58:49.7180892Z attention_output = self.EncDecAttention( 2025-12-04T08:58:49.7181132Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7181202Z return func(*args, **kwargs) 2025-12-04T08:58:49.7181441Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 513, in forward 2025-12-04T08:58:49.7181520Z key_states = self.k(current_states) 2025-12-04T08:58:49.7181525Z 2025-12-04T08:58:49.7181627Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:49.7181833Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:49.7181899Z res = mod(**inputs) 2025-12-04T08:58:49.7182142Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T08:58:49.7182216Z decoder_outputs = self.decoder( 2025-12-04T08:58:49.7182451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T08:58:49.7182527Z layer_outputs = layer_module( 2025-12-04T08:58:49.7182747Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:49.7182823Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:49.7183073Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7183141Z return func(*args, **kwargs) 2025-12-04T08:58:49.7183397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 711, in forward 2025-12-04T08:58:49.7183479Z cross_attention_outputs = self.layer[1]( 2025-12-04T08:58:49.7183718Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7183807Z return func(*args, **kwargs) 2025-12-04T08:58:49.7184041Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 640, in forward 2025-12-04T08:58:49.7184131Z attention_output = self.EncDecAttention( 2025-12-04T08:58:49.7184370Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7184440Z return func(*args, **kwargs) 2025-12-04T08:58:49.7184703Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 514, in forward 2025-12-04T08:58:49.7184781Z value_states = self.v(current_states) 2025-12-04T08:58:49.7184784Z 2025-12-04T08:58:49.7184864Z cudagraph partition due to non gpu ops 2025-12-04T08:58:49.7184950Z cudagraph partition due to non gpu ops 2025-12-04T08:58:49.7185052Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:49.7185255Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:49.7185318Z res = mod(**inputs) 2025-12-04T08:58:49.7185553Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T08:58:49.7185633Z decoder_outputs = self.decoder( 2025-12-04T08:58:49.7185874Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T08:58:49.7185960Z layer_outputs = layer_module( 2025-12-04T08:58:49.7186191Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:49.7186269Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:49.7186517Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7186584Z return func(*args, **kwargs) 2025-12-04T08:58:49.7186829Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 711, in forward 2025-12-04T08:58:49.7186912Z cross_attention_outputs = self.layer[1]( 2025-12-04T08:58:49.7187142Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7187206Z return func(*args, **kwargs) 2025-12-04T08:58:49.7187440Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 640, in forward 2025-12-04T08:58:49.7187522Z attention_output = self.EncDecAttention( 2025-12-04T08:58:49.7187762Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7187829Z return func(*args, **kwargs) 2025-12-04T08:58:49.7188055Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 572, in forward 2025-12-04T08:58:49.7188136Z attn_output = self.o(attn_output) 2025-12-04T08:58:49.7188139Z 2025-12-04T08:58:49.7188237Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:49.7188436Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:49.7188499Z res = mod(**inputs) 2025-12-04T08:58:49.7188727Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T08:58:49.7188808Z decoder_outputs = self.decoder( 2025-12-04T08:58:49.7189040Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T08:58:49.7189110Z layer_outputs = layer_module( 2025-12-04T08:58:49.7189347Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:49.7189422Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:49.7189674Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7189740Z return func(*args, **kwargs) 2025-12-04T08:58:49.7189968Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 737, in forward 2025-12-04T08:58:49.7190063Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T08:58:49.7190291Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-12-04T08:58:49.7190427Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T08:58:49.7190658Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward 2025-12-04T08:58:49.7190736Z hidden_states = self.wi(hidden_states) 2025-12-04T08:58:49.7190739Z 2025-12-04T08:58:49.7190846Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:49.7191039Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:49.7191102Z res = mod(**inputs) 2025-12-04T08:58:49.7191344Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T08:58:49.7191418Z decoder_outputs = self.decoder( 2025-12-04T08:58:49.7191660Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T08:58:49.7191747Z layer_outputs = layer_module( 2025-12-04T08:58:49.7191972Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:49.7192058Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:49.7192309Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7192374Z return func(*args, **kwargs) 2025-12-04T08:58:49.7192613Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 737, in forward 2025-12-04T08:58:49.7192699Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T08:58:49.7192935Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-12-04T08:58:49.7193045Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T08:58:49.7193279Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 289, in forward 2025-12-04T08:58:49.7193364Z hidden_states = self.act(hidden_states) 2025-12-04T08:58:49.7193367Z 2025-12-04T08:58:49.7193470Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:49.7193668Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:49.7193731Z res = mod(**inputs) 2025-12-04T08:58:49.7193969Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T08:58:49.7194046Z decoder_outputs = self.decoder( 2025-12-04T08:58:49.7194276Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T08:58:49.7194345Z layer_outputs = layer_module( 2025-12-04T08:58:49.7194569Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:49.7194648Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:49.7194897Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7194993Z return func(*args, **kwargs) 2025-12-04T08:58:49.7195229Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 737, in forward 2025-12-04T08:58:49.7195323Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T08:58:49.7195579Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-12-04T08:58:49.7195700Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T08:58:49.7195936Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 297, in forward 2025-12-04T08:58:49.7196015Z hidden_states = self.wo(hidden_states) 2025-12-04T08:58:49.7196020Z 2025-12-04T08:58:49.7196146Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:49.7196341Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:49.7196406Z res = mod(**inputs) 2025-12-04T08:58:49.7196653Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T08:58:49.7196726Z decoder_outputs = self.decoder( 2025-12-04T08:58:49.7196977Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T08:58:49.7197049Z layer_outputs = layer_module( 2025-12-04T08:58:49.7197275Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:49.7197364Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:49.7197608Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7197696Z return func(*args, **kwargs) 2025-12-04T08:58:49.7197949Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T08:58:49.7198035Z self_attention_outputs = self.layer[0]( 2025-12-04T08:58:49.7198362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7198442Z return func(*args, **kwargs) 2025-12-04T08:58:49.7198681Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 603, in forward 2025-12-04T08:58:49.7198783Z attention_output = self.SelfAttention( 2025-12-04T08:58:49.7199053Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7199137Z return func(*args, **kwargs) 2025-12-04T08:58:49.7199402Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 492, in forward 2025-12-04T08:58:49.7199489Z query_states = self.q(hidden_states) 2025-12-04T08:58:49.7199493Z 2025-12-04T08:58:49.7199617Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:49.7199836Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:49.7199907Z res = mod(**inputs) 2025-12-04T08:58:49.7200184Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T08:58:49.7200263Z decoder_outputs = self.decoder( 2025-12-04T08:58:49.7200530Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T08:58:49.7200613Z layer_outputs = layer_module( 2025-12-04T08:58:49.7200834Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:49.7200923Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:49.7201172Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7201259Z return func(*args, **kwargs) 2025-12-04T08:58:49.7201501Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T08:58:49.7201601Z self_attention_outputs = self.layer[0]( 2025-12-04T08:58:49.7201849Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7201917Z return func(*args, **kwargs) 2025-12-04T08:58:49.7202160Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 603, in forward 2025-12-04T08:58:49.7202251Z attention_output = self.SelfAttention( 2025-12-04T08:58:49.7202519Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7202597Z return func(*args, **kwargs) 2025-12-04T08:58:49.7202842Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 513, in forward 2025-12-04T08:58:49.7202919Z key_states = self.k(current_states) 2025-12-04T08:58:49.7202922Z 2025-12-04T08:58:49.7203033Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:49.7203239Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:49.7203305Z res = mod(**inputs) 2025-12-04T08:58:49.7203558Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T08:58:49.7203632Z decoder_outputs = self.decoder( 2025-12-04T08:58:49.7203890Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T08:58:49.7203982Z layer_outputs = layer_module( 2025-12-04T08:58:49.7204208Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:49.7204294Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:49.7204544Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7204613Z return func(*args, **kwargs) 2025-12-04T08:58:49.7204865Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T08:58:49.7204948Z self_attention_outputs = self.layer[0]( 2025-12-04T08:58:49.7205218Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7205288Z return func(*args, **kwargs) 2025-12-04T08:58:49.7205533Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 603, in forward 2025-12-04T08:58:49.7205626Z attention_output = self.SelfAttention( 2025-12-04T08:58:49.7205879Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7205957Z return func(*args, **kwargs) 2025-12-04T08:58:49.7206201Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 514, in forward 2025-12-04T08:58:49.7206291Z value_states = self.v(current_states) 2025-12-04T08:58:49.7206294Z 2025-12-04T08:58:49.7206382Z cudagraph partition due to non gpu ops 2025-12-04T08:58:49.7206460Z cudagraph partition due to non gpu ops 2025-12-04T08:58:49.7206563Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:49.7206769Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:49.7206837Z res = mod(**inputs) 2025-12-04T08:58:49.7207093Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T08:58:49.7207168Z decoder_outputs = self.decoder( 2025-12-04T08:58:49.7207426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T08:58:49.7207507Z layer_outputs = layer_module( 2025-12-04T08:58:49.7207733Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:49.7207846Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:49.7208106Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7208174Z return func(*args, **kwargs) 2025-12-04T08:58:49.7208414Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T08:58:49.7208495Z self_attention_outputs = self.layer[0]( 2025-12-04T08:58:49.7208751Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7208831Z return func(*args, **kwargs) 2025-12-04T08:58:49.7209070Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 603, in forward 2025-12-04T08:58:49.7209150Z attention_output = self.SelfAttention( 2025-12-04T08:58:49.7209397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7209465Z return func(*args, **kwargs) 2025-12-04T08:58:49.7209705Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 572, in forward 2025-12-04T08:58:49.7209780Z attn_output = self.o(attn_output) 2025-12-04T08:58:49.7209800Z 2025-12-04T08:58:49.7209903Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:49.7210108Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:49.7210172Z res = mod(**inputs) 2025-12-04T08:58:49.7210419Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T08:58:49.7210493Z decoder_outputs = self.decoder( 2025-12-04T08:58:49.7210729Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T08:58:49.7210809Z layer_outputs = layer_module( 2025-12-04T08:58:49.7211032Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:49.7211111Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:49.7211361Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7211430Z return func(*args, **kwargs) 2025-12-04T08:58:49.7211672Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T08:58:49.7211751Z self_attention_outputs = self.layer[0]( 2025-12-04T08:58:49.7211992Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7212067Z return func(*args, **kwargs) 2025-12-04T08:58:49.7212303Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 613, in forward 2025-12-04T08:58:49.7212444Z hidden_states = hidden_states + self.dropout(attention_output[0]) 2025-12-04T08:58:49.7212448Z 2025-12-04T08:58:49.7212551Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:49.7212745Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:49.7212819Z res = mod(**inputs) 2025-12-04T08:58:49.7213058Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T08:58:49.7213130Z decoder_outputs = self.decoder( 2025-12-04T08:58:49.7213387Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T08:58:49.7213460Z layer_outputs = layer_module( 2025-12-04T08:58:49.7213687Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:49.7213782Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:49.7214023Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7214097Z return func(*args, **kwargs) 2025-12-04T08:58:49.7214331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 711, in forward 2025-12-04T08:58:49.7214414Z cross_attention_outputs = self.layer[1]( 2025-12-04T08:58:49.7214687Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7214757Z return func(*args, **kwargs) 2025-12-04T08:58:49.7215000Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 640, in forward 2025-12-04T08:58:49.7215083Z attention_output = self.EncDecAttention( 2025-12-04T08:58:49.7215322Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7215397Z return func(*args, **kwargs) 2025-12-04T08:58:49.7215627Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 492, in forward 2025-12-04T08:58:49.7215711Z query_states = self.q(hidden_states) 2025-12-04T08:58:49.7215729Z 2025-12-04T08:58:49.7215832Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:49.7216029Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:49.7216099Z res = mod(**inputs) 2025-12-04T08:58:49.7216337Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T08:58:49.7216409Z decoder_outputs = self.decoder( 2025-12-04T08:58:49.7216654Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T08:58:49.7216725Z layer_outputs = layer_module( 2025-12-04T08:58:49.7216954Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:49.7217030Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:49.7217272Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7217348Z return func(*args, **kwargs) 2025-12-04T08:58:49.7217583Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 711, in forward 2025-12-04T08:58:49.7217663Z cross_attention_outputs = self.layer[1]( 2025-12-04T08:58:49.7217912Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7217980Z return func(*args, **kwargs) 2025-12-04T08:58:49.7218226Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 640, in forward 2025-12-04T08:58:49.7218310Z attention_output = self.EncDecAttention( 2025-12-04T08:58:49.7218551Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7218627Z return func(*args, **kwargs) 2025-12-04T08:58:49.7218861Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 513, in forward 2025-12-04T08:58:49.7218945Z key_states = self.k(current_states) 2025-12-04T08:58:49.7218948Z 2025-12-04T08:58:49.7219051Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:49.7219271Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:49.7219343Z res = mod(**inputs) 2025-12-04T08:58:49.7219585Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T08:58:49.7219674Z decoder_outputs = self.decoder( 2025-12-04T08:58:49.7219920Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T08:58:49.7219991Z layer_outputs = layer_module( 2025-12-04T08:58:49.7220218Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:49.7220297Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:49.7220561Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7220636Z return func(*args, **kwargs) 2025-12-04T08:58:49.7220993Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 711, in forward 2025-12-04T08:58:49.7221078Z cross_attention_outputs = self.layer[1]( 2025-12-04T08:58:49.7221320Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7221386Z return func(*args, **kwargs) 2025-12-04T08:58:49.7221622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 640, in forward 2025-12-04T08:58:49.7221704Z attention_output = self.EncDecAttention( 2025-12-04T08:58:49.7221936Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7222050Z return func(*args, **kwargs) 2025-12-04T08:58:49.7222279Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 514, in forward 2025-12-04T08:58:49.7222364Z value_states = self.v(current_states) 2025-12-04T08:58:49.7222367Z 2025-12-04T08:58:49.7222445Z cudagraph partition due to non gpu ops 2025-12-04T08:58:49.7222521Z cudagraph partition due to non gpu ops 2025-12-04T08:58:49.7222630Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:49.7222821Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:49.7222884Z res = mod(**inputs) 2025-12-04T08:58:49.7223119Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T08:58:49.7223188Z decoder_outputs = self.decoder( 2025-12-04T08:58:49.7223432Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T08:58:49.7223501Z layer_outputs = layer_module( 2025-12-04T08:58:49.7223718Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:49.7223800Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:49.7224031Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7224101Z return func(*args, **kwargs) 2025-12-04T08:58:49.7224338Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 711, in forward 2025-12-04T08:58:49.7224418Z cross_attention_outputs = self.layer[1]( 2025-12-04T08:58:49.7224659Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7224727Z return func(*args, **kwargs) 2025-12-04T08:58:49.7224956Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 640, in forward 2025-12-04T08:58:49.7225045Z attention_output = self.EncDecAttention( 2025-12-04T08:58:49.7225300Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7225366Z return func(*args, **kwargs) 2025-12-04T08:58:49.7225604Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 572, in forward 2025-12-04T08:58:49.7225701Z attn_output = self.o(attn_output) 2025-12-04T08:58:49.7225705Z 2025-12-04T08:58:49.7225810Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:49.7226006Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:49.7226069Z res = mod(**inputs) 2025-12-04T08:58:49.7226332Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T08:58:49.7226404Z decoder_outputs = self.decoder( 2025-12-04T08:58:49.7226642Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T08:58:49.7226710Z layer_outputs = layer_module( 2025-12-04T08:58:49.7226922Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:49.7227007Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:49.7227242Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7227309Z return func(*args, **kwargs) 2025-12-04T08:58:49.7227546Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 737, in forward 2025-12-04T08:58:49.7227651Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T08:58:49.7227891Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-12-04T08:58:49.7228004Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T08:58:49.7228236Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward 2025-12-04T08:58:49.7228319Z hidden_states = self.wi(hidden_states) 2025-12-04T08:58:49.7228324Z 2025-12-04T08:58:49.7228422Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:49.7228620Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:49.7228684Z res = mod(**inputs) 2025-12-04T08:58:49.7228918Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T08:58:49.7228995Z decoder_outputs = self.decoder( 2025-12-04T08:58:49.7229227Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T08:58:49.7229297Z layer_outputs = layer_module( 2025-12-04T08:58:49.7229521Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:49.7229594Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:49.7229832Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7229899Z return func(*args, **kwargs) 2025-12-04T08:58:49.7230126Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 737, in forward 2025-12-04T08:58:49.7230219Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T08:58:49.7230450Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-12-04T08:58:49.7230564Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T08:58:49.7230801Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 289, in forward 2025-12-04T08:58:49.7230902Z hidden_states = self.act(hidden_states) 2025-12-04T08:58:49.7230906Z 2025-12-04T08:58:49.7231013Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:49.7231211Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:49.7231290Z res = mod(**inputs) 2025-12-04T08:58:49.7231538Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T08:58:49.7231609Z decoder_outputs = self.decoder( 2025-12-04T08:58:49.7231849Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T08:58:49.7231920Z layer_outputs = layer_module( 2025-12-04T08:58:49.7232160Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:49.7232244Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:49.7232487Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7232557Z return func(*args, **kwargs) 2025-12-04T08:58:49.7232799Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 737, in forward 2025-12-04T08:58:49.7232890Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T08:58:49.7233132Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-12-04T08:58:49.7233247Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T08:58:49.7233481Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 297, in forward 2025-12-04T08:58:49.7233583Z hidden_states = self.wo(hidden_states) 2025-12-04T08:58:49.7233588Z 2025-12-04T08:58:49.7233690Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:49.7233894Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:49.7233959Z res = mod(**inputs) 2025-12-04T08:58:49.7234199Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T08:58:49.7234284Z decoder_outputs = self.decoder( 2025-12-04T08:58:49.7234526Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T08:58:49.7234600Z layer_outputs = layer_module( 2025-12-04T08:58:49.7234833Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:49.7234915Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:49.7235169Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7235240Z return func(*args, **kwargs) 2025-12-04T08:58:49.7235482Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T08:58:49.7235574Z self_attention_outputs = self.layer[0]( 2025-12-04T08:58:49.7235824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7235897Z return func(*args, **kwargs) 2025-12-04T08:58:49.7236151Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 603, in forward 2025-12-04T08:58:49.7236240Z attention_output = self.SelfAttention( 2025-12-04T08:58:49.7236508Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7236584Z return func(*args, **kwargs) 2025-12-04T08:58:49.7236839Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 492, in forward 2025-12-04T08:58:49.7236945Z query_states = self.q(hidden_states) 2025-12-04T08:58:49.7236949Z 2025-12-04T08:58:49.7237061Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:49.7237285Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:49.7237373Z res = mod(**inputs) 2025-12-04T08:58:49.7237642Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T08:58:49.7237728Z decoder_outputs = self.decoder( 2025-12-04T08:58:49.7237987Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T08:58:49.7238064Z layer_outputs = layer_module( 2025-12-04T08:58:49.7238397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:49.7238488Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:49.7238770Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7238846Z return func(*args, **kwargs) 2025-12-04T08:58:49.7239111Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T08:58:49.7239219Z self_attention_outputs = self.layer[0]( 2025-12-04T08:58:49.7239468Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7239539Z return func(*args, **kwargs) 2025-12-04T08:58:49.7239792Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 603, in forward 2025-12-04T08:58:49.7239892Z attention_output = self.SelfAttention( 2025-12-04T08:58:49.7240146Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7240216Z return func(*args, **kwargs) 2025-12-04T08:58:49.7240455Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 513, in forward 2025-12-04T08:58:49.7240542Z key_states = self.k(current_states) 2025-12-04T08:58:49.7240547Z 2025-12-04T08:58:49.7240651Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:49.7240860Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:49.7240926Z res = mod(**inputs) 2025-12-04T08:58:49.7241169Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T08:58:49.7241253Z decoder_outputs = self.decoder( 2025-12-04T08:58:49.7241495Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T08:58:49.7241568Z layer_outputs = layer_module( 2025-12-04T08:58:49.7241804Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:49.7241883Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:49.7242135Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7242205Z return func(*args, **kwargs) 2025-12-04T08:58:49.7242443Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T08:58:49.7242532Z self_attention_outputs = self.layer[0]( 2025-12-04T08:58:49.7242778Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7242848Z return func(*args, **kwargs) 2025-12-04T08:58:49.7243097Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 603, in forward 2025-12-04T08:58:49.7243180Z attention_output = self.SelfAttention( 2025-12-04T08:58:49.7243447Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7243516Z return func(*args, **kwargs) 2025-12-04T08:58:49.7243772Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 514, in forward 2025-12-04T08:58:49.7243861Z value_states = self.v(current_states) 2025-12-04T08:58:49.7243864Z 2025-12-04T08:58:49.7243946Z cudagraph partition due to non gpu ops 2025-12-04T08:58:49.7244034Z cudagraph partition due to non gpu ops 2025-12-04T08:58:49.7244139Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:49.7244342Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:49.7244432Z res = mod(**inputs) 2025-12-04T08:58:49.7244681Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T08:58:49.7244755Z decoder_outputs = self.decoder( 2025-12-04T08:58:49.7245008Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T08:58:49.7245082Z layer_outputs = layer_module( 2025-12-04T08:58:49.7245318Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:49.7245398Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:49.7245644Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7245721Z return func(*args, **kwargs) 2025-12-04T08:58:49.7245980Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T08:58:49.7246060Z self_attention_outputs = self.layer[0]( 2025-12-04T08:58:49.7246314Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7246382Z return func(*args, **kwargs) 2025-12-04T08:58:49.7246628Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 603, in forward 2025-12-04T08:58:49.7246713Z attention_output = self.SelfAttention( 2025-12-04T08:58:49.7246958Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7247035Z return func(*args, **kwargs) 2025-12-04T08:58:49.7247277Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 572, in forward 2025-12-04T08:58:49.7247363Z attn_output = self.o(attn_output) 2025-12-04T08:58:49.7247367Z 2025-12-04T08:58:49.7247472Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:49.7247673Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:49.7247748Z res = mod(**inputs) 2025-12-04T08:58:49.7247991Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T08:58:49.7248066Z decoder_outputs = self.decoder( 2025-12-04T08:58:49.7248319Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T08:58:49.7248391Z layer_outputs = layer_module( 2025-12-04T08:58:49.7248626Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:49.7248707Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:49.7248954Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7249034Z return func(*args, **kwargs) 2025-12-04T08:58:49.7249289Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 711, in forward 2025-12-04T08:58:49.7249375Z cross_attention_outputs = self.layer[1]( 2025-12-04T08:58:49.7249628Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7249712Z return func(*args, **kwargs) 2025-12-04T08:58:49.7249949Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 640, in forward 2025-12-04T08:58:49.7250033Z attention_output = self.EncDecAttention( 2025-12-04T08:58:49.7250272Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7250349Z return func(*args, **kwargs) 2025-12-04T08:58:49.7250604Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 492, in forward 2025-12-04T08:58:49.7250688Z query_states = self.q(hidden_states) 2025-12-04T08:58:49.7250691Z 2025-12-04T08:58:49.7250792Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:49.7250985Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:49.7251056Z res = mod(**inputs) 2025-12-04T08:58:49.7251289Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T08:58:49.7251360Z decoder_outputs = self.decoder( 2025-12-04T08:58:49.7251601Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T08:58:49.7251670Z layer_outputs = layer_module( 2025-12-04T08:58:49.7251918Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:49.7251998Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:49.7252244Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7252315Z return func(*args, **kwargs) 2025-12-04T08:58:49.7252550Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 711, in forward 2025-12-04T08:58:49.7252626Z cross_attention_outputs = self.layer[1]( 2025-12-04T08:58:49.7252866Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7252932Z return func(*args, **kwargs) 2025-12-04T08:58:49.7253166Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 640, in forward 2025-12-04T08:58:49.7253248Z attention_output = self.EncDecAttention( 2025-12-04T08:58:49.7253483Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7253557Z return func(*args, **kwargs) 2025-12-04T08:58:49.7253787Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 513, in forward 2025-12-04T08:58:49.7253865Z key_states = self.k(current_states) 2025-12-04T08:58:49.7253868Z 2025-12-04T08:58:49.7253967Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:49.7254159Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:49.7254230Z res = mod(**inputs) 2025-12-04T08:58:49.7254459Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T08:58:49.7254531Z decoder_outputs = self.decoder( 2025-12-04T08:58:49.7254770Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T08:58:49.7254840Z layer_outputs = layer_module( 2025-12-04T08:58:49.7255062Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:49.7255151Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:49.7255384Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7255475Z return func(*args, **kwargs) 2025-12-04T08:58:49.7255705Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 711, in forward 2025-12-04T08:58:49.7255784Z cross_attention_outputs = self.layer[1]( 2025-12-04T08:58:49.7256029Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7256096Z return func(*args, **kwargs) 2025-12-04T08:58:49.7256350Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 640, in forward 2025-12-04T08:58:49.7256432Z attention_output = self.EncDecAttention( 2025-12-04T08:58:49.7256677Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7256751Z return func(*args, **kwargs) 2025-12-04T08:58:49.7256980Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 514, in forward 2025-12-04T08:58:49.7257062Z value_states = self.v(current_states) 2025-12-04T08:58:49.7257065Z 2025-12-04T08:58:49.7257141Z cudagraph partition due to non gpu ops 2025-12-04T08:58:49.7257216Z cudagraph partition due to non gpu ops 2025-12-04T08:58:49.7257322Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:49.7257512Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:49.7257592Z res = mod(**inputs) 2025-12-04T08:58:49.7257837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T08:58:49.7257909Z decoder_outputs = self.decoder( 2025-12-04T08:58:49.7258149Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T08:58:49.7258220Z layer_outputs = layer_module( 2025-12-04T08:58:49.7258441Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:49.7258524Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:49.7258764Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7258829Z return func(*args, **kwargs) 2025-12-04T08:58:49.7259072Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 711, in forward 2025-12-04T08:58:49.7259154Z cross_attention_outputs = self.layer[1]( 2025-12-04T08:58:49.7259402Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7259471Z return func(*args, **kwargs) 2025-12-04T08:58:49.7259705Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 640, in forward 2025-12-04T08:58:49.7259796Z attention_output = self.EncDecAttention( 2025-12-04T08:58:49.7260046Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7260113Z return func(*args, **kwargs) 2025-12-04T08:58:49.7260353Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 572, in forward 2025-12-04T08:58:49.7260427Z attn_output = self.o(attn_output) 2025-12-04T08:58:49.7260433Z 2025-12-04T08:58:49.7260540Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:49.7260736Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:49.7260798Z res = mod(**inputs) 2025-12-04T08:58:49.7261056Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T08:58:49.7261128Z decoder_outputs = self.decoder( 2025-12-04T08:58:49.7261369Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T08:58:49.7261456Z layer_outputs = layer_module( 2025-12-04T08:58:49.7261671Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:49.7261752Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:49.7261987Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7262056Z return func(*args, **kwargs) 2025-12-04T08:58:49.7262311Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 711, in forward 2025-12-04T08:58:49.7262391Z cross_attention_outputs = self.layer[1]( 2025-12-04T08:58:49.7262629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7262696Z return func(*args, **kwargs) 2025-12-04T08:58:49.7262925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 652, in forward 2025-12-04T08:58:49.7263061Z layer_output = hidden_states + self.dropout(attention_output[0]) 2025-12-04T08:58:49.7263065Z 2025-12-04T08:58:49.7263163Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:49.7263367Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:49.7263450Z res = mod(**inputs) 2025-12-04T08:58:49.7263690Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T08:58:49.7263773Z decoder_outputs = self.decoder( 2025-12-04T08:58:49.7264012Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T08:58:49.7264084Z layer_outputs = layer_module( 2025-12-04T08:58:49.7264314Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:49.7264391Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:49.7264643Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7264711Z return func(*args, **kwargs) 2025-12-04T08:58:49.7264988Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 737, in forward 2025-12-04T08:58:49.7265087Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T08:58:49.7265316Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-12-04T08:58:49.7265432Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T08:58:49.7265677Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward 2025-12-04T08:58:49.7265757Z hidden_states = self.wi(hidden_states) 2025-12-04T08:58:49.7265760Z 2025-12-04T08:58:49.7265869Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:49.7266067Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:49.7266132Z res = mod(**inputs) 2025-12-04T08:58:49.7266377Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T08:58:49.7266450Z decoder_outputs = self.decoder( 2025-12-04T08:58:49.7266694Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T08:58:49.7266765Z layer_outputs = layer_module( 2025-12-04T08:58:49.7267000Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:49.7267088Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:49.7267346Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7267413Z return func(*args, **kwargs) 2025-12-04T08:58:49.7267655Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 737, in forward 2025-12-04T08:58:49.7267746Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T08:58:49.7267992Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-12-04T08:58:49.7268126Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T08:58:49.7268362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 289, in forward 2025-12-04T08:58:49.7268450Z hidden_states = self.act(hidden_states) 2025-12-04T08:58:49.7268454Z 2025-12-04T08:58:49.7268555Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:49.7268762Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:49.7268827Z res = mod(**inputs) 2025-12-04T08:58:49.7269068Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T08:58:49.7269149Z decoder_outputs = self.decoder( 2025-12-04T08:58:49.7269387Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T08:58:49.7269480Z layer_outputs = layer_module( 2025-12-04T08:58:49.7269710Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:49.7269789Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:49.7270035Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7270104Z return func(*args, **kwargs) 2025-12-04T08:58:49.7270340Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 737, in forward 2025-12-04T08:58:49.7270438Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T08:58:49.7270672Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-12-04T08:58:49.7270784Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T08:58:49.7271026Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 297, in forward 2025-12-04T08:58:49.7271104Z hidden_states = self.wo(hidden_states) 2025-12-04T08:58:49.7271108Z 2025-12-04T08:58:49.7271218Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:49.7271412Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:49.7271477Z res = mod(**inputs) 2025-12-04T08:58:49.7271722Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T08:58:49.7271794Z decoder_outputs = self.decoder( 2025-12-04T08:58:49.7272087Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T08:58:49.7272159Z layer_outputs = layer_module( 2025-12-04T08:58:49.7272378Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:49.7272464Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:49.7272713Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7272833Z return func(*args, **kwargs) 2025-12-04T08:58:49.7273075Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T08:58:49.7273172Z self_attention_outputs = self.layer[0]( 2025-12-04T08:58:49.7273421Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7273488Z return func(*args, **kwargs) 2025-12-04T08:58:49.7273721Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 603, in forward 2025-12-04T08:58:49.7273813Z attention_output = self.SelfAttention( 2025-12-04T08:58:49.7274077Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7274146Z return func(*args, **kwargs) 2025-12-04T08:58:49.7274389Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 492, in forward 2025-12-04T08:58:49.7274465Z query_states = self.q(hidden_states) 2025-12-04T08:58:49.7274469Z 2025-12-04T08:58:49.7274580Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:49.7274780Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:49.7274845Z res = mod(**inputs) 2025-12-04T08:58:49.7275091Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T08:58:49.7275163Z decoder_outputs = self.decoder( 2025-12-04T08:58:49.7275409Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T08:58:49.7275496Z layer_outputs = layer_module( 2025-12-04T08:58:49.7275719Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:49.7275807Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:49.7276045Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7276114Z return func(*args, **kwargs) 2025-12-04T08:58:49.7276359Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T08:58:49.7276439Z self_attention_outputs = self.layer[0]( 2025-12-04T08:58:49.7276685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7276752Z return func(*args, **kwargs) 2025-12-04T08:58:49.7276988Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 603, in forward 2025-12-04T08:58:49.7277081Z attention_output = self.SelfAttention( 2025-12-04T08:58:49.7277323Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7277392Z return func(*args, **kwargs) 2025-12-04T08:58:49.7277633Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 513, in forward 2025-12-04T08:58:49.7277712Z key_states = self.k(current_states) 2025-12-04T08:58:49.7277716Z 2025-12-04T08:58:49.7277825Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:49.7278024Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:49.7278090Z res = mod(**inputs) 2025-12-04T08:58:49.7278423Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T08:58:49.7278507Z decoder_outputs = self.decoder( 2025-12-04T08:58:49.7278770Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T08:58:49.7278863Z layer_outputs = layer_module( 2025-12-04T08:58:49.7279090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:49.7279180Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:49.7279443Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7279515Z return func(*args, **kwargs) 2025-12-04T08:58:49.7279769Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T08:58:49.7279856Z self_attention_outputs = self.layer[0]( 2025-12-04T08:58:49.7280129Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7280220Z return func(*args, **kwargs) 2025-12-04T08:58:49.7280477Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 603, in forward 2025-12-04T08:58:49.7280576Z attention_output = self.SelfAttention( 2025-12-04T08:58:49.7280839Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7280916Z return func(*args, **kwargs) 2025-12-04T08:58:49.7281181Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 514, in forward 2025-12-04T08:58:49.7281272Z value_states = self.v(current_states) 2025-12-04T08:58:49.7281275Z 2025-12-04T08:58:49.7281365Z cudagraph partition due to non gpu ops 2025-12-04T08:58:49.7281448Z cudagraph partition due to non gpu ops 2025-12-04T08:58:49.7281570Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:49.7281782Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:49.7281849Z res = mod(**inputs) 2025-12-04T08:58:49.7282106Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T08:58:49.7282182Z decoder_outputs = self.decoder( 2025-12-04T08:58:49.7282422Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T08:58:49.7282515Z layer_outputs = layer_module( 2025-12-04T08:58:49.7282737Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:49.7282815Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:49.7283062Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7283133Z return func(*args, **kwargs) 2025-12-04T08:58:49.7283380Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T08:58:49.7283462Z self_attention_outputs = self.layer[0]( 2025-12-04T08:58:49.7283711Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7283786Z return func(*args, **kwargs) 2025-12-04T08:58:49.7284026Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 603, in forward 2025-12-04T08:58:49.7284109Z attention_output = self.SelfAttention( 2025-12-04T08:58:49.7284363Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7284432Z return func(*args, **kwargs) 2025-12-04T08:58:49.7284678Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 572, in forward 2025-12-04T08:58:49.7284761Z attn_output = self.o(attn_output) 2025-12-04T08:58:49.7284765Z 2025-12-04T08:58:49.7284868Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:49.7285096Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:49.7285165Z res = mod(**inputs) 2025-12-04T08:58:49.7285416Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T08:58:49.7285508Z decoder_outputs = self.decoder( 2025-12-04T08:58:49.7285748Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T08:58:49.7285827Z layer_outputs = layer_module( 2025-12-04T08:58:49.7286053Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:49.7286135Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:49.7286402Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7286472Z return func(*args, **kwargs) 2025-12-04T08:58:49.7286722Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 711, in forward 2025-12-04T08:58:49.7286805Z cross_attention_outputs = self.layer[1]( 2025-12-04T08:58:49.7287052Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7287128Z return func(*args, **kwargs) 2025-12-04T08:58:49.7287368Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 640, in forward 2025-12-04T08:58:49.7287453Z attention_output = self.EncDecAttention( 2025-12-04T08:58:49.7287707Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7287794Z return func(*args, **kwargs) 2025-12-04T08:58:49.7288044Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 492, in forward 2025-12-04T08:58:49.7288124Z query_states = self.q(hidden_states) 2025-12-04T08:58:49.7288128Z 2025-12-04T08:58:49.7288233Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:49.7288444Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:49.7288512Z res = mod(**inputs) 2025-12-04T08:58:49.7288765Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T08:58:49.7288838Z decoder_outputs = self.decoder( 2025-12-04T08:58:49.7289086Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T08:58:49.7289166Z layer_outputs = layer_module( 2025-12-04T08:58:49.7289395Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:49.7289473Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:49.7289729Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7289798Z return func(*args, **kwargs) 2025-12-04T08:58:49.7290049Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 711, in forward 2025-12-04T08:58:49.7290133Z cross_attention_outputs = self.layer[1]( 2025-12-04T08:58:49.7290382Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7290458Z return func(*args, **kwargs) 2025-12-04T08:58:49.7290701Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 640, in forward 2025-12-04T08:58:49.7290790Z attention_output = self.EncDecAttention( 2025-12-04T08:58:49.7291047Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7291132Z return func(*args, **kwargs) 2025-12-04T08:58:49.7291380Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 513, in forward 2025-12-04T08:58:49.7291459Z key_states = self.k(current_states) 2025-12-04T08:58:49.7291501Z 2025-12-04T08:58:49.7291607Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:49.7291818Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:49.7291883Z res = mod(**inputs) 2025-12-04T08:58:49.7292137Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T08:58:49.7292213Z decoder_outputs = self.decoder( 2025-12-04T08:58:49.7292475Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T08:58:49.7292558Z layer_outputs = layer_module( 2025-12-04T08:58:49.7292786Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:49.7292865Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:49.7293117Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7293190Z return func(*args, **kwargs) 2025-12-04T08:58:49.7293439Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 711, in forward 2025-12-04T08:58:49.7293529Z cross_attention_outputs = self.layer[1]( 2025-12-04T08:58:49.7293761Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7293850Z return func(*args, **kwargs) 2025-12-04T08:58:49.7294079Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 640, in forward 2025-12-04T08:58:49.7294162Z attention_output = self.EncDecAttention( 2025-12-04T08:58:49.7294403Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7294469Z return func(*args, **kwargs) 2025-12-04T08:58:49.7294709Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 514, in forward 2025-12-04T08:58:49.7294784Z value_states = self.v(current_states) 2025-12-04T08:58:49.7294787Z 2025-12-04T08:58:49.7294864Z cudagraph partition due to non gpu ops 2025-12-04T08:58:49.7294950Z cudagraph partition due to non gpu ops 2025-12-04T08:58:49.7295050Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:49.7295247Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:49.7295312Z res = mod(**inputs) 2025-12-04T08:58:49.7295543Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T08:58:49.7295620Z decoder_outputs = self.decoder( 2025-12-04T08:58:49.7295848Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T08:58:49.7295923Z layer_outputs = layer_module( 2025-12-04T08:58:49.7296152Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:49.7296229Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:49.7296474Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7296543Z return func(*args, **kwargs) 2025-12-04T08:58:49.7296779Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 711, in forward 2025-12-04T08:58:49.7296866Z cross_attention_outputs = self.layer[1]( 2025-12-04T08:58:49.7297118Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7297197Z return func(*args, **kwargs) 2025-12-04T08:58:49.7297432Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 640, in forward 2025-12-04T08:58:49.7297542Z attention_output = self.EncDecAttention( 2025-12-04T08:58:49.7297783Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7297848Z return func(*args, **kwargs) 2025-12-04T08:58:49.7298076Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 572, in forward 2025-12-04T08:58:49.7298161Z attn_output = self.o(attn_output) 2025-12-04T08:58:49.7298164Z 2025-12-04T08:58:49.7298280Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:49.7298480Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:49.7298544Z res = mod(**inputs) 2025-12-04T08:58:49.7298773Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T08:58:49.7298855Z decoder_outputs = self.decoder( 2025-12-04T08:58:49.7299082Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T08:58:49.7299152Z layer_outputs = layer_module( 2025-12-04T08:58:49.7299376Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:49.7299451Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:49.7299709Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7299776Z return func(*args, **kwargs) 2025-12-04T08:58:49.7300005Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 737, in forward 2025-12-04T08:58:49.7300102Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T08:58:49.7300334Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-12-04T08:58:49.7300450Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T08:58:49.7300684Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward 2025-12-04T08:58:49.7300762Z hidden_states = self.wi(hidden_states) 2025-12-04T08:58:49.7300766Z 2025-12-04T08:58:49.7300871Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:49.7301069Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:49.7301133Z res = mod(**inputs) 2025-12-04T08:58:49.7301375Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T08:58:49.7301444Z decoder_outputs = self.decoder( 2025-12-04T08:58:49.7301682Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T08:58:49.7301752Z layer_outputs = layer_module( 2025-12-04T08:58:49.7301968Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:49.7302050Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:49.7302284Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7302352Z return func(*args, **kwargs) 2025-12-04T08:58:49.7302587Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 737, in forward 2025-12-04T08:58:49.7302676Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T08:58:49.7302925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-12-04T08:58:49.7303039Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T08:58:49.7303267Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 289, in forward 2025-12-04T08:58:49.7303370Z hidden_states = self.act(hidden_states) 2025-12-04T08:58:49.7303374Z 2025-12-04T08:58:49.7303473Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:49.7303669Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:49.7303732Z res = mod(**inputs) 2025-12-04T08:58:49.7303980Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T08:58:49.7304059Z decoder_outputs = self.decoder( 2025-12-04T08:58:49.7304290Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T08:58:49.7304359Z layer_outputs = layer_module( 2025-12-04T08:58:49.7304580Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:49.7304657Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:49.7304896Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7304964Z return func(*args, **kwargs) 2025-12-04T08:58:49.7305191Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 737, in forward 2025-12-04T08:58:49.7305305Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T08:58:49.7305534Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-12-04T08:58:49.7305644Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T08:58:49.7305882Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 297, in forward 2025-12-04T08:58:49.7305958Z hidden_states = self.wo(hidden_states) 2025-12-04T08:58:49.7305963Z 2025-12-04T08:58:49.7306070Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:49.7306262Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:49.7306326Z res = mod(**inputs) 2025-12-04T08:58:49.7306565Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T08:58:49.7306635Z decoder_outputs = self.decoder( 2025-12-04T08:58:49.7306876Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T08:58:49.7306946Z layer_outputs = layer_module( 2025-12-04T08:58:49.7307160Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:49.7307246Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:49.7307477Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7307547Z return func(*args, **kwargs) 2025-12-04T08:58:49.7307792Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 737, in forward 2025-12-04T08:58:49.7307878Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T08:58:49.7308116Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 344, in forward 2025-12-04T08:58:49.7308243Z hidden_states = hidden_states + self.dropout(forwarded_states) 2025-12-04T08:58:49.7308247Z 2025-12-04T08:58:49.7308348Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:49.7308561Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:49.7308626Z res = mod(**inputs) 2025-12-04T08:58:49.7308866Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T08:58:49.7308952Z decoder_outputs = self.decoder( 2025-12-04T08:58:49.7309183Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T08:58:49.7309266Z layer_outputs = layer_module( 2025-12-04T08:58:49.7309482Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:49.7309561Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:49.7309822Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7309889Z return func(*args, **kwargs) 2025-12-04T08:58:49.7310170Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T08:58:49.7310251Z self_attention_outputs = self.layer[0]( 2025-12-04T08:58:49.7310486Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7310563Z return func(*args, **kwargs) 2025-12-04T08:58:49.7310795Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 603, in forward 2025-12-04T08:58:49.7310875Z attention_output = self.SelfAttention( 2025-12-04T08:58:49.7311119Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7311202Z return func(*args, **kwargs) 2025-12-04T08:58:49.7311441Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 492, in forward 2025-12-04T08:58:49.7311519Z query_states = self.q(hidden_states) 2025-12-04T08:58:49.7311523Z 2025-12-04T08:58:49.7311622Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:49.7311820Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:49.7311885Z res = mod(**inputs) 2025-12-04T08:58:49.7312121Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T08:58:49.7312191Z decoder_outputs = self.decoder( 2025-12-04T08:58:49.7312422Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T08:58:49.7312500Z layer_outputs = layer_module( 2025-12-04T08:58:49.7312715Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:49.7312790Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:49.7313033Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7313101Z return func(*args, **kwargs) 2025-12-04T08:58:49.7313349Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T08:58:49.7313428Z self_attention_outputs = self.layer[0]( 2025-12-04T08:58:49.7313662Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7313735Z return func(*args, **kwargs) 2025-12-04T08:58:49.7313963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 603, in forward 2025-12-04T08:58:49.7314046Z attention_output = self.SelfAttention( 2025-12-04T08:58:49.7314290Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7314356Z return func(*args, **kwargs) 2025-12-04T08:58:49.7314607Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 513, in forward 2025-12-04T08:58:49.7314683Z key_states = self.k(current_states) 2025-12-04T08:58:49.7314702Z 2025-12-04T08:58:49.7314801Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:49.7315000Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:49.7315063Z res = mod(**inputs) 2025-12-04T08:58:49.7315295Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T08:58:49.7315366Z decoder_outputs = self.decoder( 2025-12-04T08:58:49.7315616Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T08:58:49.7315693Z layer_outputs = layer_module( 2025-12-04T08:58:49.7315910Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:49.7315985Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:49.7316232Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7316299Z return func(*args, **kwargs) 2025-12-04T08:58:49.7316541Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T08:58:49.7316622Z self_attention_outputs = self.layer[0]( 2025-12-04T08:58:49.7316861Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7316951Z return func(*args, **kwargs) 2025-12-04T08:58:49.7317185Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 603, in forward 2025-12-04T08:58:49.7317267Z attention_output = self.SelfAttention( 2025-12-04T08:58:49.7317515Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7317582Z return func(*args, **kwargs) 2025-12-04T08:58:49.7317824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 514, in forward 2025-12-04T08:58:49.7317900Z value_states = self.v(current_states) 2025-12-04T08:58:49.7317903Z 2025-12-04T08:58:49.7317982Z cudagraph partition due to non gpu ops 2025-12-04T08:58:49.7318067Z cudagraph partition due to non gpu ops 2025-12-04T08:58:49.7318355Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:49.7318590Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:49.7318664Z res = mod(**inputs) 2025-12-04T08:58:49.7318932Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T08:58:49.7319024Z decoder_outputs = self.decoder( 2025-12-04T08:58:49.7319289Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T08:58:49.7319369Z layer_outputs = layer_module( 2025-12-04T08:58:49.7319632Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:49.7319719Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:49.7320007Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7320077Z return func(*args, **kwargs) 2025-12-04T08:58:49.7320324Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 687, in forward 2025-12-04T08:58:49.7320416Z self_attention_outputs = self.layer[0]( 2025-12-04T08:58:49.7320790Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7320874Z return func(*args, **kwargs) 2025-12-04T08:58:49.7321127Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 603, in forward 2025-12-04T08:58:49.7321255Z attention_output = self.SelfAttention( 2025-12-04T08:58:49.7321509Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7321580Z return func(*args, **kwargs) 2025-12-04T08:58:49.7321820Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 572, in forward 2025-12-04T08:58:49.7321910Z attn_output = self.o(attn_output) 2025-12-04T08:58:49.7321915Z 2025-12-04T08:58:49.7322045Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:49.7322257Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:49.7322326Z res = mod(**inputs) 2025-12-04T08:58:49.7322568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T08:58:49.7322650Z decoder_outputs = self.decoder( 2025-12-04T08:58:49.7322903Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T08:58:49.7322977Z layer_outputs = layer_module( 2025-12-04T08:58:49.7323216Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:49.7323300Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:49.7323584Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7323657Z return func(*args, **kwargs) 2025-12-04T08:58:49.7323895Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 711, in forward 2025-12-04T08:58:49.7323986Z cross_attention_outputs = self.layer[1]( 2025-12-04T08:58:49.7324231Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7324302Z return func(*args, **kwargs) 2025-12-04T08:58:49.7324550Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 640, in forward 2025-12-04T08:58:49.7324635Z attention_output = self.EncDecAttention( 2025-12-04T08:58:49.7324886Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7324956Z return func(*args, **kwargs) 2025-12-04T08:58:49.7325193Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 492, in forward 2025-12-04T08:58:49.7325290Z query_states = self.q(hidden_states) 2025-12-04T08:58:49.7325293Z 2025-12-04T08:58:49.7325396Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:49.7325598Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:49.7325662Z res = mod(**inputs) 2025-12-04T08:58:49.7325899Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T08:58:49.7325977Z decoder_outputs = self.decoder( 2025-12-04T08:58:49.7326211Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T08:58:49.7326281Z layer_outputs = layer_module( 2025-12-04T08:58:49.7326512Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:49.7326590Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:49.7326856Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7326926Z return func(*args, **kwargs) 2025-12-04T08:58:49.7327162Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 711, in forward 2025-12-04T08:58:49.7327267Z cross_attention_outputs = self.layer[1]( 2025-12-04T08:58:49.7327510Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7327578Z return func(*args, **kwargs) 2025-12-04T08:58:49.7327821Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 640, in forward 2025-12-04T08:58:49.7327906Z attention_output = self.EncDecAttention( 2025-12-04T08:58:49.7328192Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7328261Z return func(*args, **kwargs) 2025-12-04T08:58:49.7328499Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 513, in forward 2025-12-04T08:58:49.7328583Z key_states = self.k(current_states) 2025-12-04T08:58:49.7328586Z 2025-12-04T08:58:49.7328688Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:49.7328894Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:49.7328957Z res = mod(**inputs) 2025-12-04T08:58:49.7329197Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T08:58:49.7329276Z decoder_outputs = self.decoder( 2025-12-04T08:58:49.7329512Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T08:58:49.7329599Z layer_outputs = layer_module( 2025-12-04T08:58:49.7329831Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:49.7329910Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:49.7330157Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7330227Z return func(*args, **kwargs) 2025-12-04T08:58:49.7330495Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 711, in forward 2025-12-04T08:58:49.7330584Z cross_attention_outputs = self.layer[1]( 2025-12-04T08:58:49.7330829Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7330899Z return func(*args, **kwargs) 2025-12-04T08:58:49.7331151Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 640, in forward 2025-12-04T08:58:49.7331236Z attention_output = self.EncDecAttention( 2025-12-04T08:58:49.7331491Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7331560Z return func(*args, **kwargs) 2025-12-04T08:58:49.7331801Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 514, in forward 2025-12-04T08:58:49.7331889Z value_states = self.v(current_states) 2025-12-04T08:58:49.7331892Z 2025-12-04T08:58:49.7331972Z cudagraph partition due to non gpu ops 2025-12-04T08:58:49.7332059Z cudagraph partition due to non gpu ops 2025-12-04T08:58:49.7332164Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:49.7332367Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:49.7332451Z res = mod(**inputs) 2025-12-04T08:58:49.7332691Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T08:58:49.7332763Z decoder_outputs = self.decoder( 2025-12-04T08:58:49.7333024Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T08:58:49.7333097Z layer_outputs = layer_module( 2025-12-04T08:58:49.7333338Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:49.7333434Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:49.7333681Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7333758Z return func(*args, **kwargs) 2025-12-04T08:58:49.7334002Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 711, in forward 2025-12-04T08:58:49.7334103Z cross_attention_outputs = self.layer[1]( 2025-12-04T08:58:49.7334359Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7334431Z return func(*args, **kwargs) 2025-12-04T08:58:49.7334678Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 640, in forward 2025-12-04T08:58:49.7334764Z attention_output = self.EncDecAttention( 2025-12-04T08:58:49.7335014Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7335091Z return func(*args, **kwargs) 2025-12-04T08:58:49.7335343Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 572, in forward 2025-12-04T08:58:49.7335420Z attn_output = self.o(attn_output) 2025-12-04T08:58:49.7335447Z 2025-12-04T08:58:49.7335550Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:49.7335751Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:49.7335823Z res = mod(**inputs) 2025-12-04T08:58:49.7336065Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T08:58:49.7336137Z decoder_outputs = self.decoder( 2025-12-04T08:58:49.7336382Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T08:58:49.7336455Z layer_outputs = layer_module( 2025-12-04T08:58:49.7336683Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:49.7336761Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:49.7337003Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7337081Z return func(*args, **kwargs) 2025-12-04T08:58:49.7337319Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 737, in forward 2025-12-04T08:58:49.7337412Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T08:58:49.7337657Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-12-04T08:58:49.7337773Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T08:58:49.7338023Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward 2025-12-04T08:58:49.7338104Z hidden_states = self.wi(hidden_states) 2025-12-04T08:58:49.7338108Z 2025-12-04T08:58:49.7338212Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:49.7338433Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:49.7338498Z res = mod(**inputs) 2025-12-04T08:58:49.7338737Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T08:58:49.7338817Z decoder_outputs = self.decoder( 2025-12-04T08:58:49.7339068Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T08:58:49.7339148Z layer_outputs = layer_module( 2025-12-04T08:58:49.7339384Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:49.7339460Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:49.7339706Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7339775Z return func(*args, **kwargs) 2025-12-04T08:58:49.7340017Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 737, in forward 2025-12-04T08:58:49.7340125Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T08:58:49.7340359Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-12-04T08:58:49.7340486Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T08:58:49.7340719Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 289, in forward 2025-12-04T08:58:49.7340802Z hidden_states = self.act(hidden_states) 2025-12-04T08:58:49.7340813Z 2025-12-04T08:58:49.7340914Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:49.7341110Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:49.7341182Z res = mod(**inputs) 2025-12-04T08:58:49.7341417Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1764, in forward 2025-12-04T08:58:49.7341515Z decoder_outputs = self.decoder( 2025-12-04T08:58:49.7341766Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1100, in forward 2025-12-04T08:58:49.7341838Z layer_outputs = layer_module( 2025-12-04T08:58:49.7342066Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:58:49.7342143Z return super().__call__(*args, **kwargs) 2025-12-04T08:58:49.7342383Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:58:49.7342457Z return func(*args, **kwargs) 2025-12-04T08:58:49.7342689Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 737, in forward 2025-12-04T08:58:49.7342779Z hidden_states = self.layer[-1](hidden_states) 2025-12-04T08:58:49.7343025Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-12-04T08:58:49.7343141Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-12-04T08:58:49.7343382Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 297, in forward 2025-12-04T08:58:49.7343461Z hidden_states = self.wo(hidden_states) 2025-12-04T08:58:49.7343465Z 2025-12-04T08:58:49.7343565Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:49.7343772Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:49.7343836Z res = mod(**inputs) 2025-12-04T08:58:49.7344079Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1793, in forward 2025-12-04T08:58:49.7344172Z lm_logits = self.lm_head(sequence_output) 2025-12-04T08:58:49.7344175Z 2025-12-04T08:58:49.7344273Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:58:49.7344485Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:58:49.7344550Z res = mod(**inputs) 2025-12-04T08:58:49.7344799Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1800, in forward 2025-12-04T08:58:49.7344950Z loss = loss_fct(lm_logits.view(-1, lm_logits.size(-1)), labels.view(-1)) 2025-12-04T08:58:49.7344954Z 2025-12-04T08:59:01.4931325Z Compilation time (from dynamo_timed): 18.919365743 2025-12-04T08:59:01.5103871Z pass 2025-12-04T08:59:01.5104318Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T08:59:01.5105186Z TIMING: _recursive_pre_grad_passes:0.01123 _recursive_joint_graph_passes:0.63258 _recursive_post_grad_passes:0.06119 async_compile.wait:1.45957 code_gen:10.67936 inductor_compile:11.96559 backend_compile:15.93261 gc:0.00012 entire_frame_compile:18.91937 total_wall_time:18.91937 2025-12-04T08:59:01.5106617Z STATS: call_* op count: 810 | FakeTensorMode.__torch_dispatch__:11485 | FakeTensor.__torch_dispatch__:4072 | ProxyTorchDispatchMode.__torch_dispatch__:3376 2025-12-04T08:59:01.5107172Z Dynamo produced 1 graphs covering 810 ops with 0 graph breaks (0 unique) 2025-12-04T08:59:04.0396727Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/__init__.py:65: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-12-04T08:59:04.0397653Z import pynvml # type: ignore[import] 2025-12-04T08:59:07.5637367Z 2025-12-04T08:59:08.8629966Z loading model: 0it [00:00, ?it/s] 2025-12-04T08:59:08.8630373Z loading model: 0it [00:01, ?it/s] 2025-12-04T08:59:08.8638873Z cpu eval T5Small 2025-12-04T08:59:11.6359554Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T08:59:12.6616991Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T08:59:13.7352146Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T08:59:29.6610141Z Compilation time (from dynamo_timed): 14.431959636 2025-12-04T08:59:29.6807111Z pass 2025-12-04T08:59:29.6807542Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T08:59:29.6809559Z TIMING: _recursive_pre_grad_passes:0.01134 _recursive_joint_graph_passes:0.60366 async_compile.wait:0.00395 inductor_compile:7.52315 backend_compile:11.5022 gc:0.00122 entire_frame_compile:14.43196 total_wall_time:14.43196 2025-12-04T08:59:29.6810376Z STATS: call_* op count: 810 | FakeTensorMode.__torch_dispatch__:9926 | FakeTensor.__torch_dispatch__:17 | ProxyTorchDispatchMode.__torch_dispatch__:3368 2025-12-04T08:59:29.6810890Z Dynamo produced 1 graphs covering 810 ops with 0 graph breaks (0 unique) 2025-12-04T08:59:31.9959194Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/__init__.py:65: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-12-04T08:59:31.9960091Z import pynvml # type: ignore[import] 2025-12-04T08:59:35.4101718Z 2025-12-04T08:59:37.7894606Z loading model: 0it [00:00, ?it/s] 2025-12-04T08:59:37.7894914Z loading model: 0it [00:02, ?it/s] 2025-12-04T08:59:37.7914666Z cpu eval TrOCRForCausalLM 2025-12-04T08:59:38.0979075Z WARNING:common:fp64 golden ref were not generated for TrOCRForCausalLM. Setting accuracy check to cosine 2025-12-04T08:59:38.1381361Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T08:59:38.6971455Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T08:59:39.2490766Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T08:59:46.6307779Z cudagraph partition due to non gpu ops 2025-12-04T08:59:46.6308092Z cudagraph partition due to non gpu ops 2025-12-04T08:59:46.6308710Z cudagraph partition due to non gpu ops 2025-12-04T08:59:46.6308953Z cudagraph partition due to non gpu ops 2025-12-04T08:59:46.6309213Z cudagraph partition due to non gpu ops 2025-12-04T08:59:46.6309459Z cudagraph partition due to non gpu ops 2025-12-04T08:59:46.6309762Z cudagraph partition due to non gpu ops 2025-12-04T08:59:46.6310073Z cudagraph partition due to non gpu ops 2025-12-04T08:59:46.6310307Z cudagraph partition due to non gpu ops 2025-12-04T08:59:46.6310540Z cudagraph partition due to non gpu ops 2025-12-04T08:59:46.6310776Z cudagraph partition due to non gpu ops 2025-12-04T08:59:46.6311001Z cudagraph partition due to non gpu ops 2025-12-04T08:59:46.6311290Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:59:46.6311827Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:59:46.6312218Z res = mod(**inputs) 2025-12-04T08:59:46.6312662Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T08:59:46.6313125Z outputs = self.model.decoder( 2025-12-04T08:59:46.6313560Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T08:59:46.6314004Z layer_outputs = decoder_layer( 2025-12-04T08:59:46.6314409Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:59:46.6314822Z return super().__call__(*args, **kwargs) 2025-12-04T08:59:46.6315256Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:59:46.6315730Z return func(*args, **kwargs) 2025-12-04T08:59:46.6316151Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 373, in forward 2025-12-04T08:59:46.6316628Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:59:46.6317083Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:59:46.6317493Z return func(*args, **kwargs) 2025-12-04T08:59:46.6317917Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 203, in forward 2025-12-04T08:59:46.6318620Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T08:59:46.6318812Z 2025-12-04T08:59:46.6318932Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:59:46.6319358Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:59:46.6319725Z res = mod(**inputs) 2025-12-04T08:59:46.6320106Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T08:59:46.6320521Z outputs = self.model.decoder( 2025-12-04T08:59:46.6321076Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T08:59:46.6321490Z layer_outputs = decoder_layer( 2025-12-04T08:59:46.6321852Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:59:46.6322242Z return super().__call__(*args, **kwargs) 2025-12-04T08:59:46.6322640Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:59:46.6323067Z return func(*args, **kwargs) 2025-12-04T08:59:46.6323467Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 373, in forward 2025-12-04T08:59:46.6323939Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:59:46.6324386Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:59:46.6324836Z return func(*args, **kwargs) 2025-12-04T08:59:46.6325277Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 223, in forward 2025-12-04T08:59:46.6325708Z key_states = self.k_proj(current_states) 2025-12-04T08:59:46.6325883Z 2025-12-04T08:59:46.6326002Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:59:46.6326399Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:59:46.6326787Z res = mod(**inputs) 2025-12-04T08:59:46.6327182Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T08:59:46.6327605Z outputs = self.model.decoder( 2025-12-04T08:59:46.6328063Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T08:59:46.6328481Z layer_outputs = decoder_layer( 2025-12-04T08:59:46.6328858Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:59:46.6329243Z return super().__call__(*args, **kwargs) 2025-12-04T08:59:46.6329643Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:59:46.6330044Z return func(*args, **kwargs) 2025-12-04T08:59:46.6330441Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 373, in forward 2025-12-04T08:59:46.6330861Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:59:46.6331276Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:59:46.6331693Z return func(*args, **kwargs) 2025-12-04T08:59:46.6332103Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 224, in forward 2025-12-04T08:59:46.6332529Z value_states = self.v_proj(current_states) 2025-12-04T08:59:46.6332685Z 2025-12-04T08:59:46.6332905Z cudagraph partition due to non gpu ops 2025-12-04T08:59:46.6333134Z cudagraph partition due to non gpu ops 2025-12-04T08:59:46.6333352Z cudagraph partition due to non gpu ops 2025-12-04T08:59:46.6333599Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:59:46.6333978Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:59:46.6334321Z res = mod(**inputs) 2025-12-04T08:59:46.6334695Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T08:59:46.6335107Z outputs = self.model.decoder( 2025-12-04T08:59:46.6335507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T08:59:46.6335908Z layer_outputs = decoder_layer( 2025-12-04T08:59:46.6336275Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:59:46.6336661Z return super().__call__(*args, **kwargs) 2025-12-04T08:59:46.6337065Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:59:46.6337452Z return func(*args, **kwargs) 2025-12-04T08:59:46.6337847Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 373, in forward 2025-12-04T08:59:46.6338282Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:59:46.6338705Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:59:46.6339091Z return func(*args, **kwargs) 2025-12-04T08:59:46.6339482Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 296, in forward 2025-12-04T08:59:46.6339916Z attn_output = self.out_proj(attn_output) 2025-12-04T08:59:46.6340066Z 2025-12-04T08:59:46.6340175Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:59:46.6340545Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:59:46.6340911Z res = mod(**inputs) 2025-12-04T08:59:46.6341295Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T08:59:46.6341719Z outputs = self.model.decoder( 2025-12-04T08:59:46.6342111Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T08:59:46.6342514Z layer_outputs = decoder_layer( 2025-12-04T08:59:46.6342883Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:59:46.6343260Z return super().__call__(*args, **kwargs) 2025-12-04T08:59:46.6343655Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:59:46.6344045Z return func(*args, **kwargs) 2025-12-04T08:59:46.6344424Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 407, in forward 2025-12-04T08:59:46.6344882Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:59:46.6345068Z 2025-12-04T08:59:46.6345176Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:59:46.6345547Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:59:46.6345893Z res = mod(**inputs) 2025-12-04T08:59:46.6346278Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T08:59:46.6346684Z outputs = self.model.decoder( 2025-12-04T08:59:46.6347221Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T08:59:46.6347626Z layer_outputs = decoder_layer( 2025-12-04T08:59:46.6348167Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:59:46.6348691Z return super().__call__(*args, **kwargs) 2025-12-04T08:59:46.6349077Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:59:46.6349455Z return func(*args, **kwargs) 2025-12-04T08:59:46.6349842Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 407, in forward 2025-12-04T08:59:46.6350288Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:59:46.6350675Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T08:59:46.6351036Z return self.act(input) 2025-12-04T08:59:46.6351148Z 2025-12-04T08:59:46.6351263Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:59:46.6351617Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:59:46.6351952Z res = mod(**inputs) 2025-12-04T08:59:46.6352314Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T08:59:46.6352704Z outputs = self.model.decoder( 2025-12-04T08:59:46.6353081Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T08:59:46.6353470Z layer_outputs = decoder_layer( 2025-12-04T08:59:46.6353830Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:59:46.6354199Z return super().__call__(*args, **kwargs) 2025-12-04T08:59:46.6354672Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:59:46.6355082Z return func(*args, **kwargs) 2025-12-04T08:59:46.6355493Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 409, in forward 2025-12-04T08:59:46.6355939Z hidden_states = self.fc2(hidden_states) 2025-12-04T08:59:46.6356096Z 2025-12-04T08:59:46.6356210Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:59:46.6356606Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:59:46.6356958Z res = mod(**inputs) 2025-12-04T08:59:46.6357368Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T08:59:46.6357797Z outputs = self.model.decoder( 2025-12-04T08:59:46.6358270Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T08:59:46.6358693Z layer_outputs = decoder_layer( 2025-12-04T08:59:46.6359085Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:59:46.6359504Z return super().__call__(*args, **kwargs) 2025-12-04T08:59:46.6359917Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:59:46.6360319Z return func(*args, **kwargs) 2025-12-04T08:59:46.6360733Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 373, in forward 2025-12-04T08:59:46.6361220Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:59:46.6361654Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:59:46.6362062Z return func(*args, **kwargs) 2025-12-04T08:59:46.6362472Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 203, in forward 2025-12-04T08:59:46.6362943Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T08:59:46.6363127Z 2025-12-04T08:59:46.6363251Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:59:46.6363632Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:59:46.6363977Z res = mod(**inputs) 2025-12-04T08:59:46.6364372Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T08:59:46.6364797Z outputs = self.model.decoder( 2025-12-04T08:59:46.6365240Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T08:59:46.6365665Z layer_outputs = decoder_layer( 2025-12-04T08:59:46.6366108Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:59:46.6366486Z return super().__call__(*args, **kwargs) 2025-12-04T08:59:46.6366903Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:59:46.6367291Z return func(*args, **kwargs) 2025-12-04T08:59:46.6367682Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 373, in forward 2025-12-04T08:59:46.6368114Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:59:46.6368524Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:59:46.6368906Z return func(*args, **kwargs) 2025-12-04T08:59:46.6369295Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 223, in forward 2025-12-04T08:59:46.6369731Z key_states = self.k_proj(current_states) 2025-12-04T08:59:46.6369874Z 2025-12-04T08:59:46.6369980Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:59:46.6370355Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:59:46.6370711Z res = mod(**inputs) 2025-12-04T08:59:46.6371087Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T08:59:46.6371485Z outputs = self.model.decoder( 2025-12-04T08:59:46.6371877Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T08:59:46.6372279Z layer_outputs = decoder_layer( 2025-12-04T08:59:46.6372652Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:59:46.6373036Z return super().__call__(*args, **kwargs) 2025-12-04T08:59:46.6373427Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:59:46.6373809Z return func(*args, **kwargs) 2025-12-04T08:59:46.6374185Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 373, in forward 2025-12-04T08:59:46.6374609Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:59:46.6375023Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:59:46.6375403Z return func(*args, **kwargs) 2025-12-04T08:59:46.6375780Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 224, in forward 2025-12-04T08:59:46.6376210Z value_states = self.v_proj(current_states) 2025-12-04T08:59:46.6376355Z 2025-12-04T08:59:46.6376443Z cudagraph partition due to non gpu ops 2025-12-04T08:59:46.6376658Z cudagraph partition due to non gpu ops 2025-12-04T08:59:46.6376878Z cudagraph partition due to non gpu ops 2025-12-04T08:59:46.6377123Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:59:46.6377493Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:59:46.6377819Z res = mod(**inputs) 2025-12-04T08:59:46.6378188Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T08:59:46.6378591Z outputs = self.model.decoder( 2025-12-04T08:59:46.6378979Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T08:59:46.6379379Z layer_outputs = decoder_layer( 2025-12-04T08:59:46.6379745Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:59:46.6380121Z return super().__call__(*args, **kwargs) 2025-12-04T08:59:46.6380506Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:59:46.6380896Z return func(*args, **kwargs) 2025-12-04T08:59:46.6381292Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 373, in forward 2025-12-04T08:59:46.6381702Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:59:46.6382105Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:59:46.6382479Z return func(*args, **kwargs) 2025-12-04T08:59:46.6382852Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 296, in forward 2025-12-04T08:59:46.6383250Z attn_output = self.out_proj(attn_output) 2025-12-04T08:59:46.6383393Z 2025-12-04T08:59:46.6383497Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:59:46.6383885Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:59:46.6384222Z res = mod(**inputs) 2025-12-04T08:59:46.6384590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T08:59:46.6385045Z outputs = self.model.decoder( 2025-12-04T08:59:46.6385438Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T08:59:46.6385828Z layer_outputs = decoder_layer( 2025-12-04T08:59:46.6386190Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:59:46.6386567Z return super().__call__(*args, **kwargs) 2025-12-04T08:59:46.6386974Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:59:46.6387342Z return func(*args, **kwargs) 2025-12-04T08:59:46.6387723Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 407, in forward 2025-12-04T08:59:46.6388162Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:59:46.6388337Z 2025-12-04T08:59:46.6388442Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:59:46.6388806Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:59:46.6389134Z res = mod(**inputs) 2025-12-04T08:59:46.6389504Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T08:59:46.6389920Z outputs = self.model.decoder( 2025-12-04T08:59:46.6390336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T08:59:46.6390729Z layer_outputs = decoder_layer( 2025-12-04T08:59:46.6391084Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:59:46.6391452Z return super().__call__(*args, **kwargs) 2025-12-04T08:59:46.6391837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:59:46.6392217Z return func(*args, **kwargs) 2025-12-04T08:59:46.6392590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 407, in forward 2025-12-04T08:59:46.6393035Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:59:46.6393439Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T08:59:46.6393806Z return self.act(input) 2025-12-04T08:59:46.6393925Z 2025-12-04T08:59:46.6394034Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:59:46.6394420Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:59:46.6394763Z res = mod(**inputs) 2025-12-04T08:59:46.6395133Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T08:59:46.6395539Z outputs = self.model.decoder( 2025-12-04T08:59:46.6395934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T08:59:46.6396336Z layer_outputs = decoder_layer( 2025-12-04T08:59:46.6396693Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:59:46.6397077Z return super().__call__(*args, **kwargs) 2025-12-04T08:59:46.6397482Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:59:46.6397866Z return func(*args, **kwargs) 2025-12-04T08:59:46.6398387Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 409, in forward 2025-12-04T08:59:46.6398837Z hidden_states = self.fc2(hidden_states) 2025-12-04T08:59:46.6398992Z 2025-12-04T08:59:46.6399159Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:59:46.6399554Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:59:46.6399915Z res = mod(**inputs) 2025-12-04T08:59:46.6400331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T08:59:46.6400765Z outputs = self.model.decoder( 2025-12-04T08:59:46.6401183Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T08:59:46.6401645Z layer_outputs = decoder_layer( 2025-12-04T08:59:46.6402019Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:59:46.6402392Z return super().__call__(*args, **kwargs) 2025-12-04T08:59:46.6402786Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:59:46.6403175Z return func(*args, **kwargs) 2025-12-04T08:59:46.6403565Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 373, in forward 2025-12-04T08:59:46.6403986Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:59:46.6404401Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:59:46.6404840Z return func(*args, **kwargs) 2025-12-04T08:59:46.6405253Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 203, in forward 2025-12-04T08:59:46.6405694Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T08:59:46.6405874Z 2025-12-04T08:59:46.6405982Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:59:46.6406349Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:59:46.6406679Z res = mod(**inputs) 2025-12-04T08:59:46.6407051Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T08:59:46.6407450Z outputs = self.model.decoder( 2025-12-04T08:59:46.6407839Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T08:59:46.6408228Z layer_outputs = decoder_layer( 2025-12-04T08:59:46.6408599Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:59:46.6408977Z return super().__call__(*args, **kwargs) 2025-12-04T08:59:46.6409366Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:59:46.6409750Z return func(*args, **kwargs) 2025-12-04T08:59:46.6410134Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 373, in forward 2025-12-04T08:59:46.6410560Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:59:46.6410964Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:59:46.6411342Z return func(*args, **kwargs) 2025-12-04T08:59:46.6411732Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 223, in forward 2025-12-04T08:59:46.6412145Z key_states = self.k_proj(current_states) 2025-12-04T08:59:46.6412285Z 2025-12-04T08:59:46.6412391Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:59:46.6412783Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:59:46.6413117Z res = mod(**inputs) 2025-12-04T08:59:46.6413485Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T08:59:46.6413918Z outputs = self.model.decoder( 2025-12-04T08:59:46.6414309Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T08:59:46.6414706Z layer_outputs = decoder_layer( 2025-12-04T08:59:46.6415060Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:59:46.6415436Z return super().__call__(*args, **kwargs) 2025-12-04T08:59:46.6415851Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:59:46.6416222Z return func(*args, **kwargs) 2025-12-04T08:59:46.6416598Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 373, in forward 2025-12-04T08:59:46.6417013Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:59:46.6417412Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:59:46.6417782Z return func(*args, **kwargs) 2025-12-04T08:59:46.6418158Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 224, in forward 2025-12-04T08:59:46.6418560Z value_states = self.v_proj(current_states) 2025-12-04T08:59:46.6418701Z 2025-12-04T08:59:46.6418786Z cudagraph partition due to non gpu ops 2025-12-04T08:59:46.6419023Z cudagraph partition due to non gpu ops 2025-12-04T08:59:46.6419237Z cudagraph partition due to non gpu ops 2025-12-04T08:59:46.6419473Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:59:46.6419828Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:59:46.6420154Z res = mod(**inputs) 2025-12-04T08:59:46.6420516Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T08:59:46.6421110Z outputs = self.model.decoder( 2025-12-04T08:59:46.6421507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T08:59:46.6421897Z layer_outputs = decoder_layer( 2025-12-04T08:59:46.6422254Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:59:46.6422622Z return super().__call__(*args, **kwargs) 2025-12-04T08:59:46.6423098Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:59:46.6423480Z return func(*args, **kwargs) 2025-12-04T08:59:46.6423859Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 373, in forward 2025-12-04T08:59:46.6424270Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:59:46.6424674Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:59:46.6425056Z return func(*args, **kwargs) 2025-12-04T08:59:46.6425442Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 296, in forward 2025-12-04T08:59:46.6425860Z attn_output = self.out_proj(attn_output) 2025-12-04T08:59:46.6426008Z 2025-12-04T08:59:46.6426115Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:59:46.6426481Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:59:46.6426798Z res = mod(**inputs) 2025-12-04T08:59:46.6427216Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T08:59:46.6427606Z outputs = self.model.decoder( 2025-12-04T08:59:46.6427977Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T08:59:46.6428394Z layer_outputs = decoder_layer( 2025-12-04T08:59:46.6428751Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:59:46.6429116Z return super().__call__(*args, **kwargs) 2025-12-04T08:59:46.6429495Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:59:46.6429881Z return func(*args, **kwargs) 2025-12-04T08:59:46.6430281Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 407, in forward 2025-12-04T08:59:46.6430722Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:59:46.6430902Z 2025-12-04T08:59:46.6431006Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:59:46.6431366Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:59:46.6431699Z res = mod(**inputs) 2025-12-04T08:59:46.6432053Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T08:59:46.6432445Z outputs = self.model.decoder( 2025-12-04T08:59:46.6432826Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T08:59:46.6433246Z layer_outputs = decoder_layer( 2025-12-04T08:59:46.6433593Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:59:46.6433960Z return super().__call__(*args, **kwargs) 2025-12-04T08:59:46.6434348Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:59:46.6434718Z return func(*args, **kwargs) 2025-12-04T08:59:46.6435098Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 407, in forward 2025-12-04T08:59:46.6435534Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:59:46.6435924Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T08:59:46.6436263Z return self.act(input) 2025-12-04T08:59:46.6436381Z 2025-12-04T08:59:46.6436485Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:59:46.6436855Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:59:46.6437189Z res = mod(**inputs) 2025-12-04T08:59:46.6437569Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T08:59:46.6437976Z outputs = self.model.decoder( 2025-12-04T08:59:46.6438479Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T08:59:46.6438910Z layer_outputs = decoder_layer( 2025-12-04T08:59:46.6439306Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:59:46.6439683Z return super().__call__(*args, **kwargs) 2025-12-04T08:59:46.6440079Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:59:46.6440463Z return func(*args, **kwargs) 2025-12-04T08:59:46.6440858Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 409, in forward 2025-12-04T08:59:46.6441274Z hidden_states = self.fc2(hidden_states) 2025-12-04T08:59:46.6441419Z 2025-12-04T08:59:46.6441556Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:59:46.6441946Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:59:46.6442299Z res = mod(**inputs) 2025-12-04T08:59:46.6442704Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T08:59:46.6443094Z outputs = self.model.decoder( 2025-12-04T08:59:46.6443565Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T08:59:46.6443969Z layer_outputs = decoder_layer( 2025-12-04T08:59:46.6444328Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:59:46.6444725Z return super().__call__(*args, **kwargs) 2025-12-04T08:59:46.6445120Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:59:46.6445506Z return func(*args, **kwargs) 2025-12-04T08:59:46.6445888Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 373, in forward 2025-12-04T08:59:46.6446336Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:59:46.6446755Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:59:46.6447144Z return func(*args, **kwargs) 2025-12-04T08:59:46.6447529Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 203, in forward 2025-12-04T08:59:46.6447997Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T08:59:46.6448169Z 2025-12-04T08:59:46.6448286Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:59:46.6448648Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:59:46.6448980Z res = mod(**inputs) 2025-12-04T08:59:46.6449350Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T08:59:46.6449755Z outputs = self.model.decoder( 2025-12-04T08:59:46.6450142Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T08:59:46.6450545Z layer_outputs = decoder_layer( 2025-12-04T08:59:46.6450904Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:59:46.6451275Z return super().__call__(*args, **kwargs) 2025-12-04T08:59:46.6451668Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:59:46.6452055Z return func(*args, **kwargs) 2025-12-04T08:59:46.6452449Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 373, in forward 2025-12-04T08:59:46.6452870Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:59:46.6453295Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:59:46.6453683Z return func(*args, **kwargs) 2025-12-04T08:59:46.6454067Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 223, in forward 2025-12-04T08:59:46.6454469Z key_states = self.k_proj(current_states) 2025-12-04T08:59:46.6454614Z 2025-12-04T08:59:46.6454729Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:59:46.6455114Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:59:46.6455431Z res = mod(**inputs) 2025-12-04T08:59:46.6455822Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T08:59:46.6456216Z outputs = self.model.decoder( 2025-12-04T08:59:46.6456602Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T08:59:46.6457002Z layer_outputs = decoder_layer( 2025-12-04T08:59:46.6457352Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:59:46.6457717Z return super().__call__(*args, **kwargs) 2025-12-04T08:59:46.6458104Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:59:46.6458480Z return func(*args, **kwargs) 2025-12-04T08:59:46.6458885Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 373, in forward 2025-12-04T08:59:46.6459306Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:59:46.6459704Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:59:46.6460081Z return func(*args, **kwargs) 2025-12-04T08:59:46.6460456Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 224, in forward 2025-12-04T08:59:46.6460865Z value_states = self.v_proj(current_states) 2025-12-04T08:59:46.6461006Z 2025-12-04T08:59:46.6461087Z cudagraph partition due to non gpu ops 2025-12-04T08:59:46.6461303Z cudagraph partition due to non gpu ops 2025-12-04T08:59:46.6461514Z cudagraph partition due to non gpu ops 2025-12-04T08:59:46.6461744Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:59:46.6462127Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:59:46.6462451Z res = mod(**inputs) 2025-12-04T08:59:46.6462809Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T08:59:46.6463192Z outputs = self.model.decoder( 2025-12-04T08:59:46.6463576Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T08:59:46.6463961Z layer_outputs = decoder_layer( 2025-12-04T08:59:46.6464302Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:59:46.6464672Z return super().__call__(*args, **kwargs) 2025-12-04T08:59:46.6465055Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:59:46.6465429Z return func(*args, **kwargs) 2025-12-04T08:59:46.6465797Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 373, in forward 2025-12-04T08:59:46.6466212Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:59:46.6466615Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:59:46.6466984Z return func(*args, **kwargs) 2025-12-04T08:59:46.6467354Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 296, in forward 2025-12-04T08:59:46.6467750Z attn_output = self.out_proj(attn_output) 2025-12-04T08:59:46.6467886Z 2025-12-04T08:59:46.6467997Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:59:46.6468349Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:59:46.6468672Z res = mod(**inputs) 2025-12-04T08:59:46.6469038Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T08:59:46.6469417Z outputs = self.model.decoder( 2025-12-04T08:59:46.6469801Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T08:59:46.6470179Z layer_outputs = decoder_layer( 2025-12-04T08:59:46.6470522Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:59:46.6470896Z return super().__call__(*args, **kwargs) 2025-12-04T08:59:46.6471272Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:59:46.6471644Z return func(*args, **kwargs) 2025-12-04T08:59:46.6472019Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 407, in forward 2025-12-04T08:59:46.6472445Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:59:46.6472621Z 2025-12-04T08:59:46.6472744Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:59:46.6473096Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:59:46.6473410Z res = mod(**inputs) 2025-12-04T08:59:46.6473765Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T08:59:46.6474145Z outputs = self.model.decoder( 2025-12-04T08:59:46.6474514Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T08:59:46.6474884Z layer_outputs = decoder_layer( 2025-12-04T08:59:46.6475225Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:59:46.6475583Z return super().__call__(*args, **kwargs) 2025-12-04T08:59:46.6475975Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:59:46.6476335Z return func(*args, **kwargs) 2025-12-04T08:59:46.6476710Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 407, in forward 2025-12-04T08:59:46.6477133Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:59:46.6477508Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T08:59:46.6477850Z return self.act(input) 2025-12-04T08:59:46.6477964Z 2025-12-04T08:59:46.6478065Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:59:46.6478547Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:59:46.6478909Z res = mod(**inputs) 2025-12-04T08:59:46.6479313Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T08:59:46.6479771Z outputs = self.model.decoder( 2025-12-04T08:59:46.6480147Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T08:59:46.6480541Z layer_outputs = decoder_layer( 2025-12-04T08:59:46.6480893Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:59:46.6481261Z return super().__call__(*args, **kwargs) 2025-12-04T08:59:46.6481637Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:59:46.6482027Z return func(*args, **kwargs) 2025-12-04T08:59:46.6482402Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 409, in forward 2025-12-04T08:59:46.6482797Z hidden_states = self.fc2(hidden_states) 2025-12-04T08:59:46.6482941Z 2025-12-04T08:59:46.6483046Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:59:46.6483406Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:59:46.6483729Z res = mod(**inputs) 2025-12-04T08:59:46.6484106Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T08:59:46.6484496Z outputs = self.model.decoder( 2025-12-04T08:59:46.6484897Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T08:59:46.6485283Z layer_outputs = decoder_layer( 2025-12-04T08:59:46.6485625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:59:46.6485993Z return super().__call__(*args, **kwargs) 2025-12-04T08:59:46.6486375Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:59:46.6486761Z return func(*args, **kwargs) 2025-12-04T08:59:46.6487142Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 373, in forward 2025-12-04T08:59:46.6487563Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:59:46.6487962Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:59:46.6488331Z return func(*args, **kwargs) 2025-12-04T08:59:46.6488706Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 203, in forward 2025-12-04T08:59:46.6489156Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T08:59:46.6489325Z 2025-12-04T08:59:46.6489439Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:59:46.6489803Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:59:46.6490158Z res = mod(**inputs) 2025-12-04T08:59:46.6490527Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T08:59:46.6490926Z outputs = self.model.decoder( 2025-12-04T08:59:46.6491325Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T08:59:46.6491729Z layer_outputs = decoder_layer( 2025-12-04T08:59:46.6492093Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:59:46.6492464Z return super().__call__(*args, **kwargs) 2025-12-04T08:59:46.6492862Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:59:46.6493252Z return func(*args, **kwargs) 2025-12-04T08:59:46.6493638Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 373, in forward 2025-12-04T08:59:46.6494070Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:59:46.6494485Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:59:46.6494866Z return func(*args, **kwargs) 2025-12-04T08:59:46.6495260Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 223, in forward 2025-12-04T08:59:46.6495672Z key_states = self.k_proj(current_states) 2025-12-04T08:59:46.6495810Z 2025-12-04T08:59:46.6495925Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:59:46.6496306Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:59:46.6496623Z res = mod(**inputs) 2025-12-04T08:59:46.6496986Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T08:59:46.6497387Z outputs = self.model.decoder( 2025-12-04T08:59:46.6497773Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T08:59:46.6498204Z layer_outputs = decoder_layer( 2025-12-04T08:59:46.6498566Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:59:46.6498955Z return super().__call__(*args, **kwargs) 2025-12-04T08:59:46.6499351Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:59:46.6499721Z return func(*args, **kwargs) 2025-12-04T08:59:46.6500107Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 373, in forward 2025-12-04T08:59:46.6500523Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:59:46.6500953Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:59:46.6501338Z return func(*args, **kwargs) 2025-12-04T08:59:46.6501725Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 224, in forward 2025-12-04T08:59:46.6502131Z value_states = self.v_proj(current_states) 2025-12-04T08:59:46.6502284Z 2025-12-04T08:59:46.6502366Z cudagraph partition due to non gpu ops 2025-12-04T08:59:46.6502592Z cudagraph partition due to non gpu ops 2025-12-04T08:59:46.6502801Z cudagraph partition due to non gpu ops 2025-12-04T08:59:46.6503042Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:59:46.6503409Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:59:46.6503747Z res = mod(**inputs) 2025-12-04T08:59:46.6504131Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T08:59:46.6504585Z outputs = self.model.decoder( 2025-12-04T08:59:46.6505008Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T08:59:46.6505425Z layer_outputs = decoder_layer( 2025-12-04T08:59:46.6505805Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:59:46.6506216Z return super().__call__(*args, **kwargs) 2025-12-04T08:59:46.6506607Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:59:46.6506985Z return func(*args, **kwargs) 2025-12-04T08:59:46.6507379Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 373, in forward 2025-12-04T08:59:46.6507808Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:59:46.6508243Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:59:46.6508644Z return func(*args, **kwargs) 2025-12-04T08:59:46.6509050Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 296, in forward 2025-12-04T08:59:46.6509488Z attn_output = self.out_proj(attn_output) 2025-12-04T08:59:46.6509626Z 2025-12-04T08:59:46.6509733Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:59:46.6510105Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:59:46.6510437Z res = mod(**inputs) 2025-12-04T08:59:46.6510805Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T08:59:46.6511201Z outputs = self.model.decoder( 2025-12-04T08:59:46.6511594Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T08:59:46.6511999Z layer_outputs = decoder_layer( 2025-12-04T08:59:46.6512371Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:59:46.6512748Z return super().__call__(*args, **kwargs) 2025-12-04T08:59:46.6513138Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:59:46.6513552Z return func(*args, **kwargs) 2025-12-04T08:59:46.6513929Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 407, in forward 2025-12-04T08:59:46.6514379Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:59:46.6514557Z 2025-12-04T08:59:46.6514673Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:59:46.6515045Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:59:46.6515389Z res = mod(**inputs) 2025-12-04T08:59:46.6515759Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T08:59:46.6516164Z outputs = self.model.decoder( 2025-12-04T08:59:46.6516548Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T08:59:46.6516946Z layer_outputs = decoder_layer( 2025-12-04T08:59:46.6517315Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:59:46.6517689Z return super().__call__(*args, **kwargs) 2025-12-04T08:59:46.6518095Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:59:46.6518593Z return func(*args, **kwargs) 2025-12-04T08:59:46.6519035Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 407, in forward 2025-12-04T08:59:46.6519513Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:59:46.6519963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T08:59:46.6520342Z return self.act(input) 2025-12-04T08:59:46.6520467Z 2025-12-04T08:59:46.6520594Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:59:46.6521145Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:59:46.6521516Z res = mod(**inputs) 2025-12-04T08:59:46.6521928Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T08:59:46.6522368Z outputs = self.model.decoder( 2025-12-04T08:59:46.6522793Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T08:59:46.6523237Z layer_outputs = decoder_layer( 2025-12-04T08:59:46.6523631Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:59:46.6524035Z return super().__call__(*args, **kwargs) 2025-12-04T08:59:46.6524466Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:59:46.6524891Z return func(*args, **kwargs) 2025-12-04T08:59:46.6525310Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 409, in forward 2025-12-04T08:59:46.6525750Z hidden_states = self.fc2(hidden_states) 2025-12-04T08:59:46.6525910Z 2025-12-04T08:59:46.6526026Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:59:46.6526429Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:59:46.6526787Z res = mod(**inputs) 2025-12-04T08:59:46.6527190Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T08:59:46.6527626Z outputs = self.model.decoder( 2025-12-04T08:59:46.6528106Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T08:59:46.6528537Z layer_outputs = decoder_layer( 2025-12-04T08:59:46.6528965Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:59:46.6529340Z return super().__call__(*args, **kwargs) 2025-12-04T08:59:46.6529733Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:59:46.6530113Z return func(*args, **kwargs) 2025-12-04T08:59:46.6530500Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 373, in forward 2025-12-04T08:59:46.6530966Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:59:46.6531373Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:59:46.6531762Z return func(*args, **kwargs) 2025-12-04T08:59:46.6532149Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 203, in forward 2025-12-04T08:59:46.6532590Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T08:59:46.6532762Z 2025-12-04T08:59:46.6532867Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:59:46.6533236Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:59:46.6533568Z res = mod(**inputs) 2025-12-04T08:59:46.6533934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T08:59:46.6534377Z outputs = self.model.decoder( 2025-12-04T08:59:46.6534771Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T08:59:46.6535168Z layer_outputs = decoder_layer( 2025-12-04T08:59:46.6535522Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:59:46.6535897Z return super().__call__(*args, **kwargs) 2025-12-04T08:59:46.6536287Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:59:46.6536676Z return func(*args, **kwargs) 2025-12-04T08:59:46.6537055Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 373, in forward 2025-12-04T08:59:46.6537485Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:59:46.6537903Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:59:46.6538291Z return func(*args, **kwargs) 2025-12-04T08:59:46.6538680Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 223, in forward 2025-12-04T08:59:46.6539092Z key_states = self.k_proj(current_states) 2025-12-04T08:59:46.6539231Z 2025-12-04T08:59:46.6539347Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:59:46.6539712Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:59:46.6540047Z res = mod(**inputs) 2025-12-04T08:59:46.6540424Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T08:59:46.6540820Z outputs = self.model.decoder( 2025-12-04T08:59:46.6541214Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T08:59:46.6541620Z layer_outputs = decoder_layer( 2025-12-04T08:59:46.6541980Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:59:46.6542378Z return super().__call__(*args, **kwargs) 2025-12-04T08:59:46.6542776Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:59:46.6543186Z return func(*args, **kwargs) 2025-12-04T08:59:46.6543568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 373, in forward 2025-12-04T08:59:46.6543985Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:59:46.6544396Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:59:46.6544780Z return func(*args, **kwargs) 2025-12-04T08:59:46.6545176Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 224, in forward 2025-12-04T08:59:46.6545588Z value_states = self.v_proj(current_states) 2025-12-04T08:59:46.6545739Z 2025-12-04T08:59:46.6545825Z cudagraph partition due to non gpu ops 2025-12-04T08:59:46.6546046Z cudagraph partition due to non gpu ops 2025-12-04T08:59:46.6546255Z cudagraph partition due to non gpu ops 2025-12-04T08:59:46.6546507Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:59:46.6546862Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:59:46.6547170Z res = mod(**inputs) 2025-12-04T08:59:46.6547524Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T08:59:46.6547902Z outputs = self.model.decoder( 2025-12-04T08:59:46.6548272Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T08:59:46.6548666Z layer_outputs = decoder_layer( 2025-12-04T08:59:46.6549006Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:59:46.6549370Z return super().__call__(*args, **kwargs) 2025-12-04T08:59:46.6549732Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:59:46.6550102Z return func(*args, **kwargs) 2025-12-04T08:59:46.6550465Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 373, in forward 2025-12-04T08:59:46.6550866Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:59:46.6551245Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:59:46.6551605Z return func(*args, **kwargs) 2025-12-04T08:59:46.6551976Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 296, in forward 2025-12-04T08:59:46.6552365Z attn_output = self.out_proj(attn_output) 2025-12-04T08:59:46.6552517Z 2025-12-04T08:59:46.6552621Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:59:46.6552968Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:59:46.6553281Z res = mod(**inputs) 2025-12-04T08:59:46.6553627Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T08:59:46.6554019Z outputs = self.model.decoder( 2025-12-04T08:59:46.6554397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T08:59:46.6554782Z layer_outputs = decoder_layer( 2025-12-04T08:59:46.6555124Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:59:46.6555489Z return super().__call__(*args, **kwargs) 2025-12-04T08:59:46.6555889Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:59:46.6556255Z return func(*args, **kwargs) 2025-12-04T08:59:46.6556630Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 407, in forward 2025-12-04T08:59:46.6557085Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:59:46.6557258Z 2025-12-04T08:59:46.6557372Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:59:46.6557732Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:59:46.6558067Z res = mod(**inputs) 2025-12-04T08:59:46.6558535Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T08:59:46.6558969Z outputs = self.model.decoder( 2025-12-04T08:59:46.6559411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T08:59:46.6559842Z layer_outputs = decoder_layer( 2025-12-04T08:59:46.6560227Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:59:46.6560594Z return super().__call__(*args, **kwargs) 2025-12-04T08:59:46.6560978Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:59:46.6561352Z return func(*args, **kwargs) 2025-12-04T08:59:46.6561719Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 407, in forward 2025-12-04T08:59:46.6562133Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:59:46.6562540Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T08:59:46.6562890Z return self.act(input) 2025-12-04T08:59:46.6562996Z 2025-12-04T08:59:46.6563096Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:59:46.6563451Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:59:46.6563765Z res = mod(**inputs) 2025-12-04T08:59:46.6564117Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T08:59:46.6564496Z outputs = self.model.decoder( 2025-12-04T08:59:46.6564872Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T08:59:46.6565255Z layer_outputs = decoder_layer( 2025-12-04T08:59:46.6565592Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:59:46.6565952Z return super().__call__(*args, **kwargs) 2025-12-04T08:59:46.6566330Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:59:46.6566698Z return func(*args, **kwargs) 2025-12-04T08:59:46.6567059Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 409, in forward 2025-12-04T08:59:46.6567446Z hidden_states = self.fc2(hidden_states) 2025-12-04T08:59:46.6567581Z 2025-12-04T08:59:46.6567689Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:59:46.6568037Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:59:46.6568343Z res = mod(**inputs) 2025-12-04T08:59:46.6568696Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T08:59:46.6569082Z outputs = self.model.decoder( 2025-12-04T08:59:46.6569457Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T08:59:46.6569840Z layer_outputs = decoder_layer( 2025-12-04T08:59:46.6570211Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:59:46.6570557Z return super().__call__(*args, **kwargs) 2025-12-04T08:59:46.6570917Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:59:46.6571294Z return func(*args, **kwargs) 2025-12-04T08:59:46.6571650Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 373, in forward 2025-12-04T08:59:46.6572045Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:59:46.6572431Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:59:46.6572811Z return func(*args, **kwargs) 2025-12-04T08:59:46.6573176Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 203, in forward 2025-12-04T08:59:46.6573595Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T08:59:46.6573770Z 2025-12-04T08:59:46.6573875Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:59:46.6574233Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:59:46.6574555Z res = mod(**inputs) 2025-12-04T08:59:46.6574913Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T08:59:46.6575298Z outputs = self.model.decoder( 2025-12-04T08:59:46.6575680Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T08:59:46.6576098Z layer_outputs = decoder_layer( 2025-12-04T08:59:46.6576441Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:59:46.6576797Z return super().__call__(*args, **kwargs) 2025-12-04T08:59:46.6577168Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:59:46.6577525Z return func(*args, **kwargs) 2025-12-04T08:59:46.6577893Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 373, in forward 2025-12-04T08:59:46.6578302Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:59:46.6578697Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:59:46.6579070Z return func(*args, **kwargs) 2025-12-04T08:59:46.6579443Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 223, in forward 2025-12-04T08:59:46.6579840Z key_states = self.k_proj(current_states) 2025-12-04T08:59:46.6579976Z 2025-12-04T08:59:46.6580080Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:59:46.6580446Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:59:46.6580768Z res = mod(**inputs) 2025-12-04T08:59:46.6581128Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T08:59:46.6581517Z outputs = self.model.decoder( 2025-12-04T08:59:46.6581896Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T08:59:46.6582289Z layer_outputs = decoder_layer( 2025-12-04T08:59:46.6582623Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:59:46.6582985Z return super().__call__(*args, **kwargs) 2025-12-04T08:59:46.6583357Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:59:46.6583744Z return func(*args, **kwargs) 2025-12-04T08:59:46.6584103Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 373, in forward 2025-12-04T08:59:46.6584508Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:59:46.6584916Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:59:46.6585271Z return func(*args, **kwargs) 2025-12-04T08:59:46.6585631Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 224, in forward 2025-12-04T08:59:46.6586027Z value_states = self.v_proj(current_states) 2025-12-04T08:59:46.6586167Z 2025-12-04T08:59:46.6586256Z cudagraph partition due to non gpu ops 2025-12-04T08:59:46.6586478Z cudagraph partition due to non gpu ops 2025-12-04T08:59:46.6586690Z cudagraph partition due to non gpu ops 2025-12-04T08:59:46.6586924Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:59:46.6587278Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:59:46.6587599Z res = mod(**inputs) 2025-12-04T08:59:46.6587966Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T08:59:46.6588363Z outputs = self.model.decoder( 2025-12-04T08:59:46.6588748Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T08:59:46.6589133Z layer_outputs = decoder_layer( 2025-12-04T08:59:46.6589477Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:59:46.6589851Z return super().__call__(*args, **kwargs) 2025-12-04T08:59:46.6590224Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:59:46.6590601Z return func(*args, **kwargs) 2025-12-04T08:59:46.6590977Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 373, in forward 2025-12-04T08:59:46.6591387Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:59:46.6591789Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:59:46.6592164Z return func(*args, **kwargs) 2025-12-04T08:59:46.6592536Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 296, in forward 2025-12-04T08:59:46.6592933Z attn_output = self.out_proj(attn_output) 2025-12-04T08:59:46.6593080Z 2025-12-04T08:59:46.6593186Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:59:46.6593546Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:59:46.6593858Z res = mod(**inputs) 2025-12-04T08:59:46.6594221Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T08:59:46.6594648Z outputs = self.model.decoder( 2025-12-04T08:59:46.6595040Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T08:59:46.6595429Z layer_outputs = decoder_layer( 2025-12-04T08:59:46.6595786Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:59:46.6596157Z return super().__call__(*args, **kwargs) 2025-12-04T08:59:46.6596539Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:59:46.6596929Z return func(*args, **kwargs) 2025-12-04T08:59:46.6597309Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 407, in forward 2025-12-04T08:59:46.6597773Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:59:46.6597949Z 2025-12-04T08:59:46.6598053Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:59:46.6598520Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:59:46.6598881Z res = mod(**inputs) 2025-12-04T08:59:46.6599291Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T08:59:46.6599726Z outputs = self.model.decoder( 2025-12-04T08:59:46.6600176Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T08:59:46.6600619Z layer_outputs = decoder_layer( 2025-12-04T08:59:46.6600996Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:59:46.6601373Z return super().__call__(*args, **kwargs) 2025-12-04T08:59:46.6601761Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:59:46.6602140Z return func(*args, **kwargs) 2025-12-04T08:59:46.6602510Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 407, in forward 2025-12-04T08:59:46.6602945Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:59:46.6603340Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T08:59:46.6603693Z return self.act(input) 2025-12-04T08:59:46.6603827Z 2025-12-04T08:59:46.6603934Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:59:46.6604309Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:59:46.6604645Z res = mod(**inputs) 2025-12-04T08:59:46.6605003Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T08:59:46.6605394Z outputs = self.model.decoder( 2025-12-04T08:59:46.6605782Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T08:59:46.6606176Z layer_outputs = decoder_layer( 2025-12-04T08:59:46.6606521Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:59:46.6606892Z return super().__call__(*args, **kwargs) 2025-12-04T08:59:46.6607280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:59:46.6607657Z return func(*args, **kwargs) 2025-12-04T08:59:46.6608038Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 409, in forward 2025-12-04T08:59:46.6608441Z hidden_states = self.fc2(hidden_states) 2025-12-04T08:59:46.6608578Z 2025-12-04T08:59:46.6608822Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:59:46.6609175Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:59:46.6609504Z res = mod(**inputs) 2025-12-04T08:59:46.6609871Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T08:59:46.6610265Z outputs = self.model.decoder( 2025-12-04T08:59:46.6610644Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T08:59:46.6611041Z layer_outputs = decoder_layer( 2025-12-04T08:59:46.6611397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:59:46.6611758Z return super().__call__(*args, **kwargs) 2025-12-04T08:59:46.6612203Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:59:46.6612583Z return func(*args, **kwargs) 2025-12-04T08:59:46.6612962Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 373, in forward 2025-12-04T08:59:46.6613386Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:59:46.6613786Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:59:46.6614162Z return func(*args, **kwargs) 2025-12-04T08:59:46.6614528Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 203, in forward 2025-12-04T08:59:46.6614973Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T08:59:46.6615148Z 2025-12-04T08:59:46.6615251Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:59:46.6615614Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:59:46.6615930Z res = mod(**inputs) 2025-12-04T08:59:46.6616296Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T08:59:46.6616701Z outputs = self.model.decoder( 2025-12-04T08:59:46.6617092Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T08:59:46.6617484Z layer_outputs = decoder_layer( 2025-12-04T08:59:46.6617844Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:59:46.6618263Z return super().__call__(*args, **kwargs) 2025-12-04T08:59:46.6618679Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:59:46.6619062Z return func(*args, **kwargs) 2025-12-04T08:59:46.6619444Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 373, in forward 2025-12-04T08:59:46.6619869Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:59:46.6620272Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:59:46.6620651Z return func(*args, **kwargs) 2025-12-04T08:59:46.6621205Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 223, in forward 2025-12-04T08:59:46.6621612Z key_states = self.k_proj(current_states) 2025-12-04T08:59:46.6621762Z 2025-12-04T08:59:46.6621870Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:59:46.6622246Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:59:46.6622584Z res = mod(**inputs) 2025-12-04T08:59:46.6622953Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T08:59:46.6623360Z outputs = self.model.decoder( 2025-12-04T08:59:46.6623757Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T08:59:46.6624163Z layer_outputs = decoder_layer( 2025-12-04T08:59:46.6624519Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:59:46.6624897Z return super().__call__(*args, **kwargs) 2025-12-04T08:59:46.6625293Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:59:46.6625674Z return func(*args, **kwargs) 2025-12-04T08:59:46.6626063Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 373, in forward 2025-12-04T08:59:46.6626539Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:59:46.6626956Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:59:46.6627329Z return func(*args, **kwargs) 2025-12-04T08:59:46.6627762Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 224, in forward 2025-12-04T08:59:46.6628181Z value_states = self.v_proj(current_states) 2025-12-04T08:59:46.6628328Z 2025-12-04T08:59:46.6628418Z cudagraph partition due to non gpu ops 2025-12-04T08:59:46.6628638Z cudagraph partition due to non gpu ops 2025-12-04T08:59:46.6628856Z cudagraph partition due to non gpu ops 2025-12-04T08:59:46.6629108Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:59:46.6629499Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:59:46.6629834Z res = mod(**inputs) 2025-12-04T08:59:46.6630207Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T08:59:46.6630599Z outputs = self.model.decoder( 2025-12-04T08:59:46.6630994Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T08:59:46.6631398Z layer_outputs = decoder_layer( 2025-12-04T08:59:46.6631759Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:59:46.6632135Z return super().__call__(*args, **kwargs) 2025-12-04T08:59:46.6632520Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:59:46.6632924Z return func(*args, **kwargs) 2025-12-04T08:59:46.6633293Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 373, in forward 2025-12-04T08:59:46.6633716Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:59:46.6634116Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:59:46.6634486Z return func(*args, **kwargs) 2025-12-04T08:59:46.6634855Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 296, in forward 2025-12-04T08:59:46.6635255Z attn_output = self.out_proj(attn_output) 2025-12-04T08:59:46.6635403Z 2025-12-04T08:59:46.6635510Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:59:46.6635879Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:59:46.6636205Z res = mod(**inputs) 2025-12-04T08:59:46.6636576Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T08:59:46.6636975Z outputs = self.model.decoder( 2025-12-04T08:59:46.6637360Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T08:59:46.6637756Z layer_outputs = decoder_layer( 2025-12-04T08:59:46.6638119Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:59:46.6638568Z return super().__call__(*args, **kwargs) 2025-12-04T08:59:46.6638969Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:59:46.6639380Z return func(*args, **kwargs) 2025-12-04T08:59:46.6639786Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 407, in forward 2025-12-04T08:59:46.6640268Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:59:46.6640462Z 2025-12-04T08:59:46.6640577Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:59:46.6641006Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:59:46.6641342Z res = mod(**inputs) 2025-12-04T08:59:46.6641716Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T08:59:46.6642141Z outputs = self.model.decoder( 2025-12-04T08:59:46.6642537Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T08:59:46.6642924Z layer_outputs = decoder_layer( 2025-12-04T08:59:46.6643266Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:59:46.6643637Z return super().__call__(*args, **kwargs) 2025-12-04T08:59:46.6644036Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:59:46.6644408Z return func(*args, **kwargs) 2025-12-04T08:59:46.6644790Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 407, in forward 2025-12-04T08:59:46.6645225Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:59:46.6645622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T08:59:46.6645965Z return self.act(input) 2025-12-04T08:59:46.6646083Z 2025-12-04T08:59:46.6646189Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:59:46.6646553Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:59:46.6646891Z res = mod(**inputs) 2025-12-04T08:59:46.6647251Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T08:59:46.6647651Z outputs = self.model.decoder( 2025-12-04T08:59:46.6648038Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T08:59:46.6648416Z layer_outputs = decoder_layer( 2025-12-04T08:59:46.6648766Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:59:46.6649135Z return super().__call__(*args, **kwargs) 2025-12-04T08:59:46.6649516Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:59:46.6649888Z return func(*args, **kwargs) 2025-12-04T08:59:46.6650260Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 409, in forward 2025-12-04T08:59:46.6650655Z hidden_states = self.fc2(hidden_states) 2025-12-04T08:59:46.6650792Z 2025-12-04T08:59:46.6650898Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:59:46.6651269Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:59:46.6651585Z res = mod(**inputs) 2025-12-04T08:59:46.6651937Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T08:59:46.6652312Z outputs = self.model.decoder( 2025-12-04T08:59:46.6652684Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T08:59:46.6653063Z layer_outputs = decoder_layer( 2025-12-04T08:59:46.6653406Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:59:46.6653772Z return super().__call__(*args, **kwargs) 2025-12-04T08:59:46.6654164Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:59:46.6654528Z return func(*args, **kwargs) 2025-12-04T08:59:46.6654906Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 373, in forward 2025-12-04T08:59:46.6655317Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:59:46.6655715Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:59:46.6656096Z return func(*args, **kwargs) 2025-12-04T08:59:46.6656454Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 203, in forward 2025-12-04T08:59:46.6656867Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T08:59:46.6657029Z 2025-12-04T08:59:46.6657135Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:59:46.6657496Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:59:46.6657814Z res = mod(**inputs) 2025-12-04T08:59:46.6658171Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T08:59:46.6658554Z outputs = self.model.decoder( 2025-12-04T08:59:46.6658920Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T08:59:46.6659304Z layer_outputs = decoder_layer( 2025-12-04T08:59:46.6659650Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:59:46.6660001Z return super().__call__(*args, **kwargs) 2025-12-04T08:59:46.6660375Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:59:46.6660759Z return func(*args, **kwargs) 2025-12-04T08:59:46.6661127Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 373, in forward 2025-12-04T08:59:46.6661528Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:59:46.6661927Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:59:46.6662292Z return func(*args, **kwargs) 2025-12-04T08:59:46.6662663Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 223, in forward 2025-12-04T08:59:46.6663043Z key_states = self.k_proj(current_states) 2025-12-04T08:59:46.6663185Z 2025-12-04T08:59:46.6663288Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:59:46.6663642Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:59:46.6663951Z res = mod(**inputs) 2025-12-04T08:59:46.6664304Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T08:59:46.6664687Z outputs = self.model.decoder( 2025-12-04T08:59:46.6665061Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T08:59:46.6665437Z layer_outputs = decoder_layer( 2025-12-04T08:59:46.6665780Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:59:46.6666144Z return super().__call__(*args, **kwargs) 2025-12-04T08:59:46.6666513Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:59:46.6666895Z return func(*args, **kwargs) 2025-12-04T08:59:46.6667279Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 373, in forward 2025-12-04T08:59:46.6667688Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:59:46.6668074Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:59:46.6668458Z return func(*args, **kwargs) 2025-12-04T08:59:46.6668832Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 224, in forward 2025-12-04T08:59:46.6669227Z value_states = self.v_proj(current_states) 2025-12-04T08:59:46.6669387Z 2025-12-04T08:59:46.6669466Z cudagraph partition due to non gpu ops 2025-12-04T08:59:46.6669677Z cudagraph partition due to non gpu ops 2025-12-04T08:59:46.6669887Z cudagraph partition due to non gpu ops 2025-12-04T08:59:46.6670110Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:59:46.6670457Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:59:46.6670772Z res = mod(**inputs) 2025-12-04T08:59:46.6671148Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T08:59:46.6671525Z outputs = self.model.decoder( 2025-12-04T08:59:46.6671899Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T08:59:46.6672293Z layer_outputs = decoder_layer( 2025-12-04T08:59:46.6672634Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:59:46.6672994Z return super().__call__(*args, **kwargs) 2025-12-04T08:59:46.6673370Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:59:46.6673741Z return func(*args, **kwargs) 2025-12-04T08:59:46.6674102Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 373, in forward 2025-12-04T08:59:46.6674525Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:59:46.6674915Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:59:46.6675274Z return func(*args, **kwargs) 2025-12-04T08:59:46.6675641Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 296, in forward 2025-12-04T08:59:46.6676034Z attn_output = self.out_proj(attn_output) 2025-12-04T08:59:46.6676174Z 2025-12-04T08:59:46.6676284Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:59:46.6676643Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:59:46.6676958Z res = mod(**inputs) 2025-12-04T08:59:46.6677310Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T08:59:46.6677726Z outputs = self.model.decoder( 2025-12-04T08:59:46.6678103Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T08:59:46.6678570Z layer_outputs = decoder_layer( 2025-12-04T08:59:46.6678940Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:59:46.6679313Z return super().__call__(*args, **kwargs) 2025-12-04T08:59:46.6679737Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:59:46.6680161Z return func(*args, **kwargs) 2025-12-04T08:59:46.6680574Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 407, in forward 2025-12-04T08:59:46.6681046Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:59:46.6681229Z 2025-12-04T08:59:46.6681337Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:59:46.6681706Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:59:46.6682028Z res = mod(**inputs) 2025-12-04T08:59:46.6682421Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T08:59:46.6682813Z outputs = self.model.decoder( 2025-12-04T08:59:46.6683193Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T08:59:46.6683593Z layer_outputs = decoder_layer( 2025-12-04T08:59:46.6683947Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:59:46.6684320Z return super().__call__(*args, **kwargs) 2025-12-04T08:59:46.6684710Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:59:46.6685090Z return func(*args, **kwargs) 2025-12-04T08:59:46.6685488Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 407, in forward 2025-12-04T08:59:46.6685924Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:59:46.6686309Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T08:59:46.6686657Z return self.act(input) 2025-12-04T08:59:46.6686779Z 2025-12-04T08:59:46.6686882Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:59:46.6687240Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:59:46.6687559Z res = mod(**inputs) 2025-12-04T08:59:46.6687915Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T08:59:46.6688305Z outputs = self.model.decoder( 2025-12-04T08:59:46.6688698Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T08:59:46.6689085Z layer_outputs = decoder_layer( 2025-12-04T08:59:46.6689434Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:59:46.6689797Z return super().__call__(*args, **kwargs) 2025-12-04T08:59:46.6690173Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:59:46.6690552Z return func(*args, **kwargs) 2025-12-04T08:59:46.6690927Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 409, in forward 2025-12-04T08:59:46.6691342Z hidden_states = self.fc2(hidden_states) 2025-12-04T08:59:46.6691492Z 2025-12-04T08:59:46.6691599Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:59:46.6691974Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:59:46.6692298Z res = mod(**inputs) 2025-12-04T08:59:46.6692648Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T08:59:46.6693037Z outputs = self.model.decoder( 2025-12-04T08:59:46.6693417Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T08:59:46.6693805Z layer_outputs = decoder_layer( 2025-12-04T08:59:46.6694146Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:59:46.6694509Z return super().__call__(*args, **kwargs) 2025-12-04T08:59:46.6694889Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:59:46.6695259Z return func(*args, **kwargs) 2025-12-04T08:59:46.6695636Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 373, in forward 2025-12-04T08:59:46.6696053Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:59:46.6696468Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:59:46.6696849Z return func(*args, **kwargs) 2025-12-04T08:59:46.6697235Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 203, in forward 2025-12-04T08:59:46.6697695Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T08:59:46.6697864Z 2025-12-04T08:59:46.6697979Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:59:46.6698347Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:59:46.6698687Z res = mod(**inputs) 2025-12-04T08:59:46.6699070Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T08:59:46.6699456Z outputs = self.model.decoder( 2025-12-04T08:59:46.6699837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T08:59:46.6700225Z layer_outputs = decoder_layer( 2025-12-04T08:59:46.6700574Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:59:46.6700934Z return super().__call__(*args, **kwargs) 2025-12-04T08:59:46.6701318Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:59:46.6701695Z return func(*args, **kwargs) 2025-12-04T08:59:46.6702062Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 373, in forward 2025-12-04T08:59:46.6702498Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:59:46.6702900Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:59:46.6703274Z return func(*args, **kwargs) 2025-12-04T08:59:46.6703644Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 223, in forward 2025-12-04T08:59:46.6704043Z key_states = self.k_proj(current_states) 2025-12-04T08:59:46.6704180Z 2025-12-04T08:59:46.6704291Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:59:46.6704655Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:59:46.6704971Z res = mod(**inputs) 2025-12-04T08:59:46.6705330Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T08:59:46.6705720Z outputs = self.model.decoder( 2025-12-04T08:59:46.6706098Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T08:59:46.6706488Z layer_outputs = decoder_layer( 2025-12-04T08:59:46.6706836Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:59:46.6707205Z return super().__call__(*args, **kwargs) 2025-12-04T08:59:46.6707586Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:59:46.6707970Z return func(*args, **kwargs) 2025-12-04T08:59:46.6708354Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 373, in forward 2025-12-04T08:59:46.6708768Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:59:46.6709179Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:59:46.6709550Z return func(*args, **kwargs) 2025-12-04T08:59:46.6709928Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 224, in forward 2025-12-04T08:59:46.6710357Z value_states = self.v_proj(current_states) 2025-12-04T08:59:46.6710509Z 2025-12-04T08:59:46.6710591Z cudagraph partition due to non gpu ops 2025-12-04T08:59:46.6710806Z cudagraph partition due to non gpu ops 2025-12-04T08:59:46.6711012Z cudagraph partition due to non gpu ops 2025-12-04T08:59:46.6711271Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:59:46.6711630Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:59:46.6711955Z res = mod(**inputs) 2025-12-04T08:59:46.6712310Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T08:59:46.6712704Z outputs = self.model.decoder( 2025-12-04T08:59:46.6713147Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T08:59:46.6713531Z layer_outputs = decoder_layer( 2025-12-04T08:59:46.6713888Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:59:46.6714264Z return super().__call__(*args, **kwargs) 2025-12-04T08:59:46.6714645Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:59:46.6715016Z return func(*args, **kwargs) 2025-12-04T08:59:46.6715392Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 373, in forward 2025-12-04T08:59:46.6715808Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:59:46.6716224Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:59:46.6716611Z return func(*args, **kwargs) 2025-12-04T08:59:46.6716996Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 296, in forward 2025-12-04T08:59:46.6717407Z attn_output = self.out_proj(attn_output) 2025-12-04T08:59:46.6717548Z 2025-12-04T08:59:46.6717655Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:59:46.6718028Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:59:46.6718440Z res = mod(**inputs) 2025-12-04T08:59:46.6718817Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T08:59:46.6719221Z outputs = self.model.decoder( 2025-12-04T08:59:46.6719637Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T08:59:46.6720069Z layer_outputs = decoder_layer( 2025-12-04T08:59:46.6720452Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:59:46.6720983Z return super().__call__(*args, **kwargs) 2025-12-04T08:59:46.6721407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:59:46.6721796Z return func(*args, **kwargs) 2025-12-04T08:59:46.6722183Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 407, in forward 2025-12-04T08:59:46.6722637Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:59:46.6722818Z 2025-12-04T08:59:46.6722937Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:59:46.6723318Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:59:46.6723647Z res = mod(**inputs) 2025-12-04T08:59:46.6724024Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T08:59:46.6724428Z outputs = self.model.decoder( 2025-12-04T08:59:46.6724912Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T08:59:46.6725311Z layer_outputs = decoder_layer( 2025-12-04T08:59:46.6725669Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:59:46.6726077Z return super().__call__(*args, **kwargs) 2025-12-04T08:59:46.6726469Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:59:46.6726856Z return func(*args, **kwargs) 2025-12-04T08:59:46.6727243Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 407, in forward 2025-12-04T08:59:46.6727690Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:59:46.6728124Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T08:59:46.6728486Z return self.act(input) 2025-12-04T08:59:46.6728601Z 2025-12-04T08:59:46.6728720Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:59:46.6729088Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:59:46.6729422Z res = mod(**inputs) 2025-12-04T08:59:46.6729795Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T08:59:46.6730198Z outputs = self.model.decoder( 2025-12-04T08:59:46.6730580Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T08:59:46.6730978Z layer_outputs = decoder_layer( 2025-12-04T08:59:46.6731367Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:59:46.6731735Z return super().__call__(*args, **kwargs) 2025-12-04T08:59:46.6732129Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:59:46.6732515Z return func(*args, **kwargs) 2025-12-04T08:59:46.6732901Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 409, in forward 2025-12-04T08:59:46.6733308Z hidden_states = self.fc2(hidden_states) 2025-12-04T08:59:46.6733451Z 2025-12-04T08:59:46.6733552Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:59:46.6733932Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:59:46.6734246Z res = mod(**inputs) 2025-12-04T08:59:46.6734606Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T08:59:46.6734998Z outputs = self.model.decoder( 2025-12-04T08:59:46.6735377Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T08:59:46.6735758Z layer_outputs = decoder_layer( 2025-12-04T08:59:46.6736109Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:59:46.6736479Z return super().__call__(*args, **kwargs) 2025-12-04T08:59:46.6736842Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:59:46.6737206Z return func(*args, **kwargs) 2025-12-04T08:59:46.6737570Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 373, in forward 2025-12-04T08:59:46.6737977Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:59:46.6738375Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:59:46.6738747Z return func(*args, **kwargs) 2025-12-04T08:59:46.6739140Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 203, in forward 2025-12-04T08:59:46.6739586Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T08:59:46.6739755Z 2025-12-04T08:59:46.6739890Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:59:46.6740254Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:59:46.6740583Z res = mod(**inputs) 2025-12-04T08:59:46.6740947Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T08:59:46.6741364Z outputs = self.model.decoder( 2025-12-04T08:59:46.6741758Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T08:59:46.6742179Z layer_outputs = decoder_layer( 2025-12-04T08:59:46.6742537Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:59:46.6742912Z return super().__call__(*args, **kwargs) 2025-12-04T08:59:46.6743307Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:59:46.6743704Z return func(*args, **kwargs) 2025-12-04T08:59:46.6744071Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 373, in forward 2025-12-04T08:59:46.6744487Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:59:46.6744890Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:59:46.6745290Z return func(*args, **kwargs) 2025-12-04T08:59:46.6745562Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 223, in forward 2025-12-04T08:59:46.6745645Z key_states = self.k_proj(current_states) 2025-12-04T08:59:46.6745649Z 2025-12-04T08:59:46.6745764Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:59:46.6745969Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:59:46.6746037Z res = mod(**inputs) 2025-12-04T08:59:46.6746305Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T08:59:46.6746382Z outputs = self.model.decoder( 2025-12-04T08:59:46.6746644Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T08:59:46.6746734Z layer_outputs = decoder_layer( 2025-12-04T08:59:46.6746962Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:59:46.6747048Z return super().__call__(*args, **kwargs) 2025-12-04T08:59:46.6747294Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:59:46.6747363Z return func(*args, **kwargs) 2025-12-04T08:59:46.6747630Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 373, in forward 2025-12-04T08:59:46.6747730Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:59:46.6747989Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:59:46.6748058Z return func(*args, **kwargs) 2025-12-04T08:59:46.6748307Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 224, in forward 2025-12-04T08:59:46.6748402Z value_states = self.v_proj(current_states) 2025-12-04T08:59:46.6748405Z 2025-12-04T08:59:46.6748485Z cudagraph partition due to non gpu ops 2025-12-04T08:59:46.6748563Z cudagraph partition due to non gpu ops 2025-12-04T08:59:46.6748666Z cudagraph partition due to non gpu ops 2025-12-04T08:59:46.6748770Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:59:46.6748972Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:59:46.6749053Z res = mod(**inputs) 2025-12-04T08:59:46.6749303Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T08:59:46.6749381Z outputs = self.model.decoder( 2025-12-04T08:59:46.6749631Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T08:59:46.6749700Z layer_outputs = decoder_layer( 2025-12-04T08:59:46.6749945Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:59:46.6750022Z return super().__call__(*args, **kwargs) 2025-12-04T08:59:46.6750266Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:59:46.6750333Z return func(*args, **kwargs) 2025-12-04T08:59:46.6750582Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 373, in forward 2025-12-04T08:59:46.6750687Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:59:46.6750922Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:59:46.6750987Z return func(*args, **kwargs) 2025-12-04T08:59:46.6751236Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 296, in forward 2025-12-04T08:59:46.6751331Z attn_output = self.out_proj(attn_output) 2025-12-04T08:59:46.6751335Z 2025-12-04T08:59:46.6751444Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:59:46.6751639Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:59:46.6751701Z res = mod(**inputs) 2025-12-04T08:59:46.6751954Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T08:59:46.6752027Z outputs = self.model.decoder( 2025-12-04T08:59:46.6752278Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T08:59:46.6752348Z layer_outputs = decoder_layer( 2025-12-04T08:59:46.6752564Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:59:46.6752649Z return super().__call__(*args, **kwargs) 2025-12-04T08:59:46.6752887Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:59:46.6752955Z return func(*args, **kwargs) 2025-12-04T08:59:46.6753211Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 407, in forward 2025-12-04T08:59:46.6753331Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:59:46.6753335Z 2025-12-04T08:59:46.6753444Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:59:46.6753635Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:59:46.6753698Z res = mod(**inputs) 2025-12-04T08:59:46.6753952Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T08:59:46.6754024Z outputs = self.model.decoder( 2025-12-04T08:59:46.6754277Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T08:59:46.6754349Z layer_outputs = decoder_layer( 2025-12-04T08:59:46.6754583Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:59:46.6754668Z return super().__call__(*args, **kwargs) 2025-12-04T08:59:46.6754904Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:59:46.6754991Z return func(*args, **kwargs) 2025-12-04T08:59:46.6755249Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 407, in forward 2025-12-04T08:59:46.6755363Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:59:46.6755579Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T08:59:46.6755650Z return self.act(input) 2025-12-04T08:59:46.6755653Z 2025-12-04T08:59:46.6755769Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:59:46.6755968Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:59:46.6756032Z res = mod(**inputs) 2025-12-04T08:59:46.6756278Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T08:59:46.6756356Z outputs = self.model.decoder( 2025-12-04T08:59:46.6756602Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T08:59:46.6756676Z layer_outputs = decoder_layer( 2025-12-04T08:59:46.6756891Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:59:46.6756967Z return super().__call__(*args, **kwargs) 2025-12-04T08:59:46.6757224Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:59:46.6757292Z return func(*args, **kwargs) 2025-12-04T08:59:46.6757556Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 409, in forward 2025-12-04T08:59:46.6757638Z hidden_states = self.fc2(hidden_states) 2025-12-04T08:59:46.6757641Z 2025-12-04T08:59:46.6757741Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:59:46.6757947Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:59:46.6758010Z res = mod(**inputs) 2025-12-04T08:59:46.6758326Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T08:59:46.6758409Z outputs = self.model.decoder( 2025-12-04T08:59:46.6758663Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T08:59:46.6758753Z layer_outputs = decoder_layer( 2025-12-04T08:59:46.6759004Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:59:46.6759092Z return super().__call__(*args, **kwargs) 2025-12-04T08:59:46.6759377Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:59:46.6759464Z return func(*args, **kwargs) 2025-12-04T08:59:46.6759748Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 373, in forward 2025-12-04T08:59:46.6759863Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:59:46.6760138Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:59:46.6760217Z return func(*args, **kwargs) 2025-12-04T08:59:46.6760487Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 203, in forward 2025-12-04T08:59:46.6760604Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T08:59:46.6760608Z 2025-12-04T08:59:46.6760750Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:59:46.6760954Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:59:46.6761028Z res = mod(**inputs) 2025-12-04T08:59:46.6761307Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T08:59:46.6761382Z outputs = self.model.decoder( 2025-12-04T08:59:46.6761657Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T08:59:46.6761728Z layer_outputs = decoder_layer( 2025-12-04T08:59:46.6761952Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:59:46.6762056Z return super().__call__(*args, **kwargs) 2025-12-04T08:59:46.6762300Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:59:46.6762375Z return func(*args, **kwargs) 2025-12-04T08:59:46.6762626Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 373, in forward 2025-12-04T08:59:46.6762728Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:59:46.6762978Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:59:46.6763045Z return func(*args, **kwargs) 2025-12-04T08:59:46.6763304Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 223, in forward 2025-12-04T08:59:46.6763404Z key_states = self.k_proj(current_states) 2025-12-04T08:59:46.6763408Z 2025-12-04T08:59:46.6763515Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:59:46.6763720Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:59:46.6763786Z res = mod(**inputs) 2025-12-04T08:59:46.6764038Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T08:59:46.6764120Z outputs = self.model.decoder( 2025-12-04T08:59:46.6764371Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T08:59:46.6764448Z layer_outputs = decoder_layer( 2025-12-04T08:59:46.6764671Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:59:46.6764746Z return super().__call__(*args, **kwargs) 2025-12-04T08:59:46.6764999Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:59:46.6765070Z return func(*args, **kwargs) 2025-12-04T08:59:46.6765324Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 373, in forward 2025-12-04T08:59:46.6765429Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:59:46.6765672Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:59:46.6765749Z return func(*args, **kwargs) 2025-12-04T08:59:46.6766002Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 224, in forward 2025-12-04T08:59:46.6766089Z value_states = self.v_proj(current_states) 2025-12-04T08:59:46.6766092Z 2025-12-04T08:59:46.6766178Z cudagraph partition due to non gpu ops 2025-12-04T08:59:46.6766256Z cudagraph partition due to non gpu ops 2025-12-04T08:59:46.6766340Z cudagraph partition due to non gpu ops 2025-12-04T08:59:46.6766443Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:59:46.6766641Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:59:46.6766730Z res = mod(**inputs) 2025-12-04T08:59:46.6766982Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T08:59:46.6767077Z outputs = self.model.decoder( 2025-12-04T08:59:46.6767339Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T08:59:46.6767409Z layer_outputs = decoder_layer( 2025-12-04T08:59:46.6767642Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:59:46.6767722Z return super().__call__(*args, **kwargs) 2025-12-04T08:59:46.6767984Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:59:46.6768062Z return func(*args, **kwargs) 2025-12-04T08:59:46.6768317Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 373, in forward 2025-12-04T08:59:46.6768414Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T08:59:46.6768666Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:59:46.6768736Z return func(*args, **kwargs) 2025-12-04T08:59:46.6768997Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 296, in forward 2025-12-04T08:59:46.6769077Z attn_output = self.out_proj(attn_output) 2025-12-04T08:59:46.6769081Z 2025-12-04T08:59:46.6769182Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:59:46.6769879Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:59:46.6769944Z res = mod(**inputs) 2025-12-04T08:59:46.6770205Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T08:59:46.6770282Z outputs = self.model.decoder( 2025-12-04T08:59:46.6770533Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T08:59:46.6770614Z layer_outputs = decoder_layer( 2025-12-04T08:59:46.6770836Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:59:46.6770911Z return super().__call__(*args, **kwargs) 2025-12-04T08:59:46.6771163Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:59:46.6771231Z return func(*args, **kwargs) 2025-12-04T08:59:46.6771491Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 407, in forward 2025-12-04T08:59:46.6771609Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:59:46.6771614Z 2025-12-04T08:59:46.6771716Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:59:46.6771921Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:59:46.6771987Z res = mod(**inputs) 2025-12-04T08:59:46.6772249Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T08:59:46.6772322Z outputs = self.model.decoder( 2025-12-04T08:59:46.6772578Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T08:59:46.6772655Z layer_outputs = decoder_layer( 2025-12-04T08:59:46.6772877Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:59:46.6772956Z return super().__call__(*args, **kwargs) 2025-12-04T08:59:46.6773217Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:59:46.6773288Z return func(*args, **kwargs) 2025-12-04T08:59:46.6773547Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 407, in forward 2025-12-04T08:59:46.6773682Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T08:59:46.6773900Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T08:59:46.6773977Z return self.act(input) 2025-12-04T08:59:46.6773980Z 2025-12-04T08:59:46.6774084Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:59:46.6774283Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:59:46.6774354Z res = mod(**inputs) 2025-12-04T08:59:46.6774629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 828, in forward 2025-12-04T08:59:46.6774711Z outputs = self.model.decoder( 2025-12-04T08:59:46.6774974Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 652, in forward 2025-12-04T08:59:46.6775044Z layer_outputs = decoder_layer( 2025-12-04T08:59:46.6775271Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T08:59:46.6775348Z return super().__call__(*args, **kwargs) 2025-12-04T08:59:46.6775595Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T08:59:46.6775661Z return func(*args, **kwargs) 2025-12-04T08:59:46.6775921Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 409, in forward 2025-12-04T08:59:46.6776010Z hidden_states = self.fc2(hidden_states) 2025-12-04T08:59:46.6776014Z 2025-12-04T08:59:46.6776113Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:59:46.6776308Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:59:46.6776383Z res = mod(**inputs) 2025-12-04T08:59:46.6776628Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 844, in forward 2025-12-04T08:59:46.6776726Z logits = self.output_projection(outputs[0]) 2025-12-04T08:59:46.6776729Z 2025-12-04T08:59:46.6776827Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T08:59:46.6777018Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T08:59:46.6777087Z res = mod(**inputs) 2025-12-04T08:59:46.6777336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 849, in forward 2025-12-04T08:59:46.6777491Z loss = loss_fct(logits.view(-1, self.config.vocab_size), labels.view(-1)) 2025-12-04T08:59:46.6777494Z 2025-12-04T08:59:57.3296932Z Compilation time (from dynamo_timed): 17.060062358 2025-12-04T08:59:57.3312747Z pass 2025-12-04T08:59:57.3313386Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T08:59:57.3314528Z TIMING: _recursive_pre_grad_passes:0.00771 _recursive_joint_graph_passes:0.73606 _recursive_post_grad_passes:0.06665 async_compile.wait:1.02177 code_gen:9.93187 inductor_compile:11.19062 backend_compile:14.56083 gc:0.00019 entire_frame_compile:17.06006 total_wall_time:17.06006 2025-12-04T08:59:57.3315639Z STATS: call_* op count: 443 | FakeTensorMode.__torch_dispatch__:8334 | FakeTensor.__torch_dispatch__:4316 | ProxyTorchDispatchMode.__torch_dispatch__:2529 2025-12-04T08:59:57.3316176Z Dynamo produced 1 graphs covering 443 ops with 0 graph breaks (0 unique) 2025-12-04T08:59:59.8100090Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/__init__.py:65: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-12-04T08:59:59.8101079Z import pynvml # type: ignore[import] 2025-12-04T09:00:03.2046709Z 2025-12-04T09:00:09.0930332Z loading model: 0it [00:00, ?it/s] 2025-12-04T09:00:09.0930705Z loading model: 0it [00:05, ?it/s] 2025-12-04T09:00:09.0953862Z cpu eval XGLMForCausalLM 2025-12-04T09:00:09.8313824Z WARNING:common:fp64 golden ref were not generated for XGLMForCausalLM. Setting accuracy check to cosine 2025-12-04T09:00:09.9239818Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:00:10.9310145Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:00:11.9782878Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:00:27.0338202Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0338748Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0339096Z res = mod(**inputs) 2025-12-04T09:00:27.0339496Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0339906Z outputs = self.model( 2025-12-04T09:00:27.0340275Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0340670Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0341032Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0341724Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0342139Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0342546Z return func(*args, **kwargs) 2025-12-04T09:00:27.0342957Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.0343394Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.0343816Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0344208Z return func(*args, **kwargs) 2025-12-04T09:00:27.0344591Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 157, in forward 2025-12-04T09:00:27.0345073Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:00:27.0345255Z 2025-12-04T09:00:27.0345368Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0345753Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0346121Z res = mod(**inputs) 2025-12-04T09:00:27.0346515Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0346936Z outputs = self.model( 2025-12-04T09:00:27.0347425Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0347835Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0348195Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0348579Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0348969Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0349343Z return func(*args, **kwargs) 2025-12-04T09:00:27.0349720Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.0350213Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.0350623Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0350991Z return func(*args, **kwargs) 2025-12-04T09:00:27.0351412Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 177, in forward 2025-12-04T09:00:27.0351812Z key_states = self.k_proj(current_states) 2025-12-04T09:00:27.0351951Z 2025-12-04T09:00:27.0352067Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0352429Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0352756Z res = mod(**inputs) 2025-12-04T09:00:27.0353163Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0353584Z outputs = self.model( 2025-12-04T09:00:27.0353961Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0354366Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0354756Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0355178Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0355574Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0355966Z return func(*args, **kwargs) 2025-12-04T09:00:27.0356343Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.0356812Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.0357234Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0357642Z return func(*args, **kwargs) 2025-12-04T09:00:27.0358050Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 157, in forward 2025-12-04T09:00:27.0358666Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:00:27.0358861Z 2025-12-04T09:00:27.0358978Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0359373Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0359740Z res = mod(**inputs) 2025-12-04T09:00:27.0360129Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0360564Z outputs = self.model( 2025-12-04T09:00:27.0360957Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0361356Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0361718Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0362098Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0362486Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0362879Z return func(*args, **kwargs) 2025-12-04T09:00:27.0363261Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.0363684Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.0364094Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0364488Z return func(*args, **kwargs) 2025-12-04T09:00:27.0364866Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 199, in forward 2025-12-04T09:00:27.0365580Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-12-04T09:00:27.0365787Z 2025-12-04T09:00:27.0365894Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0366308Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0366642Z res = mod(**inputs) 2025-12-04T09:00:27.0367004Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0367399Z outputs = self.model( 2025-12-04T09:00:27.0367770Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0368172Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0368553Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0368942Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0369333Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0369711Z return func(*args, **kwargs) 2025-12-04T09:00:27.0370097Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.0370517Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.0370925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0371298Z return func(*args, **kwargs) 2025-12-04T09:00:27.0371693Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 178, in forward 2025-12-04T09:00:27.0372105Z value_states = self.v_proj(current_states) 2025-12-04T09:00:27.0372253Z 2025-12-04T09:00:27.0372367Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0372734Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0373067Z res = mod(**inputs) 2025-12-04T09:00:27.0373429Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0373823Z outputs = self.model( 2025-12-04T09:00:27.0374190Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0374590Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0374951Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0375324Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0375717Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0376105Z return func(*args, **kwargs) 2025-12-04T09:00:27.0376478Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.0376900Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.0377326Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0377741Z return func(*args, **kwargs) 2025-12-04T09:00:27.0378111Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 245, in forward 2025-12-04T09:00:27.0378531Z attn_output = torch.bmm(attn_probs, value_states) 2025-12-04T09:00:27.0378687Z 2025-12-04T09:00:27.0378799Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0379166Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0379529Z res = mod(**inputs) 2025-12-04T09:00:27.0379925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0380313Z outputs = self.model( 2025-12-04T09:00:27.0380675Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0381087Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0381448Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0381817Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0382218Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0382597Z return func(*args, **kwargs) 2025-12-04T09:00:27.0382988Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.0383393Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.0383796Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0384186Z return func(*args, **kwargs) 2025-12-04T09:00:27.0384548Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-12-04T09:00:27.0384994Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-12-04T09:00:27.0385181Z 2025-12-04T09:00:27.0385284Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0385645Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0386031Z res = mod(**inputs) 2025-12-04T09:00:27.0386401Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0398899Z outputs = self.model( 2025-12-04T09:00:27.0399452Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0399889Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0400321Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0400758Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0401200Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0401653Z return func(*args, **kwargs) 2025-12-04T09:00:27.0402090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.0402583Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.0403046Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0403483Z return func(*args, **kwargs) 2025-12-04T09:00:27.0403911Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 260, in forward 2025-12-04T09:00:27.0404368Z attn_output = self.out_proj(attn_output) 2025-12-04T09:00:27.0404534Z 2025-12-04T09:00:27.0404659Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0405081Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0405459Z res = mod(**inputs) 2025-12-04T09:00:27.0405826Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0406227Z outputs = self.model( 2025-12-04T09:00:27.0406608Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0407010Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0407466Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0407846Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0408285Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0408672Z return func(*args, **kwargs) 2025-12-04T09:00:27.0409063Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-12-04T09:00:27.0409517Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:00:27.0409703Z 2025-12-04T09:00:27.0409833Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0410253Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0410619Z res = mod(**inputs) 2025-12-04T09:00:27.0410999Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0411414Z outputs = self.model( 2025-12-04T09:00:27.0411808Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0412209Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0412582Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0412954Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0413348Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0413769Z return func(*args, **kwargs) 2025-12-04T09:00:27.0414157Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-12-04T09:00:27.0414597Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:00:27.0415005Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:00:27.0415366Z return self.act(input) 2025-12-04T09:00:27.0415482Z 2025-12-04T09:00:27.0415601Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0415973Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0416333Z res = mod(**inputs) 2025-12-04T09:00:27.0416724Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0417138Z outputs = self.model( 2025-12-04T09:00:27.0417535Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0417966Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0418354Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0418752Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0419180Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0419569Z return func(*args, **kwargs) 2025-12-04T09:00:27.0419943Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 367, in forward 2025-12-04T09:00:27.0420356Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:00:27.0420514Z 2025-12-04T09:00:27.0420631Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0421246Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0421591Z res = mod(**inputs) 2025-12-04T09:00:27.0422050Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0422472Z outputs = self.model( 2025-12-04T09:00:27.0422864Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0423314Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0423701Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0424106Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0424518Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0424927Z return func(*args, **kwargs) 2025-12-04T09:00:27.0425365Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.0425821Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.0426261Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0426668Z return func(*args, **kwargs) 2025-12-04T09:00:27.0427081Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 157, in forward 2025-12-04T09:00:27.0427554Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:00:27.0427746Z 2025-12-04T09:00:27.0427862Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0428258Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0428611Z res = mod(**inputs) 2025-12-04T09:00:27.0429028Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0429452Z outputs = self.model( 2025-12-04T09:00:27.0429850Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0430266Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0430656Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0431053Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0431461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0431872Z return func(*args, **kwargs) 2025-12-04T09:00:27.0432275Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.0432719Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.0433154Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0433563Z return func(*args, **kwargs) 2025-12-04T09:00:27.0433967Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 177, in forward 2025-12-04T09:00:27.0434395Z key_states = self.k_proj(current_states) 2025-12-04T09:00:27.0434543Z 2025-12-04T09:00:27.0434659Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0435053Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0435408Z res = mod(**inputs) 2025-12-04T09:00:27.0435799Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0436228Z outputs = self.model( 2025-12-04T09:00:27.0436625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0437079Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0437486Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0437896Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0438416Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0438877Z return func(*args, **kwargs) 2025-12-04T09:00:27.0439297Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.0439810Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.0440271Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0440689Z return func(*args, **kwargs) 2025-12-04T09:00:27.0441144Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 157, in forward 2025-12-04T09:00:27.0441635Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:00:27.0441823Z 2025-12-04T09:00:27.0441949Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0442348Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0442719Z res = mod(**inputs) 2025-12-04T09:00:27.0443120Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0443553Z outputs = self.model( 2025-12-04T09:00:27.0443972Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0444408Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0444827Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0445231Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0445663Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0446090Z return func(*args, **kwargs) 2025-12-04T09:00:27.0446516Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.0446995Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.0447451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0447891Z return func(*args, **kwargs) 2025-12-04T09:00:27.0448311Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 199, in forward 2025-12-04T09:00:27.0448812Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-12-04T09:00:27.0449032Z 2025-12-04T09:00:27.0449147Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0449566Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0449930Z res = mod(**inputs) 2025-12-04T09:00:27.0450350Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0450776Z outputs = self.model( 2025-12-04T09:00:27.0451175Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0451607Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0451989Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0452393Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0452797Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0453246Z return func(*args, **kwargs) 2025-12-04T09:00:27.0453667Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.0454112Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.0454541Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0454989Z return func(*args, **kwargs) 2025-12-04T09:00:27.0455396Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 178, in forward 2025-12-04T09:00:27.0455826Z value_states = self.v_proj(current_states) 2025-12-04T09:00:27.0455989Z 2025-12-04T09:00:27.0456103Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0456514Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0456871Z res = mod(**inputs) 2025-12-04T09:00:27.0457257Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0457680Z outputs = self.model( 2025-12-04T09:00:27.0458081Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0458501Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0458895Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0459297Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0459716Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0460136Z return func(*args, **kwargs) 2025-12-04T09:00:27.0460536Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.0460983Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.0461416Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0461815Z return func(*args, **kwargs) 2025-12-04T09:00:27.0462219Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 245, in forward 2025-12-04T09:00:27.0462666Z attn_output = torch.bmm(attn_probs, value_states) 2025-12-04T09:00:27.0462827Z 2025-12-04T09:00:27.0462938Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0463330Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0463685Z res = mod(**inputs) 2025-12-04T09:00:27.0464076Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0464483Z outputs = self.model( 2025-12-04T09:00:27.0464886Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0465301Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0465678Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0466066Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0466481Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0466888Z return func(*args, **kwargs) 2025-12-04T09:00:27.0467287Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.0467726Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.0468158Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0468564Z return func(*args, **kwargs) 2025-12-04T09:00:27.0468975Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-12-04T09:00:27.0469458Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-12-04T09:00:27.0469682Z 2025-12-04T09:00:27.0469797Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0470197Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0470550Z res = mod(**inputs) 2025-12-04T09:00:27.0470939Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0471365Z outputs = self.model( 2025-12-04T09:00:27.0471781Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0472198Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0472584Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0472983Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0473392Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0473801Z return func(*args, **kwargs) 2025-12-04T09:00:27.0474208Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.0474660Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.0475093Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0475517Z return func(*args, **kwargs) 2025-12-04T09:00:27.0475920Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 260, in forward 2025-12-04T09:00:27.0476347Z attn_output = self.out_proj(attn_output) 2025-12-04T09:00:27.0476504Z 2025-12-04T09:00:27.0476616Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0477007Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0477359Z res = mod(**inputs) 2025-12-04T09:00:27.0477733Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0478243Z outputs = self.model( 2025-12-04T09:00:27.0478659Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0479104Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0479499Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0479895Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0480329Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0480748Z return func(*args, **kwargs) 2025-12-04T09:00:27.0481175Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-12-04T09:00:27.0481676Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:00:27.0481871Z 2025-12-04T09:00:27.0481998Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0482394Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0482756Z res = mod(**inputs) 2025-12-04T09:00:27.0483157Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0483596Z outputs = self.model( 2025-12-04T09:00:27.0484035Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0484473Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0484865Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0485282Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0485714Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0486145Z return func(*args, **kwargs) 2025-12-04T09:00:27.0486560Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-12-04T09:00:27.0487061Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:00:27.0487521Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:00:27.0487922Z return self.act(input) 2025-12-04T09:00:27.0488048Z 2025-12-04T09:00:27.0488170Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0488577Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0488995Z res = mod(**inputs) 2025-12-04T09:00:27.0489399Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0489827Z outputs = self.model( 2025-12-04T09:00:27.0490229Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0490676Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0491066Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0491499Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0491942Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0492360Z return func(*args, **kwargs) 2025-12-04T09:00:27.0492766Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 367, in forward 2025-12-04T09:00:27.0493220Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:00:27.0493370Z 2025-12-04T09:00:27.0493487Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0493865Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0494214Z res = mod(**inputs) 2025-12-04T09:00:27.0494596Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0495009Z outputs = self.model( 2025-12-04T09:00:27.0495393Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0495807Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0496189Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0496581Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0496988Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0497397Z return func(*args, **kwargs) 2025-12-04T09:00:27.0497799Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.0498238Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.0498676Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0499084Z return func(*args, **kwargs) 2025-12-04T09:00:27.0499501Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 157, in forward 2025-12-04T09:00:27.0499956Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:00:27.0500144Z 2025-12-04T09:00:27.0500256Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0500666Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0501009Z res = mod(**inputs) 2025-12-04T09:00:27.0501396Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0501813Z outputs = self.model( 2025-12-04T09:00:27.0502205Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0502619Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0503019Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0503413Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0503828Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0504224Z return func(*args, **kwargs) 2025-12-04T09:00:27.0504626Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.0505068Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.0505493Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0505895Z return func(*args, **kwargs) 2025-12-04T09:00:27.0506314Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 177, in forward 2025-12-04T09:00:27.0506748Z key_states = self.k_proj(current_states) 2025-12-04T09:00:27.0506894Z 2025-12-04T09:00:27.0507005Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0507392Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0507741Z res = mod(**inputs) 2025-12-04T09:00:27.0508118Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0508522Z outputs = self.model( 2025-12-04T09:00:27.0508905Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0509312Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0509681Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0510077Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0510497Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0510915Z return func(*args, **kwargs) 2025-12-04T09:00:27.0511317Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.0511776Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.0512218Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0512631Z return func(*args, **kwargs) 2025-12-04T09:00:27.0513038Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 157, in forward 2025-12-04T09:00:27.0513491Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:00:27.0513671Z 2025-12-04T09:00:27.0513788Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0514179Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0514542Z res = mod(**inputs) 2025-12-04T09:00:27.0514971Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0515393Z outputs = self.model( 2025-12-04T09:00:27.0515810Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0516247Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0516640Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0517047Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0517471Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0517932Z return func(*args, **kwargs) 2025-12-04T09:00:27.0518426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.0518900Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.0519346Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0519765Z return func(*args, **kwargs) 2025-12-04T09:00:27.0520169Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 199, in forward 2025-12-04T09:00:27.0520678Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-12-04T09:00:27.0521062Z 2025-12-04T09:00:27.0521183Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0521674Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0522100Z res = mod(**inputs) 2025-12-04T09:00:27.0522507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0522941Z outputs = self.model( 2025-12-04T09:00:27.0523346Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0523787Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0524191Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0524611Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0525041Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0525471Z return func(*args, **kwargs) 2025-12-04T09:00:27.0525912Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.0526389Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.0526825Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0527238Z return func(*args, **kwargs) 2025-12-04T09:00:27.0527653Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 178, in forward 2025-12-04T09:00:27.0528090Z value_states = self.v_proj(current_states) 2025-12-04T09:00:27.0528252Z 2025-12-04T09:00:27.0528368Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0528762Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0529122Z res = mod(**inputs) 2025-12-04T09:00:27.0529505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0529927Z outputs = self.model( 2025-12-04T09:00:27.0530326Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0530772Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0531155Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0531548Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0531994Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0532393Z return func(*args, **kwargs) 2025-12-04T09:00:27.0532791Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.0533232Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.0533663Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0534081Z return func(*args, **kwargs) 2025-12-04T09:00:27.0534482Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 245, in forward 2025-12-04T09:00:27.0534919Z attn_output = torch.bmm(attn_probs, value_states) 2025-12-04T09:00:27.0535077Z 2025-12-04T09:00:27.0535189Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0535579Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0535932Z res = mod(**inputs) 2025-12-04T09:00:27.0536316Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0536723Z outputs = self.model( 2025-12-04T09:00:27.0537117Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0537569Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0537950Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0538351Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0538770Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0539176Z return func(*args, **kwargs) 2025-12-04T09:00:27.0539578Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.0540024Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.0540461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0540868Z return func(*args, **kwargs) 2025-12-04T09:00:27.0541276Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-12-04T09:00:27.0541757Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-12-04T09:00:27.0541953Z 2025-12-04T09:00:27.0542075Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0542461Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0542813Z res = mod(**inputs) 2025-12-04T09:00:27.0543204Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0543622Z outputs = self.model( 2025-12-04T09:00:27.0544056Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0544459Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0544824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0545204Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0545604Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0546008Z return func(*args, **kwargs) 2025-12-04T09:00:27.0546391Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.0546827Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.0547243Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0547631Z return func(*args, **kwargs) 2025-12-04T09:00:27.0548014Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 260, in forward 2025-12-04T09:00:27.0548416Z attn_output = self.out_proj(attn_output) 2025-12-04T09:00:27.0548563Z 2025-12-04T09:00:27.0548686Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0549057Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0549386Z res = mod(**inputs) 2025-12-04T09:00:27.0549751Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0550147Z outputs = self.model( 2025-12-04T09:00:27.0550526Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0550921Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0551280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0551656Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0552045Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0552446Z return func(*args, **kwargs) 2025-12-04T09:00:27.0552826Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-12-04T09:00:27.0553273Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:00:27.0553450Z 2025-12-04T09:00:27.0553557Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0553931Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0554278Z res = mod(**inputs) 2025-12-04T09:00:27.0554670Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0555081Z outputs = self.model( 2025-12-04T09:00:27.0555475Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0555896Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0556272Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0556674Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0557091Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0557503Z return func(*args, **kwargs) 2025-12-04T09:00:27.0557905Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-12-04T09:00:27.0558467Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:00:27.0558912Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:00:27.0559299Z return self.act(input) 2025-12-04T09:00:27.0559440Z 2025-12-04T09:00:27.0559556Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0559964Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0560327Z res = mod(**inputs) 2025-12-04T09:00:27.0560741Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0561173Z outputs = self.model( 2025-12-04T09:00:27.0561585Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0562030Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0562425Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0562847Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0563278Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0563700Z return func(*args, **kwargs) 2025-12-04T09:00:27.0564145Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 367, in forward 2025-12-04T09:00:27.0564587Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:00:27.0564744Z 2025-12-04T09:00:27.0564865Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0565257Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0565639Z res = mod(**inputs) 2025-12-04T09:00:27.0566033Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0566461Z outputs = self.model( 2025-12-04T09:00:27.0566872Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0567308Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0567733Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0568133Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0568574Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0568993Z return func(*args, **kwargs) 2025-12-04T09:00:27.0569411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 369, in forward 2025-12-04T09:00:27.0569865Z hidden_states = residual + hidden_states 2025-12-04T09:00:27.0570023Z 2025-12-04T09:00:27.0570139Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0570541Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0570895Z res = mod(**inputs) 2025-12-04T09:00:27.0571305Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0571734Z outputs = self.model( 2025-12-04T09:00:27.0572139Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0572584Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0572979Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0573390Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0573796Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0574205Z return func(*args, **kwargs) 2025-12-04T09:00:27.0574607Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.0575052Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.0575487Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0575894Z return func(*args, **kwargs) 2025-12-04T09:00:27.0576316Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 157, in forward 2025-12-04T09:00:27.0576774Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:00:27.0576962Z 2025-12-04T09:00:27.0577093Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0577486Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0577837Z res = mod(**inputs) 2025-12-04T09:00:27.0578217Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0578631Z outputs = self.model( 2025-12-04T09:00:27.0579023Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0579447Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0579833Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0580230Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0580645Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0581049Z return func(*args, **kwargs) 2025-12-04T09:00:27.0581449Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.0581895Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.0582331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0582755Z return func(*args, **kwargs) 2025-12-04T09:00:27.0583156Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 177, in forward 2025-12-04T09:00:27.0583603Z key_states = self.k_proj(current_states) 2025-12-04T09:00:27.0583751Z 2025-12-04T09:00:27.0583866Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0584259Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0584612Z res = mod(**inputs) 2025-12-04T09:00:27.0584998Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0585405Z outputs = self.model( 2025-12-04T09:00:27.0585795Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0586212Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0586587Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0586983Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0587398Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0587802Z return func(*args, **kwargs) 2025-12-04T09:00:27.0588194Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.0588638Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.0589074Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0589476Z return func(*args, **kwargs) 2025-12-04T09:00:27.0589870Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 157, in forward 2025-12-04T09:00:27.0590332Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:00:27.0590510Z 2025-12-04T09:00:27.0590629Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0591019Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0591372Z res = mod(**inputs) 2025-12-04T09:00:27.0591754Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0592189Z outputs = self.model( 2025-12-04T09:00:27.0592572Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0592990Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0593370Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0593764Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0594178Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0594593Z return func(*args, **kwargs) 2025-12-04T09:00:27.0595006Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.0595448Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.0595891Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0596309Z return func(*args, **kwargs) 2025-12-04T09:00:27.0596715Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 199, in forward 2025-12-04T09:00:27.0597199Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-12-04T09:00:27.0597412Z 2025-12-04T09:00:27.0597528Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0597935Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0598358Z res = mod(**inputs) 2025-12-04T09:00:27.0598770Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0599205Z outputs = self.model( 2025-12-04T09:00:27.0599609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0600038Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0600420Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0600822Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0601234Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0601621Z return func(*args, **kwargs) 2025-12-04T09:00:27.0602003Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.0602422Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.0602825Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0603207Z return func(*args, **kwargs) 2025-12-04T09:00:27.0603586Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 178, in forward 2025-12-04T09:00:27.0603995Z value_states = self.v_proj(current_states) 2025-12-04T09:00:27.0604139Z 2025-12-04T09:00:27.0604244Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0604613Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0604945Z res = mod(**inputs) 2025-12-04T09:00:27.0605301Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0605692Z outputs = self.model( 2025-12-04T09:00:27.0606079Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0606471Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0606821Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0607213Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0607603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0607978Z return func(*args, **kwargs) 2025-12-04T09:00:27.0608354Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.0608774Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.0609200Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0609576Z return func(*args, **kwargs) 2025-12-04T09:00:27.0609954Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 245, in forward 2025-12-04T09:00:27.0610374Z attn_output = torch.bmm(attn_probs, value_states) 2025-12-04T09:00:27.0610527Z 2025-12-04T09:00:27.0610639Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0611002Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0611333Z res = mod(**inputs) 2025-12-04T09:00:27.0611699Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0612091Z outputs = self.model( 2025-12-04T09:00:27.0612503Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0612922Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0613305Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0613694Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0614106Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0614514Z return func(*args, **kwargs) 2025-12-04T09:00:27.0614906Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.0615350Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.0615788Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0616193Z return func(*args, **kwargs) 2025-12-04T09:00:27.0616588Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-12-04T09:00:27.0617064Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-12-04T09:00:27.0617266Z 2025-12-04T09:00:27.0617376Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0617767Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0618113Z res = mod(**inputs) 2025-12-04T09:00:27.0618496Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0618906Z outputs = self.model( 2025-12-04T09:00:27.0619285Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0619701Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0620087Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0620474Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0621063Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0621460Z return func(*args, **kwargs) 2025-12-04T09:00:27.0621847Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.0622326Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.0622743Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0623127Z return func(*args, **kwargs) 2025-12-04T09:00:27.0623519Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 260, in forward 2025-12-04T09:00:27.0623912Z attn_output = self.out_proj(attn_output) 2025-12-04T09:00:27.0624057Z 2025-12-04T09:00:27.0624194Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0624561Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0624883Z res = mod(**inputs) 2025-12-04T09:00:27.0625233Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0625614Z outputs = self.model( 2025-12-04T09:00:27.0625973Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0626350Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0626715Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0627089Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0627508Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0627888Z return func(*args, **kwargs) 2025-12-04T09:00:27.0628271Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-12-04T09:00:27.0628714Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:00:27.0628889Z 2025-12-04T09:00:27.0629002Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0629368Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0629700Z res = mod(**inputs) 2025-12-04T09:00:27.0630064Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0630445Z outputs = self.model( 2025-12-04T09:00:27.0630817Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0631214Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0631574Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0631946Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0632336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0632724Z return func(*args, **kwargs) 2025-12-04T09:00:27.0633101Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-12-04T09:00:27.0633565Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:00:27.0633984Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:00:27.0634361Z return self.act(input) 2025-12-04T09:00:27.0634481Z 2025-12-04T09:00:27.0634595Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0634986Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0635338Z res = mod(**inputs) 2025-12-04T09:00:27.0635735Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0636148Z outputs = self.model( 2025-12-04T09:00:27.0636559Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0636977Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0637349Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0637745Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0638209Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0638654Z return func(*args, **kwargs) 2025-12-04T09:00:27.0639062Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 367, in forward 2025-12-04T09:00:27.0639500Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:00:27.0639651Z 2025-12-04T09:00:27.0639773Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0640156Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0640513Z res = mod(**inputs) 2025-12-04T09:00:27.0640898Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0641310Z outputs = self.model( 2025-12-04T09:00:27.0641692Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0642125Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0642504Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0642907Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0643321Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0643729Z return func(*args, **kwargs) 2025-12-04T09:00:27.0644127Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.0644565Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.0644999Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0645404Z return func(*args, **kwargs) 2025-12-04T09:00:27.0645794Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 157, in forward 2025-12-04T09:00:27.0646254Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:00:27.0646438Z 2025-12-04T09:00:27.0646550Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0646944Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0647289Z res = mod(**inputs) 2025-12-04T09:00:27.0647673Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0648045Z outputs = self.model( 2025-12-04T09:00:27.0648401Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0648780Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0649134Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0649501Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0649879Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0650254Z return func(*args, **kwargs) 2025-12-04T09:00:27.0650638Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.0651051Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.0651472Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0651844Z return func(*args, **kwargs) 2025-12-04T09:00:27.0652207Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 177, in forward 2025-12-04T09:00:27.0652584Z key_states = self.k_proj(current_states) 2025-12-04T09:00:27.0652727Z 2025-12-04T09:00:27.0652828Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0653213Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0653550Z res = mod(**inputs) 2025-12-04T09:00:27.0653899Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0654282Z outputs = self.model( 2025-12-04T09:00:27.0654644Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0655030Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0655379Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0655744Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0656130Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0656517Z return func(*args, **kwargs) 2025-12-04T09:00:27.0656890Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.0657300Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.0657698Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0658066Z return func(*args, **kwargs) 2025-12-04T09:00:27.0658440Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 157, in forward 2025-12-04T09:00:27.0658864Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:00:27.0659028Z 2025-12-04T09:00:27.0659139Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0659490Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0659815Z res = mod(**inputs) 2025-12-04T09:00:27.0660176Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0660559Z outputs = self.model( 2025-12-04T09:00:27.0660929Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0661334Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0661686Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0662060Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0662455Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0662843Z return func(*args, **kwargs) 2025-12-04T09:00:27.0663215Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.0663638Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.0664051Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0664452Z return func(*args, **kwargs) 2025-12-04T09:00:27.0664828Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 199, in forward 2025-12-04T09:00:27.0665305Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-12-04T09:00:27.0665499Z 2025-12-04T09:00:27.0665612Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0665985Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0666310Z res = mod(**inputs) 2025-12-04T09:00:27.0666675Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0667067Z outputs = self.model( 2025-12-04T09:00:27.0667445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0667847Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0668215Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0668594Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0668986Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0669375Z return func(*args, **kwargs) 2025-12-04T09:00:27.0669758Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.0670183Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.0670632Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0671040Z return func(*args, **kwargs) 2025-12-04T09:00:27.0671450Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 178, in forward 2025-12-04T09:00:27.0671876Z value_states = self.v_proj(current_states) 2025-12-04T09:00:27.0672038Z 2025-12-04T09:00:27.0672150Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0672545Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0672887Z res = mod(**inputs) 2025-12-04T09:00:27.0673278Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0673689Z outputs = self.model( 2025-12-04T09:00:27.0674083Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0674499Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0674886Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0675298Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0675727Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0676139Z return func(*args, **kwargs) 2025-12-04T09:00:27.0676552Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.0677004Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.0677440Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0677858Z return func(*args, **kwargs) 2025-12-04T09:00:27.0678344Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 245, in forward 2025-12-04T09:00:27.0678827Z attn_output = torch.bmm(attn_probs, value_states) 2025-12-04T09:00:27.0678997Z 2025-12-04T09:00:27.0679110Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0679537Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0679895Z res = mod(**inputs) 2025-12-04T09:00:27.0680273Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0680721Z outputs = self.model( 2025-12-04T09:00:27.0681116Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0681528Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0681894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0682291Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0682728Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0683145Z return func(*args, **kwargs) 2025-12-04T09:00:27.0683545Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.0683994Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.0684431Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0684833Z return func(*args, **kwargs) 2025-12-04T09:00:27.0685238Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-12-04T09:00:27.0685721Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-12-04T09:00:27.0685935Z 2025-12-04T09:00:27.0686057Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0686442Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0686790Z res = mod(**inputs) 2025-12-04T09:00:27.0687178Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0687583Z outputs = self.model( 2025-12-04T09:00:27.0687978Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0688404Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0688785Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0689172Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0689587Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0689995Z return func(*args, **kwargs) 2025-12-04T09:00:27.0690405Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.0690818Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.0691232Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0691620Z return func(*args, **kwargs) 2025-12-04T09:00:27.0691995Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 260, in forward 2025-12-04T09:00:27.0692403Z attn_output = self.out_proj(attn_output) 2025-12-04T09:00:27.0692550Z 2025-12-04T09:00:27.0692656Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0693026Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0693360Z res = mod(**inputs) 2025-12-04T09:00:27.0693731Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0694124Z outputs = self.model( 2025-12-04T09:00:27.0694504Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0694901Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0695282Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0695656Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0696045Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0696429Z return func(*args, **kwargs) 2025-12-04T09:00:27.0696809Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-12-04T09:00:27.0697272Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:00:27.0697452Z 2025-12-04T09:00:27.0697557Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0697926Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0698257Z res = mod(**inputs) 2025-12-04T09:00:27.0698617Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0699004Z outputs = self.model( 2025-12-04T09:00:27.0699373Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0699769Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0700121Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0700516Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0700911Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0701289Z return func(*args, **kwargs) 2025-12-04T09:00:27.0701672Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-12-04T09:00:27.0702117Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:00:27.0702523Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:00:27.0702876Z return self.act(input) 2025-12-04T09:00:27.0702998Z 2025-12-04T09:00:27.0703105Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0703497Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0703852Z res = mod(**inputs) 2025-12-04T09:00:27.0704236Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0704660Z outputs = self.model( 2025-12-04T09:00:27.0705035Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0705423Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0705787Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0706164Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0706558Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0706942Z return func(*args, **kwargs) 2025-12-04T09:00:27.0707322Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 367, in forward 2025-12-04T09:00:27.0707732Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:00:27.0707875Z 2025-12-04T09:00:27.0707982Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0708372Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0708707Z res = mod(**inputs) 2025-12-04T09:00:27.0709073Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0709476Z outputs = self.model( 2025-12-04T09:00:27.0709850Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0710248Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0710603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0710977Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0711394Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0711788Z return func(*args, **kwargs) 2025-12-04T09:00:27.0712179Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 369, in forward 2025-12-04T09:00:27.0712609Z hidden_states = residual + hidden_states 2025-12-04T09:00:27.0712755Z 2025-12-04T09:00:27.0712875Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0713267Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0713612Z res = mod(**inputs) 2025-12-04T09:00:27.0714002Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0714392Z outputs = self.model( 2025-12-04T09:00:27.0714756Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0715167Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0715529Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0715902Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0716296Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0716703Z return func(*args, **kwargs) 2025-12-04T09:00:27.0717103Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.0717540Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.0717976Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0718472Z return func(*args, **kwargs) 2025-12-04T09:00:27.0718886Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 157, in forward 2025-12-04T09:00:27.0719338Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:00:27.0719528Z 2025-12-04T09:00:27.0719645Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0720041Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0720393Z res = mod(**inputs) 2025-12-04T09:00:27.0720929Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0721359Z outputs = self.model( 2025-12-04T09:00:27.0721760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0722182Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0722570Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0722975Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0723391Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0723842Z return func(*args, **kwargs) 2025-12-04T09:00:27.0724248Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.0724720Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.0725148Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0725562Z return func(*args, **kwargs) 2025-12-04T09:00:27.0725965Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 177, in forward 2025-12-04T09:00:27.0726392Z key_states = self.k_proj(current_states) 2025-12-04T09:00:27.0726539Z 2025-12-04T09:00:27.0726680Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0727079Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0727434Z res = mod(**inputs) 2025-12-04T09:00:27.0727821Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0728226Z outputs = self.model( 2025-12-04T09:00:27.0728622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0729043Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0729418Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0729811Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0730254Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0730665Z return func(*args, **kwargs) 2025-12-04T09:00:27.0731061Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.0731507Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.0731940Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0732344Z return func(*args, **kwargs) 2025-12-04T09:00:27.0732761Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 157, in forward 2025-12-04T09:00:27.0733244Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:00:27.0733425Z 2025-12-04T09:00:27.0733547Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0733945Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0734308Z res = mod(**inputs) 2025-12-04T09:00:27.0734719Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0735158Z outputs = self.model( 2025-12-04T09:00:27.0735554Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0735974Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0736359Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0736750Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0737168Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0737577Z return func(*args, **kwargs) 2025-12-04T09:00:27.0737981Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.0738421Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.0738894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0739307Z return func(*args, **kwargs) 2025-12-04T09:00:27.0739700Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 199, in forward 2025-12-04T09:00:27.0740205Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-12-04T09:00:27.0740416Z 2025-12-04T09:00:27.0740531Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0740924Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0741268Z res = mod(**inputs) 2025-12-04T09:00:27.0741655Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0742091Z outputs = self.model( 2025-12-04T09:00:27.0742484Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0742904Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0743294Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0743696Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0744104Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0744517Z return func(*args, **kwargs) 2025-12-04T09:00:27.0744921Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.0745370Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.0745823Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0746239Z return func(*args, **kwargs) 2025-12-04T09:00:27.0746641Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 178, in forward 2025-12-04T09:00:27.0747067Z value_states = self.v_proj(current_states) 2025-12-04T09:00:27.0747233Z 2025-12-04T09:00:27.0747346Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0747757Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0748112Z res = mod(**inputs) 2025-12-04T09:00:27.0748489Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0748904Z outputs = self.model( 2025-12-04T09:00:27.0749297Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0749716Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0750089Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0750484Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0750894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0751294Z return func(*args, **kwargs) 2025-12-04T09:00:27.0751691Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.0752134Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.0752601Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0753005Z return func(*args, **kwargs) 2025-12-04T09:00:27.0753405Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 245, in forward 2025-12-04T09:00:27.0753847Z attn_output = torch.bmm(attn_probs, value_states) 2025-12-04T09:00:27.0754010Z 2025-12-04T09:00:27.0754138Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0754526Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0754928Z res = mod(**inputs) 2025-12-04T09:00:27.0755315Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0755724Z outputs = self.model( 2025-12-04T09:00:27.0756120Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0756537Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0756915Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0757331Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0757754Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0758246Z return func(*args, **kwargs) 2025-12-04T09:00:27.0758670Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.0758797Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.0759072Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0759160Z return func(*args, **kwargs) 2025-12-04T09:00:27.0759492Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-12-04T09:00:27.0759654Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-12-04T09:00:27.0759659Z 2025-12-04T09:00:27.0759780Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0760001Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0760076Z res = mod(**inputs) 2025-12-04T09:00:27.0760360Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0760437Z outputs = self.model( 2025-12-04T09:00:27.0760718Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0760800Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0761047Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0761138Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0761395Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0761466Z return func(*args, **kwargs) 2025-12-04T09:00:27.0761732Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.0761843Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.0762108Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0762184Z return func(*args, **kwargs) 2025-12-04T09:00:27.0762456Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 260, in forward 2025-12-04T09:00:27.0762552Z attn_output = self.out_proj(attn_output) 2025-12-04T09:00:27.0762556Z 2025-12-04T09:00:27.0762668Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0762896Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0762969Z res = mod(**inputs) 2025-12-04T09:00:27.0763245Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0763351Z outputs = self.model( 2025-12-04T09:00:27.0763622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0763718Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0763971Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0764056Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0764327Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0764401Z return func(*args, **kwargs) 2025-12-04T09:00:27.0764685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-12-04T09:00:27.0764824Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:00:27.0764828Z 2025-12-04T09:00:27.0764942Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0765166Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0765236Z res = mod(**inputs) 2025-12-04T09:00:27.0765510Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0765592Z outputs = self.model( 2025-12-04T09:00:27.0765862Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0765940Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0766188Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0766292Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0766563Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0766639Z return func(*args, **kwargs) 2025-12-04T09:00:27.0766908Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-12-04T09:00:27.0767046Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:00:27.0767280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:00:27.0767356Z return self.act(input) 2025-12-04T09:00:27.0767367Z 2025-12-04T09:00:27.0767477Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0767694Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0767772Z res = mod(**inputs) 2025-12-04T09:00:27.0768043Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0768117Z outputs = self.model( 2025-12-04T09:00:27.0768392Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0768472Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0768729Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0768813Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0769074Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0769155Z return func(*args, **kwargs) 2025-12-04T09:00:27.0769420Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 367, in forward 2025-12-04T09:00:27.0769511Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:00:27.0769523Z 2025-12-04T09:00:27.0769634Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0769865Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0769942Z res = mod(**inputs) 2025-12-04T09:00:27.0770217Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0770310Z outputs = self.model( 2025-12-04T09:00:27.0770587Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0770665Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0770911Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0770995Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0771275Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0771356Z return func(*args, **kwargs) 2025-12-04T09:00:27.0771628Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.0771737Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.0772011Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0772085Z return func(*args, **kwargs) 2025-12-04T09:00:27.0772361Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 157, in forward 2025-12-04T09:00:27.0772484Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:00:27.0772514Z 2025-12-04T09:00:27.0772625Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0772851Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0772922Z res = mod(**inputs) 2025-12-04T09:00:27.0773194Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0773275Z outputs = self.model( 2025-12-04T09:00:27.0773545Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0773633Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0773874Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0773958Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0774231Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0774309Z return func(*args, **kwargs) 2025-12-04T09:00:27.0774582Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.0774691Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.0774971Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0775054Z return func(*args, **kwargs) 2025-12-04T09:00:27.0775324Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 177, in forward 2025-12-04T09:00:27.0775412Z key_states = self.k_proj(current_states) 2025-12-04T09:00:27.0775415Z 2025-12-04T09:00:27.0775533Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0775750Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0775831Z res = mod(**inputs) 2025-12-04T09:00:27.0776112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0776187Z outputs = self.model( 2025-12-04T09:00:27.0776493Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0776573Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0776813Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0776926Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0777193Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0777274Z return func(*args, **kwargs) 2025-12-04T09:00:27.0777546Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.0777653Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.0777955Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0778031Z return func(*args, **kwargs) 2025-12-04T09:00:27.0778307Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 157, in forward 2025-12-04T09:00:27.0778426Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:00:27.0778432Z 2025-12-04T09:00:27.0778542Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0778766Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0778837Z res = mod(**inputs) 2025-12-04T09:00:27.0779113Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0779215Z outputs = self.model( 2025-12-04T09:00:27.0779505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0779593Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0779835Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0779918Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0780207Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0780283Z return func(*args, **kwargs) 2025-12-04T09:00:27.0780577Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.0780681Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.0781006Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0781085Z return func(*args, **kwargs) 2025-12-04T09:00:27.0781343Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 199, in forward 2025-12-04T09:00:27.0781482Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-12-04T09:00:27.0781493Z 2025-12-04T09:00:27.0781596Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0781813Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0781894Z res = mod(**inputs) 2025-12-04T09:00:27.0782164Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0782240Z outputs = self.model( 2025-12-04T09:00:27.0782515Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0782597Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0782846Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0782929Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0783209Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0783291Z return func(*args, **kwargs) 2025-12-04T09:00:27.0783580Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.0783684Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.0783957Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0784030Z return func(*args, **kwargs) 2025-12-04T09:00:27.0784306Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 178, in forward 2025-12-04T09:00:27.0784417Z value_states = self.v_proj(current_states) 2025-12-04T09:00:27.0784422Z 2025-12-04T09:00:27.0784534Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0784767Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0784833Z res = mod(**inputs) 2025-12-04T09:00:27.0785096Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0785174Z outputs = self.model( 2025-12-04T09:00:27.0785427Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0785507Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0785733Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0785835Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0786102Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0786177Z return func(*args, **kwargs) 2025-12-04T09:00:27.0786452Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.0786555Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.0786816Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0786895Z return func(*args, **kwargs) 2025-12-04T09:00:27.0787161Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 245, in forward 2025-12-04T09:00:27.0787264Z attn_output = torch.bmm(attn_probs, value_states) 2025-12-04T09:00:27.0787270Z 2025-12-04T09:00:27.0787386Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0787599Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0787676Z res = mod(**inputs) 2025-12-04T09:00:27.0787943Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0788016Z outputs = self.model( 2025-12-04T09:00:27.0788291Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0788369Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0788615Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0788700Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0788946Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0789030Z return func(*args, **kwargs) 2025-12-04T09:00:27.0789297Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.0789417Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.0789706Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0789780Z return func(*args, **kwargs) 2025-12-04T09:00:27.0790075Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-12-04T09:00:27.0790217Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-12-04T09:00:27.0790220Z 2025-12-04T09:00:27.0790331Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0790555Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0790626Z res = mod(**inputs) 2025-12-04T09:00:27.0790913Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0790996Z outputs = self.model( 2025-12-04T09:00:27.0791266Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0791353Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0791597Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0791680Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0791960Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0792035Z return func(*args, **kwargs) 2025-12-04T09:00:27.0792314Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.0792430Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.0792678Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0792758Z return func(*args, **kwargs) 2025-12-04T09:00:27.0793011Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 260, in forward 2025-12-04T09:00:27.0793098Z attn_output = self.out_proj(attn_output) 2025-12-04T09:00:27.0793101Z 2025-12-04T09:00:27.0793211Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0793415Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0793492Z res = mod(**inputs) 2025-12-04T09:00:27.0793758Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0793832Z outputs = self.model( 2025-12-04T09:00:27.0794109Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0794191Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0794433Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0794524Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0794788Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0794870Z return func(*args, **kwargs) 2025-12-04T09:00:27.0795136Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-12-04T09:00:27.0795263Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:00:27.0795267Z 2025-12-04T09:00:27.0795388Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0795604Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0795680Z res = mod(**inputs) 2025-12-04T09:00:27.0795967Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0796042Z outputs = self.model( 2025-12-04T09:00:27.0796319Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0796415Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0796658Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0796747Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0797015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0797098Z return func(*args, **kwargs) 2025-12-04T09:00:27.0797395Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-12-04T09:00:27.0797527Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:00:27.0797772Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:00:27.0797849Z return self.act(input) 2025-12-04T09:00:27.0797855Z 2025-12-04T09:00:27.0797972Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0798259Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0798338Z res = mod(**inputs) 2025-12-04T09:00:27.0798622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0798698Z outputs = self.model( 2025-12-04T09:00:27.0799000Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0799093Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0799343Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0799440Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0799714Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0799792Z return func(*args, **kwargs) 2025-12-04T09:00:27.0800077Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 367, in forward 2025-12-04T09:00:27.0800170Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:00:27.0800174Z 2025-12-04T09:00:27.0800303Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0800523Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0800596Z res = mod(**inputs) 2025-12-04T09:00:27.0800871Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0800948Z outputs = self.model( 2025-12-04T09:00:27.0801230Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0801322Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0801563Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0801653Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0801917Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0801992Z return func(*args, **kwargs) 2025-12-04T09:00:27.0802268Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 369, in forward 2025-12-04T09:00:27.0802356Z hidden_states = residual + hidden_states 2025-12-04T09:00:27.0802359Z 2025-12-04T09:00:27.0802489Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0802714Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0802782Z res = mod(**inputs) 2025-12-04T09:00:27.0803075Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0803148Z outputs = self.model( 2025-12-04T09:00:27.0803414Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0803500Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0803742Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0803845Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0804120Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0804196Z return func(*args, **kwargs) 2025-12-04T09:00:27.0804472Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.0804578Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.0804841Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0804923Z return func(*args, **kwargs) 2025-12-04T09:00:27.0805192Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 157, in forward 2025-12-04T09:00:27.0805322Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:00:27.0805340Z 2025-12-04T09:00:27.0805454Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0805672Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0805752Z res = mod(**inputs) 2025-12-04T09:00:27.0806022Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0806106Z outputs = self.model( 2025-12-04T09:00:27.0806372Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0806446Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0806685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0806765Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0807017Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0807098Z return func(*args, **kwargs) 2025-12-04T09:00:27.0807354Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.0807462Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.0807712Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0807783Z return func(*args, **kwargs) 2025-12-04T09:00:27.0808050Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 177, in forward 2025-12-04T09:00:27.0808133Z key_states = self.k_proj(current_states) 2025-12-04T09:00:27.0808137Z 2025-12-04T09:00:27.0808241Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0808452Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0808520Z res = mod(**inputs) 2025-12-04T09:00:27.0808785Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0808856Z outputs = self.model( 2025-12-04T09:00:27.0809127Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0809211Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0809457Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0809536Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0809793Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0809862Z return func(*args, **kwargs) 2025-12-04T09:00:27.0810123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.0810239Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.0810490Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0810566Z return func(*args, **kwargs) 2025-12-04T09:00:27.0810820Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 157, in forward 2025-12-04T09:00:27.0810941Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:00:27.0810944Z 2025-12-04T09:00:27.0811048Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0811248Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0811321Z res = mod(**inputs) 2025-12-04T09:00:27.0811583Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0811670Z outputs = self.model( 2025-12-04T09:00:27.0811940Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0812016Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0812261Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0812342Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0812598Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0812676Z return func(*args, **kwargs) 2025-12-04T09:00:27.0812937Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.0813044Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.0813304Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0813375Z return func(*args, **kwargs) 2025-12-04T09:00:27.0813643Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 199, in forward 2025-12-04T09:00:27.0813781Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-12-04T09:00:27.0813785Z 2025-12-04T09:00:27.0814429Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0814640Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0814708Z res = mod(**inputs) 2025-12-04T09:00:27.0814983Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0815054Z outputs = self.model( 2025-12-04T09:00:27.0815315Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0815401Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0815634Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0815744Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0815997Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0816085Z return func(*args, **kwargs) 2025-12-04T09:00:27.0816345Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.0816447Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.0816695Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0816771Z return func(*args, **kwargs) 2025-12-04T09:00:27.0817042Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 178, in forward 2025-12-04T09:00:27.0817138Z value_states = self.v_proj(current_states) 2025-12-04T09:00:27.0817141Z 2025-12-04T09:00:27.0817248Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0817452Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0817523Z res = mod(**inputs) 2025-12-04T09:00:27.0817778Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0817854Z outputs = self.model( 2025-12-04T09:00:27.0818109Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0818184Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0818418Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0818521Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0818768Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0818849Z return func(*args, **kwargs) 2025-12-04T09:00:27.0819103Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.0819211Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.0819462Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0819532Z return func(*args, **kwargs) 2025-12-04T09:00:27.0819802Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 245, in forward 2025-12-04T09:00:27.0819902Z attn_output = torch.bmm(attn_probs, value_states) 2025-12-04T09:00:27.0819906Z 2025-12-04T09:00:27.0820017Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0820221Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0820291Z res = mod(**inputs) 2025-12-04T09:00:27.0820551Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0820623Z outputs = self.model( 2025-12-04T09:00:27.0821034Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0821123Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0821351Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0821441Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0821694Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0821766Z return func(*args, **kwargs) 2025-12-04T09:00:27.0822076Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.0822177Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.0822428Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0822542Z return func(*args, **kwargs) 2025-12-04T09:00:27.0822799Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-12-04T09:00:27.0822941Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-12-04T09:00:27.0822945Z 2025-12-04T09:00:27.0823048Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0823257Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0823357Z res = mod(**inputs) 2025-12-04T09:00:27.0823611Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0823691Z outputs = self.model( 2025-12-04T09:00:27.0823946Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0824022Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0824259Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0824350Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0824596Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0824675Z return func(*args, **kwargs) 2025-12-04T09:00:27.0824955Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.0825061Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.0825310Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0825381Z return func(*args, **kwargs) 2025-12-04T09:00:27.0825649Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 260, in forward 2025-12-04T09:00:27.0825734Z attn_output = self.out_proj(attn_output) 2025-12-04T09:00:27.0825738Z 2025-12-04T09:00:27.0825848Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0826049Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0826115Z res = mod(**inputs) 2025-12-04T09:00:27.0826385Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0826458Z outputs = self.model( 2025-12-04T09:00:27.0826710Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0826802Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0827026Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0827110Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0827356Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0827425Z return func(*args, **kwargs) 2025-12-04T09:00:27.0827678Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-12-04T09:00:27.0827799Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:00:27.0827804Z 2025-12-04T09:00:27.0827906Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0828110Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0828175Z res = mod(**inputs) 2025-12-04T09:00:27.0828449Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0828519Z outputs = self.model( 2025-12-04T09:00:27.0828781Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0828861Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0829084Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0829169Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0829412Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0829497Z return func(*args, **kwargs) 2025-12-04T09:00:27.0829749Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-12-04T09:00:27.0829868Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:00:27.0830081Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:00:27.0830160Z return self.act(input) 2025-12-04T09:00:27.0830164Z 2025-12-04T09:00:27.0830264Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0830467Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0830531Z res = mod(**inputs) 2025-12-04T09:00:27.0830776Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0830866Z outputs = self.model( 2025-12-04T09:00:27.0831114Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0831185Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0831415Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0831493Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0831756Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0831828Z return func(*args, **kwargs) 2025-12-04T09:00:27.0832078Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 367, in forward 2025-12-04T09:00:27.0832168Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:00:27.0832172Z 2025-12-04T09:00:27.0832279Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0832487Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0832555Z res = mod(**inputs) 2025-12-04T09:00:27.0832810Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0832886Z outputs = self.model( 2025-12-04T09:00:27.0833138Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0833213Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0833458Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0833541Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0833811Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0833887Z return func(*args, **kwargs) 2025-12-04T09:00:27.0834157Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.0834269Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.0834545Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0834621Z return func(*args, **kwargs) 2025-12-04T09:00:27.0834915Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 157, in forward 2025-12-04T09:00:27.0835035Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:00:27.0835039Z 2025-12-04T09:00:27.0835155Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0835370Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0835443Z res = mod(**inputs) 2025-12-04T09:00:27.0835735Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0835810Z outputs = self.model( 2025-12-04T09:00:27.0836090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0836171Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0836412Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0836505Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0836767Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0836842Z return func(*args, **kwargs) 2025-12-04T09:00:27.0837114Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.0837235Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.0837508Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0837582Z return func(*args, **kwargs) 2025-12-04T09:00:27.0837855Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 177, in forward 2025-12-04T09:00:27.0837950Z key_states = self.k_proj(current_states) 2025-12-04T09:00:27.0837955Z 2025-12-04T09:00:27.0838067Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0838352Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0838429Z res = mod(**inputs) 2025-12-04T09:00:27.0838709Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0838796Z outputs = self.model( 2025-12-04T09:00:27.0839073Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0839162Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0839410Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0839495Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0839765Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0839843Z return func(*args, **kwargs) 2025-12-04T09:00:27.0840112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.0840226Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.0840487Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0840564Z return func(*args, **kwargs) 2025-12-04T09:00:27.0840838Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 157, in forward 2025-12-04T09:00:27.0840976Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:00:27.0840981Z 2025-12-04T09:00:27.0841103Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0841318Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0841400Z res = mod(**inputs) 2025-12-04T09:00:27.0841666Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0841737Z outputs = self.model( 2025-12-04T09:00:27.0841999Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0842076Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0842320Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0842413Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0842677Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0842751Z return func(*args, **kwargs) 2025-12-04T09:00:27.0843025Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.0843131Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.0843399Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0843472Z return func(*args, **kwargs) 2025-12-04T09:00:27.0843738Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 199, in forward 2025-12-04T09:00:27.0843911Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-12-04T09:00:27.0843914Z 2025-12-04T09:00:27.0844025Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0844247Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0844316Z res = mod(**inputs) 2025-12-04T09:00:27.0844583Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0844666Z outputs = self.model( 2025-12-04T09:00:27.0844932Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0845011Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0845280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0845367Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0845636Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0845709Z return func(*args, **kwargs) 2025-12-04T09:00:27.0845979Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.0846091Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.0846352Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0846424Z return func(*args, **kwargs) 2025-12-04T09:00:27.0846697Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 178, in forward 2025-12-04T09:00:27.0846790Z value_states = self.v_proj(current_states) 2025-12-04T09:00:27.0846794Z 2025-12-04T09:00:27.0846912Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0847126Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0847195Z res = mod(**inputs) 2025-12-04T09:00:27.0847493Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0847568Z outputs = self.model( 2025-12-04T09:00:27.0847843Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0847939Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0848178Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0848269Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0848528Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0848604Z return func(*args, **kwargs) 2025-12-04T09:00:27.0848893Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.0849000Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.0849271Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0849345Z return func(*args, **kwargs) 2025-12-04T09:00:27.0849617Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 245, in forward 2025-12-04T09:00:27.0849728Z attn_output = torch.bmm(attn_probs, value_states) 2025-12-04T09:00:27.0849732Z 2025-12-04T09:00:27.0849842Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0850064Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0850151Z res = mod(**inputs) 2025-12-04T09:00:27.0850421Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0850502Z outputs = self.model( 2025-12-04T09:00:27.0850771Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0850850Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0851100Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0851186Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0851463Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0851532Z return func(*args, **kwargs) 2025-12-04T09:00:27.0851781Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.0851887Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.0852131Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0852203Z return func(*args, **kwargs) 2025-12-04T09:00:27.0852455Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-12-04T09:00:27.0852592Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-12-04T09:00:27.0852597Z 2025-12-04T09:00:27.0852709Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0852912Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0852980Z res = mod(**inputs) 2025-12-04T09:00:27.0853253Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0853324Z outputs = self.model( 2025-12-04T09:00:27.0853586Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0853661Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0853909Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0853996Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0854241Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0854332Z return func(*args, **kwargs) 2025-12-04T09:00:27.0854598Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.0854696Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.0854953Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0855039Z return func(*args, **kwargs) 2025-12-04T09:00:27.0855295Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 260, in forward 2025-12-04T09:00:27.0855388Z attn_output = self.out_proj(attn_output) 2025-12-04T09:00:27.0855392Z 2025-12-04T09:00:27.0855495Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0855704Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0855783Z res = mod(**inputs) 2025-12-04T09:00:27.0856027Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0856102Z outputs = self.model( 2025-12-04T09:00:27.0856396Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0856486Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0856720Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0856798Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0857057Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0857128Z return func(*args, **kwargs) 2025-12-04T09:00:27.0857380Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-12-04T09:00:27.0857511Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:00:27.0857515Z 2025-12-04T09:00:27.0857618Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0857819Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0857893Z res = mod(**inputs) 2025-12-04T09:00:27.0858144Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0858222Z outputs = self.model( 2025-12-04T09:00:27.0858476Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0858550Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0858789Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0858875Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0859147Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0859222Z return func(*args, **kwargs) 2025-12-04T09:00:27.0859488Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-12-04T09:00:27.0859623Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:00:27.0859857Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:00:27.0859931Z return self.act(input) 2025-12-04T09:00:27.0859984Z 2025-12-04T09:00:27.0860107Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0860311Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0860401Z res = mod(**inputs) 2025-12-04T09:00:27.0860653Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0860721Z outputs = self.model( 2025-12-04T09:00:27.0860982Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0861056Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0861303Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0861390Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0861640Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0861718Z return func(*args, **kwargs) 2025-12-04T09:00:27.0861969Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 367, in forward 2025-12-04T09:00:27.0862054Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:00:27.0862057Z 2025-12-04T09:00:27.0862167Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0862371Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0862444Z res = mod(**inputs) 2025-12-04T09:00:27.0862703Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0862786Z outputs = self.model( 2025-12-04T09:00:27.0863044Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0863120Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0863348Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0863433Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0863687Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0863765Z return func(*args, **kwargs) 2025-12-04T09:00:27.0864024Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 369, in forward 2025-12-04T09:00:27.0864106Z hidden_states = residual + hidden_states 2025-12-04T09:00:27.0864111Z 2025-12-04T09:00:27.0864223Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0864429Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0864501Z res = mod(**inputs) 2025-12-04T09:00:27.0864762Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0864833Z outputs = self.model( 2025-12-04T09:00:27.0865097Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0865173Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0865404Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0865495Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0865762Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0865845Z return func(*args, **kwargs) 2025-12-04T09:00:27.0866125Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.0866248Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.0866521Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0866612Z return func(*args, **kwargs) 2025-12-04T09:00:27.0866880Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 157, in forward 2025-12-04T09:00:27.0867010Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:00:27.0867013Z 2025-12-04T09:00:27.0867125Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0867355Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0867427Z res = mod(**inputs) 2025-12-04T09:00:27.0867714Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0867797Z outputs = self.model( 2025-12-04T09:00:27.0868075Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0868155Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0868386Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0868466Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0868723Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0868794Z return func(*args, **kwargs) 2025-12-04T09:00:27.0869044Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.0869187Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.0869451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0869533Z return func(*args, **kwargs) 2025-12-04T09:00:27.0869800Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 177, in forward 2025-12-04T09:00:27.0869890Z key_states = self.k_proj(current_states) 2025-12-04T09:00:27.0869893Z 2025-12-04T09:00:27.0870010Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0870224Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0870301Z res = mod(**inputs) 2025-12-04T09:00:27.0870568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0870643Z outputs = self.model( 2025-12-04T09:00:27.0870921Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0870998Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0871243Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0871335Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0871600Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0871681Z return func(*args, **kwargs) 2025-12-04T09:00:27.0871949Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.0872052Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.0872321Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0872398Z return func(*args, **kwargs) 2025-12-04T09:00:27.0872667Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 157, in forward 2025-12-04T09:00:27.0872810Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:00:27.0872814Z 2025-12-04T09:00:27.0872924Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0873164Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0873234Z res = mod(**inputs) 2025-12-04T09:00:27.0873503Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0873584Z outputs = self.model( 2025-12-04T09:00:27.0873853Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0873939Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0874195Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0874283Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0874552Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0874627Z return func(*args, **kwargs) 2025-12-04T09:00:27.0874894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.0875009Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.0875270Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0875351Z return func(*args, **kwargs) 2025-12-04T09:00:27.0875635Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 199, in forward 2025-12-04T09:00:27.0875785Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-12-04T09:00:27.0875788Z 2025-12-04T09:00:27.0875909Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0876122Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0876197Z res = mod(**inputs) 2025-12-04T09:00:27.0876468Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0876542Z outputs = self.model( 2025-12-04T09:00:27.0876818Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0876894Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0877135Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0877228Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0877491Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0877573Z return func(*args, **kwargs) 2025-12-04T09:00:27.0877844Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.0877948Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.0878284Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0878366Z return func(*args, **kwargs) 2025-12-04T09:00:27.0878632Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 178, in forward 2025-12-04T09:00:27.0878735Z value_states = self.v_proj(current_states) 2025-12-04T09:00:27.0878742Z 2025-12-04T09:00:27.0878853Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0879076Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0879146Z res = mod(**inputs) 2025-12-04T09:00:27.0879438Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0879520Z outputs = self.model( 2025-12-04T09:00:27.0879810Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0879898Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0880143Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0880227Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0880500Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0880591Z return func(*args, **kwargs) 2025-12-04T09:00:27.0880860Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.0880973Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.0881234Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0881320Z return func(*args, **kwargs) 2025-12-04T09:00:27.0881590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 245, in forward 2025-12-04T09:00:27.0881700Z attn_output = torch.bmm(attn_probs, value_states) 2025-12-04T09:00:27.0881704Z 2025-12-04T09:00:27.0881815Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0882017Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0882108Z res = mod(**inputs) 2025-12-04T09:00:27.0882365Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0882435Z outputs = self.model( 2025-12-04T09:00:27.0882698Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0882775Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0883004Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0883091Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0883339Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0883416Z return func(*args, **kwargs) 2025-12-04T09:00:27.0883672Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.0883774Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.0884031Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0884099Z return func(*args, **kwargs) 2025-12-04T09:00:27.0884366Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-12-04T09:00:27.0884509Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-12-04T09:00:27.0884512Z 2025-12-04T09:00:27.0884616Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0884827Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0884893Z res = mod(**inputs) 2025-12-04T09:00:27.0885147Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0885224Z outputs = self.model( 2025-12-04T09:00:27.0885479Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0885576Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0885804Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0885884Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0886158Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0886228Z return func(*args, **kwargs) 2025-12-04T09:00:27.0886480Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.0886587Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.0886853Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0886933Z return func(*args, **kwargs) 2025-12-04T09:00:27.0887188Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 260, in forward 2025-12-04T09:00:27.0887272Z attn_output = self.out_proj(attn_output) 2025-12-04T09:00:27.0887275Z 2025-12-04T09:00:27.0887386Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0887592Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0887663Z res = mod(**inputs) 2025-12-04T09:00:27.0887921Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0887991Z outputs = self.model( 2025-12-04T09:00:27.0888251Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0888348Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0888579Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0888669Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0888917Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0888994Z return func(*args, **kwargs) 2025-12-04T09:00:27.0889246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-12-04T09:00:27.0889370Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:00:27.0889374Z 2025-12-04T09:00:27.0889486Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0889688Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0889756Z res = mod(**inputs) 2025-12-04T09:00:27.0890019Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0890089Z outputs = self.model( 2025-12-04T09:00:27.0890351Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0890425Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0890653Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0890740Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0890987Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0891065Z return func(*args, **kwargs) 2025-12-04T09:00:27.0891316Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-12-04T09:00:27.0891439Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:00:27.0891666Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:00:27.0891752Z return self.act(input) 2025-12-04T09:00:27.0891756Z 2025-12-04T09:00:27.0891864Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0892074Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0892156Z res = mod(**inputs) 2025-12-04T09:00:27.0892418Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0892487Z outputs = self.model( 2025-12-04T09:00:27.0892743Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0892826Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0893079Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0893157Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0893414Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0893484Z return func(*args, **kwargs) 2025-12-04T09:00:27.0893745Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 367, in forward 2025-12-04T09:00:27.0893830Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:00:27.0893833Z 2025-12-04T09:00:27.0893937Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0894145Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0894210Z res = mod(**inputs) 2025-12-04T09:00:27.0894490Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0894560Z outputs = self.model( 2025-12-04T09:00:27.0894815Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0894896Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0895122Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0895203Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0895459Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0895529Z return func(*args, **kwargs) 2025-12-04T09:00:27.0895788Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.0895890Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.0896144Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0896222Z return func(*args, **kwargs) 2025-12-04T09:00:27.0896479Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 157, in forward 2025-12-04T09:00:27.0896596Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:00:27.0896608Z 2025-12-04T09:00:27.0896718Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0896933Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0897009Z res = mod(**inputs) 2025-12-04T09:00:27.0897285Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0897356Z outputs = self.model( 2025-12-04T09:00:27.0897619Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0897694Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0897946Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0898027Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0898277Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0898374Z return func(*args, **kwargs) 2025-12-04T09:00:27.0898630Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.0898730Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.0898986Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0899057Z return func(*args, **kwargs) 2025-12-04T09:00:27.0899332Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 177, in forward 2025-12-04T09:00:27.0899416Z key_states = self.k_proj(current_states) 2025-12-04T09:00:27.0899421Z 2025-12-04T09:00:27.0899528Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0899740Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0899809Z res = mod(**inputs) 2025-12-04T09:00:27.0900068Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0900137Z outputs = self.model( 2025-12-04T09:00:27.0900387Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0900478Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0900720Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0900796Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0901048Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0901116Z return func(*args, **kwargs) 2025-12-04T09:00:27.0901368Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.0901465Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.0901705Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0901782Z return func(*args, **kwargs) 2025-12-04T09:00:27.0902026Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 157, in forward 2025-12-04T09:00:27.0902136Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:00:27.0902150Z 2025-12-04T09:00:27.0902249Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0902451Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0902523Z res = mod(**inputs) 2025-12-04T09:00:27.0902769Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0902840Z outputs = self.model( 2025-12-04T09:00:27.0903091Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0903163Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0903392Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0903473Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0903724Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0903801Z return func(*args, **kwargs) 2025-12-04T09:00:27.0904069Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.0904169Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.0904426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0904522Z return func(*args, **kwargs) 2025-12-04T09:00:27.0904781Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 199, in forward 2025-12-04T09:00:27.0904919Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-12-04T09:00:27.0904922Z 2025-12-04T09:00:27.0905028Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0905254Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0905321Z res = mod(**inputs) 2025-12-04T09:00:27.0905586Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0905656Z outputs = self.model( 2025-12-04T09:00:27.0905905Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0905989Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0906213Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0906292Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0906553Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0906640Z return func(*args, **kwargs) 2025-12-04T09:00:27.0906892Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.0906988Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.0907231Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0907307Z return func(*args, **kwargs) 2025-12-04T09:00:27.0907551Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 178, in forward 2025-12-04T09:00:27.0907637Z value_states = self.v_proj(current_states) 2025-12-04T09:00:27.0907647Z 2025-12-04T09:00:27.0907749Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0907950Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0908023Z res = mod(**inputs) 2025-12-04T09:00:27.0908274Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0908343Z outputs = self.model( 2025-12-04T09:00:27.0908604Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0908678Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0908911Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0908991Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0909247Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0909323Z return func(*args, **kwargs) 2025-12-04T09:00:27.0909580Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.0909681Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.0909936Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0910006Z return func(*args, **kwargs) 2025-12-04T09:00:27.0910283Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 245, in forward 2025-12-04T09:00:27.0910383Z attn_output = torch.bmm(attn_probs, value_states) 2025-12-04T09:00:27.0910402Z 2025-12-04T09:00:27.0910506Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0910717Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0910784Z res = mod(**inputs) 2025-12-04T09:00:27.0911050Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0911125Z outputs = self.model( 2025-12-04T09:00:27.0911394Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0911481Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0911724Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0911809Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0912089Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0912161Z return func(*args, **kwargs) 2025-12-04T09:00:27.0912423Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.0912524Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.0912776Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0912874Z return func(*args, **kwargs) 2025-12-04T09:00:27.0913132Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-12-04T09:00:27.0913265Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-12-04T09:00:27.0913279Z 2025-12-04T09:00:27.0913385Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0913589Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0913665Z res = mod(**inputs) 2025-12-04T09:00:27.0913922Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0913992Z outputs = self.model( 2025-12-04T09:00:27.0914254Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0914331Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0914573Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0914654Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0914910Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0914989Z return func(*args, **kwargs) 2025-12-04T09:00:27.0915261Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.0915366Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.0915642Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0915716Z return func(*args, **kwargs) 2025-12-04T09:00:27.0915998Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 260, in forward 2025-12-04T09:00:27.0916090Z attn_output = self.out_proj(attn_output) 2025-12-04T09:00:27.0916093Z 2025-12-04T09:00:27.0916205Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0916449Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0916519Z res = mod(**inputs) 2025-12-04T09:00:27.0916796Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0916899Z outputs = self.model( 2025-12-04T09:00:27.0917168Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0917254Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0917492Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0917577Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0917863Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0917940Z return func(*args, **kwargs) 2025-12-04T09:00:27.0918419Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-12-04T09:00:27.0918555Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:00:27.0918561Z 2025-12-04T09:00:27.0918673Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0918904Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0918977Z res = mod(**inputs) 2025-12-04T09:00:27.0919255Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0919340Z outputs = self.model( 2025-12-04T09:00:27.0919640Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0919730Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0919981Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0920069Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0920353Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0920426Z return func(*args, **kwargs) 2025-12-04T09:00:27.0920869Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-12-04T09:00:27.0920999Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:00:27.0921222Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:00:27.0921306Z return self.act(input) 2025-12-04T09:00:27.0921309Z 2025-12-04T09:00:27.0921418Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0921622Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0921699Z res = mod(**inputs) 2025-12-04T09:00:27.0921965Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0922045Z outputs = self.model( 2025-12-04T09:00:27.0922298Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0922374Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0922612Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0922691Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0922949Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0923022Z return func(*args, **kwargs) 2025-12-04T09:00:27.0923318Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 367, in forward 2025-12-04T09:00:27.0923411Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:00:27.0923415Z 2025-12-04T09:00:27.0923520Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0923753Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0923827Z res = mod(**inputs) 2025-12-04T09:00:27.0924079Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0924153Z outputs = self.model( 2025-12-04T09:00:27.0924404Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0924481Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0924740Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0924823Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0925069Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0925149Z return func(*args, **kwargs) 2025-12-04T09:00:27.0925401Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 369, in forward 2025-12-04T09:00:27.0925491Z hidden_states = residual + hidden_states 2025-12-04T09:00:27.0925495Z 2025-12-04T09:00:27.0925597Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0925799Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0925893Z res = mod(**inputs) 2025-12-04T09:00:27.0926155Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0926231Z outputs = self.model( 2025-12-04T09:00:27.0926486Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0926559Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0926795Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0926876Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0927137Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0927215Z return func(*args, **kwargs) 2025-12-04T09:00:27.0927470Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.0927580Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.0927830Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0927903Z return func(*args, **kwargs) 2025-12-04T09:00:27.0928165Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 157, in forward 2025-12-04T09:00:27.0928279Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:00:27.0928284Z 2025-12-04T09:00:27.0928394Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0928596Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0928663Z res = mod(**inputs) 2025-12-04T09:00:27.0928924Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0928995Z outputs = self.model( 2025-12-04T09:00:27.0929251Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0929334Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0929581Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0929667Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0929916Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0930004Z return func(*args, **kwargs) 2025-12-04T09:00:27.0930270Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.0930369Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.0930629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0930720Z return func(*args, **kwargs) 2025-12-04T09:00:27.0930963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 177, in forward 2025-12-04T09:00:27.0931053Z key_states = self.k_proj(current_states) 2025-12-04T09:00:27.0931056Z 2025-12-04T09:00:27.0931157Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0931353Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0931427Z res = mod(**inputs) 2025-12-04T09:00:27.0931674Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0931750Z outputs = self.model( 2025-12-04T09:00:27.0932000Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0932087Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0932318Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0932395Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0932634Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0932712Z return func(*args, **kwargs) 2025-12-04T09:00:27.0932957Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.0933061Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.0933302Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0933369Z return func(*args, **kwargs) 2025-12-04T09:00:27.0933622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 157, in forward 2025-12-04T09:00:27.0933735Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:00:27.0933738Z 2025-12-04T09:00:27.0933845Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0934043Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0934108Z res = mod(**inputs) 2025-12-04T09:00:27.0934361Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0934431Z outputs = self.model( 2025-12-04T09:00:27.0934678Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0934758Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0934978Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0935064Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0935313Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0935384Z return func(*args, **kwargs) 2025-12-04T09:00:27.0935664Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.0935764Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.0936032Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0936110Z return func(*args, **kwargs) 2025-12-04T09:00:27.0936367Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 199, in forward 2025-12-04T09:00:27.0936510Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-12-04T09:00:27.0936515Z 2025-12-04T09:00:27.0936619Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0936837Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0936912Z res = mod(**inputs) 2025-12-04T09:00:27.0937181Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0937255Z outputs = self.model( 2025-12-04T09:00:27.0937509Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0937586Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0937819Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0937899Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0938145Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0938242Z return func(*args, **kwargs) 2025-12-04T09:00:27.0938494Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.0938601Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.0938848Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0938919Z return func(*args, **kwargs) 2025-12-04T09:00:27.0939177Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 178, in forward 2025-12-04T09:00:27.0939264Z value_states = self.v_proj(current_states) 2025-12-04T09:00:27.0939268Z 2025-12-04T09:00:27.0939378Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0939582Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0939650Z res = mod(**inputs) 2025-12-04T09:00:27.0939908Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0939977Z outputs = self.model( 2025-12-04T09:00:27.0940236Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0940315Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0940533Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0940620Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0940860Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0940927Z return func(*args, **kwargs) 2025-12-04T09:00:27.0941178Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.0941277Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.0941517Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0941622Z return func(*args, **kwargs) 2025-12-04T09:00:27.0941866Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 245, in forward 2025-12-04T09:00:27.0941983Z attn_output = torch.bmm(attn_probs, value_states) 2025-12-04T09:00:27.0941986Z 2025-12-04T09:00:27.0942087Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0942285Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0942358Z res = mod(**inputs) 2025-12-04T09:00:27.0942600Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0942678Z outputs = self.model( 2025-12-04T09:00:27.0942951Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0943026Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0943263Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0943341Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0943582Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0943661Z return func(*args, **kwargs) 2025-12-04T09:00:27.0943908Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.0944012Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.0944257Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0944344Z return func(*args, **kwargs) 2025-12-04T09:00:27.0944601Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-12-04T09:00:27.0944729Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-12-04T09:00:27.0944733Z 2025-12-04T09:00:27.0944841Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0945041Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0945106Z res = mod(**inputs) 2025-12-04T09:00:27.0945361Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0945428Z outputs = self.model( 2025-12-04T09:00:27.0945675Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0945758Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0945980Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0946068Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0946313Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0946383Z return func(*args, **kwargs) 2025-12-04T09:00:27.0946650Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.0946746Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.0946987Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0947064Z return func(*args, **kwargs) 2025-12-04T09:00:27.0947312Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 260, in forward 2025-12-04T09:00:27.0947401Z attn_output = self.out_proj(attn_output) 2025-12-04T09:00:27.0947404Z 2025-12-04T09:00:27.0947505Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0947721Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0947794Z res = mod(**inputs) 2025-12-04T09:00:27.0948052Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0948146Z outputs = self.model( 2025-12-04T09:00:27.0948390Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0948463Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0948691Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0948771Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0949029Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0949107Z return func(*args, **kwargs) 2025-12-04T09:00:27.0949353Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-12-04T09:00:27.0949480Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:00:27.0949485Z 2025-12-04T09:00:27.0949587Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0949784Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0949856Z res = mod(**inputs) 2025-12-04T09:00:27.0950102Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0950187Z outputs = self.model( 2025-12-04T09:00:27.0950454Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0950529Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0950768Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0950847Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0951096Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0951175Z return func(*args, **kwargs) 2025-12-04T09:00:27.0951432Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-12-04T09:00:27.0951557Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:00:27.0951781Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:00:27.0951853Z return self.act(input) 2025-12-04T09:00:27.0951858Z 2025-12-04T09:00:27.0951975Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0952182Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0952249Z res = mod(**inputs) 2025-12-04T09:00:27.0952513Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0952585Z outputs = self.model( 2025-12-04T09:00:27.0952847Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0952922Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0953153Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0953242Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0953495Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0953567Z return func(*args, **kwargs) 2025-12-04T09:00:27.0953846Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 367, in forward 2025-12-04T09:00:27.0953930Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:00:27.0953934Z 2025-12-04T09:00:27.0954064Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0954267Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0954333Z res = mod(**inputs) 2025-12-04T09:00:27.0954606Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0954675Z outputs = self.model( 2025-12-04T09:00:27.0954936Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0955023Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0955250Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0955339Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0955586Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0955657Z return func(*args, **kwargs) 2025-12-04T09:00:27.0955918Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.0956019Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.0956275Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0956392Z return func(*args, **kwargs) 2025-12-04T09:00:27.0956652Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 157, in forward 2025-12-04T09:00:27.0956774Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:00:27.0956779Z 2025-12-04T09:00:27.0956885Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0957119Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0957190Z res = mod(**inputs) 2025-12-04T09:00:27.0957468Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0957550Z outputs = self.model( 2025-12-04T09:00:27.0957825Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0957905Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0958238Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0958334Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0958608Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0958686Z return func(*args, **kwargs) 2025-12-04T09:00:27.0958953Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.0959071Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.0959341Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0959416Z return func(*args, **kwargs) 2025-12-04T09:00:27.0959693Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 177, in forward 2025-12-04T09:00:27.0959783Z key_states = self.k_proj(current_states) 2025-12-04T09:00:27.0959787Z 2025-12-04T09:00:27.0959910Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0960146Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0960217Z res = mod(**inputs) 2025-12-04T09:00:27.0960498Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0960589Z outputs = self.model( 2025-12-04T09:00:27.0960867Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0960946Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0961193Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0961287Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0961565Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0961640Z return func(*args, **kwargs) 2025-12-04T09:00:27.0961920Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.0962025Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.0962301Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0962377Z return func(*args, **kwargs) 2025-12-04T09:00:27.0962650Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 157, in forward 2025-12-04T09:00:27.0962776Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:00:27.0962779Z 2025-12-04T09:00:27.0962890Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0963141Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0963212Z res = mod(**inputs) 2025-12-04T09:00:27.0963489Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0963570Z outputs = self.model( 2025-12-04T09:00:27.0963839Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0963920Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0964170Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0964254Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0964525Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0964600Z return func(*args, **kwargs) 2025-12-04T09:00:27.0964871Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.0964982Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.0965249Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0965323Z return func(*args, **kwargs) 2025-12-04T09:00:27.0965598Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 199, in forward 2025-12-04T09:00:27.0965746Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-12-04T09:00:27.0965750Z 2025-12-04T09:00:27.0965869Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0966083Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0966153Z res = mod(**inputs) 2025-12-04T09:00:27.0966438Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0966511Z outputs = self.model( 2025-12-04T09:00:27.0966802Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0966882Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0967124Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0967234Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0967499Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0967574Z return func(*args, **kwargs) 2025-12-04T09:00:27.0967848Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.0967951Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.0968239Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0968314Z return func(*args, **kwargs) 2025-12-04T09:00:27.0968587Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 178, in forward 2025-12-04T09:00:27.0968689Z value_states = self.v_proj(current_states) 2025-12-04T09:00:27.0968695Z 2025-12-04T09:00:27.0968804Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0969026Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0969092Z res = mod(**inputs) 2025-12-04T09:00:27.0969359Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0969440Z outputs = self.model( 2025-12-04T09:00:27.0969728Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0969805Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0970055Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0970139Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0970409Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0970485Z return func(*args, **kwargs) 2025-12-04T09:00:27.0970753Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.0970862Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.0971124Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0971199Z return func(*args, **kwargs) 2025-12-04T09:00:27.0971477Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 245, in forward 2025-12-04T09:00:27.0971581Z attn_output = torch.bmm(attn_probs, value_states) 2025-12-04T09:00:27.0971586Z 2025-12-04T09:00:27.0971702Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0971915Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0971987Z res = mod(**inputs) 2025-12-04T09:00:27.0972273Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0972345Z outputs = self.model( 2025-12-04T09:00:27.0972621Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0972698Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0972944Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0973037Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0973316Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0973391Z return func(*args, **kwargs) 2025-12-04T09:00:27.0973668Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.0973786Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.0974056Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0974131Z return func(*args, **kwargs) 2025-12-04T09:00:27.0974403Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-12-04T09:00:27.0974552Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-12-04T09:00:27.0974572Z 2025-12-04T09:00:27.0974685Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0974910Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0974980Z res = mod(**inputs) 2025-12-04T09:00:27.0975251Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0975337Z outputs = self.model( 2025-12-04T09:00:27.0975608Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0975685Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0975933Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0976037Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0976307Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0976381Z return func(*args, **kwargs) 2025-12-04T09:00:27.0976649Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.0976760Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.0977019Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0977093Z return func(*args, **kwargs) 2025-12-04T09:00:27.0977366Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 260, in forward 2025-12-04T09:00:27.0977453Z attn_output = self.out_proj(attn_output) 2025-12-04T09:00:27.0977457Z 2025-12-04T09:00:27.0977572Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0977788Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0977857Z res = mod(**inputs) 2025-12-04T09:00:27.0978133Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0978206Z outputs = self.model( 2025-12-04T09:00:27.0978477Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0978557Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0978795Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0978886Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0979148Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0979222Z return func(*args, **kwargs) 2025-12-04T09:00:27.0979500Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-12-04T09:00:27.0979629Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:00:27.0979633Z 2025-12-04T09:00:27.0979769Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0979982Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0980068Z res = mod(**inputs) 2025-12-04T09:00:27.0980348Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0980423Z outputs = self.model( 2025-12-04T09:00:27.0980701Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0980781Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0981024Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0981132Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0981403Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0981478Z return func(*args, **kwargs) 2025-12-04T09:00:27.0981753Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-12-04T09:00:27.0981884Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:00:27.0982122Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:00:27.0982199Z return self.act(input) 2025-12-04T09:00:27.0982203Z 2025-12-04T09:00:27.0982312Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0982531Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0982616Z res = mod(**inputs) 2025-12-04T09:00:27.0982898Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0982976Z outputs = self.model( 2025-12-04T09:00:27.0983230Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0983311Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0983546Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0983629Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0983900Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0983975Z return func(*args, **kwargs) 2025-12-04T09:00:27.0984253Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 367, in forward 2025-12-04T09:00:27.0984341Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:00:27.0984345Z 2025-12-04T09:00:27.0984455Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0984677Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0984746Z res = mod(**inputs) 2025-12-04T09:00:27.0985015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0985097Z outputs = self.model( 2025-12-04T09:00:27.0985366Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0985450Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0985687Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0985771Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0986043Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0986136Z return func(*args, **kwargs) 2025-12-04T09:00:27.0986406Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 369, in forward 2025-12-04T09:00:27.0986501Z hidden_states = residual + hidden_states 2025-12-04T09:00:27.0986520Z 2025-12-04T09:00:27.0986630Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0986854Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0986922Z res = mod(**inputs) 2025-12-04T09:00:27.0987190Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0987271Z outputs = self.model( 2025-12-04T09:00:27.0987568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0987653Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0987897Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0987982Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0988254Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0988330Z return func(*args, **kwargs) 2025-12-04T09:00:27.0988599Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.0988712Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.0988977Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0989078Z return func(*args, **kwargs) 2025-12-04T09:00:27.0989350Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 157, in forward 2025-12-04T09:00:27.0989471Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:00:27.0989475Z 2025-12-04T09:00:27.0989591Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0989807Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0989885Z res = mod(**inputs) 2025-12-04T09:00:27.0990156Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0990230Z outputs = self.model( 2025-12-04T09:00:27.0990511Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0990591Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0990839Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0990932Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0991207Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0991289Z return func(*args, **kwargs) 2025-12-04T09:00:27.0991580Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.0991688Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.0991966Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0992042Z return func(*args, **kwargs) 2025-12-04T09:00:27.0992318Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 177, in forward 2025-12-04T09:00:27.0992418Z key_states = self.k_proj(current_states) 2025-12-04T09:00:27.0992421Z 2025-12-04T09:00:27.0992534Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0992783Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0992863Z res = mod(**inputs) 2025-12-04T09:00:27.0993129Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0993228Z outputs = self.model( 2025-12-04T09:00:27.0993509Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0993595Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0993847Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0993935Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0994229Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0994307Z return func(*args, **kwargs) 2025-12-04T09:00:27.0994588Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.0994703Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.0994975Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0995059Z return func(*args, **kwargs) 2025-12-04T09:00:27.0995334Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 157, in forward 2025-12-04T09:00:27.0995457Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:00:27.0995461Z 2025-12-04T09:00:27.0995599Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0995820Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0995897Z res = mod(**inputs) 2025-12-04T09:00:27.0996173Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0996248Z outputs = self.model( 2025-12-04T09:00:27.0996530Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0996612Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0996868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.0996963Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.0997233Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0997317Z return func(*args, **kwargs) 2025-12-04T09:00:27.0997594Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.0997702Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.0997978Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.0998053Z return func(*args, **kwargs) 2025-12-04T09:00:27.0998408Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 199, in forward 2025-12-04T09:00:27.0998571Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-12-04T09:00:27.0998575Z 2025-12-04T09:00:27.0998688Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.0998917Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.0998991Z res = mod(**inputs) 2025-12-04T09:00:27.0999268Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.0999365Z outputs = self.model( 2025-12-04T09:00:27.0999653Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.0999740Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.0999987Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.1000092Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.1000368Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1000445Z return func(*args, **kwargs) 2025-12-04T09:00:27.1000718Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.1000854Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.1001124Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1001209Z return func(*args, **kwargs) 2025-12-04T09:00:27.1001485Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 178, in forward 2025-12-04T09:00:27.1001579Z value_states = self.v_proj(current_states) 2025-12-04T09:00:27.1001585Z 2025-12-04T09:00:27.1001705Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.1001933Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.1002011Z res = mod(**inputs) 2025-12-04T09:00:27.1002287Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.1002381Z outputs = self.model( 2025-12-04T09:00:27.1002665Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.1002747Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.1002996Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.1003090Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.1003359Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1003443Z return func(*args, **kwargs) 2025-12-04T09:00:27.1003718Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.1003825Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.1004104Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1004183Z return func(*args, **kwargs) 2025-12-04T09:00:27.1004455Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 245, in forward 2025-12-04T09:00:27.1004571Z attn_output = torch.bmm(attn_probs, value_states) 2025-12-04T09:00:27.1004575Z 2025-12-04T09:00:27.1004687Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.1004914Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.1004989Z res = mod(**inputs) 2025-12-04T09:00:27.1005266Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.1005349Z outputs = self.model( 2025-12-04T09:00:27.1005621Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.1005711Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.1005961Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.1006047Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.1006356Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1006436Z return func(*args, **kwargs) 2025-12-04T09:00:27.1006732Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.1006844Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.1007113Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1007196Z return func(*args, **kwargs) 2025-12-04T09:00:27.1007472Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-12-04T09:00:27.1007631Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-12-04T09:00:27.1007635Z 2025-12-04T09:00:27.1007756Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.1007981Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.1008060Z res = mod(**inputs) 2025-12-04T09:00:27.1008337Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.1008414Z outputs = self.model( 2025-12-04T09:00:27.1008698Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.1008779Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.1009026Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.1009139Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.1009413Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1009497Z return func(*args, **kwargs) 2025-12-04T09:00:27.1009773Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.1009880Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.1010166Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1010242Z return func(*args, **kwargs) 2025-12-04T09:00:27.1010517Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 260, in forward 2025-12-04T09:00:27.1010615Z attn_output = self.out_proj(attn_output) 2025-12-04T09:00:27.1010620Z 2025-12-04T09:00:27.1010734Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.1010961Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.1011033Z res = mod(**inputs) 2025-12-04T09:00:27.1011307Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.1011391Z outputs = self.model( 2025-12-04T09:00:27.1011665Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.1011747Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.1011970Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.1012049Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.1012300Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1012372Z return func(*args, **kwargs) 2025-12-04T09:00:27.1012624Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-12-04T09:00:27.1012770Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:00:27.1012773Z 2025-12-04T09:00:27.1012879Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.1013090Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.1013183Z res = mod(**inputs) 2025-12-04T09:00:27.1013438Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.1013515Z outputs = self.model( 2025-12-04T09:00:27.1013767Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.1013841Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.1014093Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.1014173Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.1014431Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1014501Z return func(*args, **kwargs) 2025-12-04T09:00:27.1014755Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-12-04T09:00:27.1014887Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:00:27.1015105Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:00:27.1015182Z return self.act(input) 2025-12-04T09:00:27.1015186Z 2025-12-04T09:00:27.1015288Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.1015515Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.1015587Z res = mod(**inputs) 2025-12-04T09:00:27.1015841Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.1015911Z outputs = self.model( 2025-12-04T09:00:27.1016173Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.1016249Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.1016482Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.1016562Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.1016812Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1016892Z return func(*args, **kwargs) 2025-12-04T09:00:27.1017159Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 367, in forward 2025-12-04T09:00:27.1017248Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:00:27.1017252Z 2025-12-04T09:00:27.1017356Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.1017558Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.1017629Z res = mod(**inputs) 2025-12-04T09:00:27.1017883Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.1017951Z outputs = self.model( 2025-12-04T09:00:27.1018213Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.1018286Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.1018518Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.1018598Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.1018863Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1018942Z return func(*args, **kwargs) 2025-12-04T09:00:27.1019194Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.1019309Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.1019563Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1019633Z return func(*args, **kwargs) 2025-12-04T09:00:27.1019892Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 157, in forward 2025-12-04T09:00:27.1020007Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:00:27.1020010Z 2025-12-04T09:00:27.1020127Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.1020340Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.1020407Z res = mod(**inputs) 2025-12-04T09:00:27.1020667Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.1020927Z outputs = self.model( 2025-12-04T09:00:27.1021188Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.1021272Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.1021499Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.1021580Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.1021883Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1021955Z return func(*args, **kwargs) 2025-12-04T09:00:27.1022221Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.1022320Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.1022571Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1022651Z return func(*args, **kwargs) 2025-12-04T09:00:27.1022964Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 177, in forward 2025-12-04T09:00:27.1023051Z key_states = self.k_proj(current_states) 2025-12-04T09:00:27.1023063Z 2025-12-04T09:00:27.1023173Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.1023391Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.1023471Z res = mod(**inputs) 2025-12-04T09:00:27.1023745Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.1023823Z outputs = self.model( 2025-12-04T09:00:27.1024113Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.1024195Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.1024449Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.1024535Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.1024804Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1024889Z return func(*args, **kwargs) 2025-12-04T09:00:27.1025163Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.1025262Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.1025550Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1025623Z return func(*args, **kwargs) 2025-12-04T09:00:27.1025886Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 157, in forward 2025-12-04T09:00:27.1026023Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:00:27.1026027Z 2025-12-04T09:00:27.1026129Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.1026340Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.1026404Z res = mod(**inputs) 2025-12-04T09:00:27.1026665Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.1026757Z outputs = self.model( 2025-12-04T09:00:27.1027015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.1027097Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.1027324Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.1027408Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.1027680Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1027754Z return func(*args, **kwargs) 2025-12-04T09:00:27.1028070Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.1028167Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.1028434Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1028513Z return func(*args, **kwargs) 2025-12-04T09:00:27.1028773Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 199, in forward 2025-12-04T09:00:27.1028913Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-12-04T09:00:27.1028923Z 2025-12-04T09:00:27.1029030Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.1029232Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.1029305Z res = mod(**inputs) 2025-12-04T09:00:27.1029570Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.1029639Z outputs = self.model( 2025-12-04T09:00:27.1029907Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.1029983Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.1030226Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.1030306Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.1030556Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1030636Z return func(*args, **kwargs) 2025-12-04T09:00:27.1030890Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.1030989Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.1031250Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1031322Z return func(*args, **kwargs) 2025-12-04T09:00:27.1031584Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 178, in forward 2025-12-04T09:00:27.1031672Z value_states = self.v_proj(current_states) 2025-12-04T09:00:27.1031676Z 2025-12-04T09:00:27.1031800Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.1032011Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.1032097Z res = mod(**inputs) 2025-12-04T09:00:27.1032381Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.1032455Z outputs = self.model( 2025-12-04T09:00:27.1032724Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.1032809Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.1033051Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.1033154Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.1033431Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1033505Z return func(*args, **kwargs) 2025-12-04T09:00:27.1033778Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.1033885Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.1034144Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1034225Z return func(*args, **kwargs) 2025-12-04T09:00:27.1034493Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 245, in forward 2025-12-04T09:00:27.1034622Z attn_output = torch.bmm(attn_probs, value_states) 2025-12-04T09:00:27.1034633Z 2025-12-04T09:00:27.1034745Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.1034963Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.1035040Z res = mod(**inputs) 2025-12-04T09:00:27.1035311Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.1035386Z outputs = self.model( 2025-12-04T09:00:27.1035670Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.1035751Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.1036003Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.1036089Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.1036362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1036450Z return func(*args, **kwargs) 2025-12-04T09:00:27.1036730Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.1036840Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.1037121Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1037198Z return func(*args, **kwargs) 2025-12-04T09:00:27.1037484Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-12-04T09:00:27.1037627Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-12-04T09:00:27.1037631Z 2025-12-04T09:00:27.1037743Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.1037976Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.1038050Z res = mod(**inputs) 2025-12-04T09:00:27.1038425Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.1038508Z outputs = self.model( 2025-12-04T09:00:27.1038791Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.1038902Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.1039153Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.1039241Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.1039539Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1039616Z return func(*args, **kwargs) 2025-12-04T09:00:27.1039916Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.1040021Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.1040289Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1040376Z return func(*args, **kwargs) 2025-12-04T09:00:27.1040650Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 260, in forward 2025-12-04T09:00:27.1040744Z attn_output = self.out_proj(attn_output) 2025-12-04T09:00:27.1040758Z 2025-12-04T09:00:27.1040872Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.1041092Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.1041167Z res = mod(**inputs) 2025-12-04T09:00:27.1041452Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.1041527Z outputs = self.model( 2025-12-04T09:00:27.1041806Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.1041885Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.1042137Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.1042222Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.1042484Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1042567Z return func(*args, **kwargs) 2025-12-04T09:00:27.1042838Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-12-04T09:00:27.1042968Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:00:27.1042980Z 2025-12-04T09:00:27.1043091Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.1043310Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.1043390Z res = mod(**inputs) 2025-12-04T09:00:27.1043661Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.1043735Z outputs = self.model( 2025-12-04T09:00:27.1044015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.1044093Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.1044341Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.1044428Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.1044692Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1044776Z return func(*args, **kwargs) 2025-12-04T09:00:27.1045065Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-12-04T09:00:27.1045194Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:00:27.1045437Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:00:27.1045530Z return self.act(input) 2025-12-04T09:00:27.1045533Z 2025-12-04T09:00:27.1045653Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.1045867Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.1045935Z res = mod(**inputs) 2025-12-04T09:00:27.1046209Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.1046284Z outputs = self.model( 2025-12-04T09:00:27.1046569Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.1046664Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.1046891Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.1046976Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.1047224Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1047295Z return func(*args, **kwargs) 2025-12-04T09:00:27.1047557Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 367, in forward 2025-12-04T09:00:27.1047640Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:00:27.1047659Z 2025-12-04T09:00:27.1047770Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.1047976Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.1048043Z res = mod(**inputs) 2025-12-04T09:00:27.1048304Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.1048373Z outputs = self.model( 2025-12-04T09:00:27.1048625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.1048709Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.1048936Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.1049020Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.1049269Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1049341Z return func(*args, **kwargs) 2025-12-04T09:00:27.1049605Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 369, in forward 2025-12-04T09:00:27.1049690Z hidden_states = residual + hidden_states 2025-12-04T09:00:27.1049693Z 2025-12-04T09:00:27.1049803Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.1050002Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.1050070Z res = mod(**inputs) 2025-12-04T09:00:27.1050328Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.1050396Z outputs = self.model( 2025-12-04T09:00:27.1050651Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.1050733Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.1050962Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.1051049Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.1051317Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1051388Z return func(*args, **kwargs) 2025-12-04T09:00:27.1051651Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.1051768Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.1052015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1052093Z return func(*args, **kwargs) 2025-12-04T09:00:27.1052345Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 157, in forward 2025-12-04T09:00:27.1052469Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:00:27.1052491Z 2025-12-04T09:00:27.1052596Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.1052803Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.1052876Z res = mod(**inputs) 2025-12-04T09:00:27.1053134Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.1053210Z outputs = self.model( 2025-12-04T09:00:27.1053467Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.1053541Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.1053778Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.1053873Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.1054124Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1054202Z return func(*args, **kwargs) 2025-12-04T09:00:27.1054459Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.1054567Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.1054816Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1054889Z return func(*args, **kwargs) 2025-12-04T09:00:27.1055149Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 177, in forward 2025-12-04T09:00:27.1055232Z key_states = self.k_proj(current_states) 2025-12-04T09:00:27.1055235Z 2025-12-04T09:00:27.1055346Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.1055551Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.1055617Z res = mod(**inputs) 2025-12-04T09:00:27.1055880Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.1055949Z outputs = self.model( 2025-12-04T09:00:27.1056200Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.1056284Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.1056511Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.1056597Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.1056845Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1056917Z return func(*args, **kwargs) 2025-12-04T09:00:27.1057178Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.1057275Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.1057542Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1057618Z return func(*args, **kwargs) 2025-12-04T09:00:27.1057869Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 157, in forward 2025-12-04T09:00:27.1058007Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:00:27.1058011Z 2025-12-04T09:00:27.1058115Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.1058316Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.1058391Z res = mod(**inputs) 2025-12-04T09:00:27.1058685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.1058762Z outputs = self.model( 2025-12-04T09:00:27.1059021Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.1059095Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.1059328Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.1059410Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.1059660Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1059736Z return func(*args, **kwargs) 2025-12-04T09:00:27.1059987Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.1060119Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.1060368Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1060438Z return func(*args, **kwargs) 2025-12-04T09:00:27.1060700Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 199, in forward 2025-12-04T09:00:27.1060841Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-12-04T09:00:27.1060846Z 2025-12-04T09:00:27.1060955Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.1061157Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.1061222Z res = mod(**inputs) 2025-12-04T09:00:27.1061481Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.1061550Z outputs = self.model( 2025-12-04T09:00:27.1061802Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.1061882Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.1062110Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.1062195Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.1062441Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1062511Z return func(*args, **kwargs) 2025-12-04T09:00:27.1062768Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.1062865Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.1063110Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1063188Z return func(*args, **kwargs) 2025-12-04T09:00:27.1063442Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 178, in forward 2025-12-04T09:00:27.1063552Z value_states = self.v_proj(current_states) 2025-12-04T09:00:27.1063556Z 2025-12-04T09:00:27.1063661Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.1063876Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.1063961Z res = mod(**inputs) 2025-12-04T09:00:27.1064214Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.1064286Z outputs = self.model( 2025-12-04T09:00:27.1064537Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.1064609Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.1064858Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.1064935Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.1065181Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1065257Z return func(*args, **kwargs) 2025-12-04T09:00:27.1065507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.1065613Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.1065856Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1065926Z return func(*args, **kwargs) 2025-12-04T09:00:27.1066185Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 245, in forward 2025-12-04T09:00:27.1066302Z attn_output = torch.bmm(attn_probs, value_states) 2025-12-04T09:00:27.1066305Z 2025-12-04T09:00:27.1066416Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.1066619Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.1066684Z res = mod(**inputs) 2025-12-04T09:00:27.1066943Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.1067013Z outputs = self.model( 2025-12-04T09:00:27.1067266Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.1067347Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.1067572Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.1067660Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.1067908Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1067978Z return func(*args, **kwargs) 2025-12-04T09:00:27.1068237Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.1068334Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.1068582Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1068659Z return func(*args, **kwargs) 2025-12-04T09:00:27.1068911Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-12-04T09:00:27.1069050Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-12-04T09:00:27.1069054Z 2025-12-04T09:00:27.1069159Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.1069360Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.1069433Z res = mod(**inputs) 2025-12-04T09:00:27.1069701Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.1069780Z outputs = self.model( 2025-12-04T09:00:27.1070041Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.1070130Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.1070359Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.1070436Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.1070679Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1070756Z return func(*args, **kwargs) 2025-12-04T09:00:27.1071019Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.1071125Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.1071372Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1071441Z return func(*args, **kwargs) 2025-12-04T09:00:27.1071708Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 260, in forward 2025-12-04T09:00:27.1071801Z attn_output = self.out_proj(attn_output) 2025-12-04T09:00:27.1071804Z 2025-12-04T09:00:27.1071911Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.1072107Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.1072186Z res = mod(**inputs) 2025-12-04T09:00:27.1072447Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.1072516Z outputs = self.model( 2025-12-04T09:00:27.1072769Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.1072850Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.1073074Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.1073163Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.1073411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1073482Z return func(*args, **kwargs) 2025-12-04T09:00:27.1073743Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-12-04T09:00:27.1073866Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:00:27.1073872Z 2025-12-04T09:00:27.1073976Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.1074187Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.1074253Z res = mod(**inputs) 2025-12-04T09:00:27.1074511Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.1074581Z outputs = self.model( 2025-12-04T09:00:27.1074833Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.1074916Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.1075143Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.1075230Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.1075488Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1075562Z return func(*args, **kwargs) 2025-12-04T09:00:27.1075849Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-12-04T09:00:27.1075978Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:00:27.1076211Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:00:27.1076312Z return self.act(input) 2025-12-04T09:00:27.1076315Z 2025-12-04T09:00:27.1076427Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.1076649Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.1076718Z res = mod(**inputs) 2025-12-04T09:00:27.1076990Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.1077088Z outputs = self.model( 2025-12-04T09:00:27.1077358Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.1077436Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.1077685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.1077771Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.1078036Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1078109Z return func(*args, **kwargs) 2025-12-04T09:00:27.1078460Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 367, in forward 2025-12-04T09:00:27.1078587Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:00:27.1078592Z 2025-12-04T09:00:27.1078709Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.1078951Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.1079025Z res = mod(**inputs) 2025-12-04T09:00:27.1079311Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.1079393Z outputs = self.model( 2025-12-04T09:00:27.1079677Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.1079757Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.1080023Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.1080105Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.1080381Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1080461Z return func(*args, **kwargs) 2025-12-04T09:00:27.1080744Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.1080862Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.1081141Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1081219Z return func(*args, **kwargs) 2025-12-04T09:00:27.1081507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 157, in forward 2025-12-04T09:00:27.1081631Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:00:27.1081635Z 2025-12-04T09:00:27.1081754Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.1081979Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.1082050Z res = mod(**inputs) 2025-12-04T09:00:27.1082348Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.1082444Z outputs = self.model( 2025-12-04T09:00:27.1082727Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.1082808Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.1083073Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.1083169Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.1083443Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1083520Z return func(*args, **kwargs) 2025-12-04T09:00:27.1083804Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.1083939Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.1084218Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1084296Z return func(*args, **kwargs) 2025-12-04T09:00:27.1084575Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 177, in forward 2025-12-04T09:00:27.1084673Z key_states = self.k_proj(current_states) 2025-12-04T09:00:27.1084677Z 2025-12-04T09:00:27.1084790Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.1085016Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.1085088Z res = mod(**inputs) 2025-12-04T09:00:27.1085363Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.1085464Z outputs = self.model( 2025-12-04T09:00:27.1085757Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.1085840Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.1086091Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.1086176Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.1086467Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1086543Z return func(*args, **kwargs) 2025-12-04T09:00:27.1086828Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.1086945Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.1087233Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1087309Z return func(*args, **kwargs) 2025-12-04T09:00:27.1087604Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 157, in forward 2025-12-04T09:00:27.1087726Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:00:27.1087730Z 2025-12-04T09:00:27.1087850Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.1088073Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.1088144Z res = mod(**inputs) 2025-12-04T09:00:27.1088428Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.1088502Z outputs = self.model( 2025-12-04T09:00:27.1088795Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.1088878Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.1089133Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.1089244Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.1089524Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1089618Z return func(*args, **kwargs) 2025-12-04T09:00:27.1089908Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.1090014Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.1090304Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1090380Z return func(*args, **kwargs) 2025-12-04T09:00:27.1090684Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 199, in forward 2025-12-04T09:00:27.1090844Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-12-04T09:00:27.1090848Z 2025-12-04T09:00:27.1090964Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.1091192Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.1091263Z res = mod(**inputs) 2025-12-04T09:00:27.1091544Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.1091625Z outputs = self.model( 2025-12-04T09:00:27.1091914Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.1091994Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.1092269Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.1092360Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.1092648Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1092723Z return func(*args, **kwargs) 2025-12-04T09:00:27.1093012Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.1093127Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.1093403Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1093477Z return func(*args, **kwargs) 2025-12-04T09:00:27.1093824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 178, in forward 2025-12-04T09:00:27.1093923Z value_states = self.v_proj(current_states) 2025-12-04T09:00:27.1093927Z 2025-12-04T09:00:27.1094047Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.1094267Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.1094346Z res = mod(**inputs) 2025-12-04T09:00:27.1094609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.1094677Z outputs = self.model( 2025-12-04T09:00:27.1094987Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.1095066Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.1095302Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.1095392Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.1095657Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1095730Z return func(*args, **kwargs) 2025-12-04T09:00:27.1096019Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.1096122Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.1096392Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1096482Z return func(*args, **kwargs) 2025-12-04T09:00:27.1096748Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 245, in forward 2025-12-04T09:00:27.1096868Z attn_output = torch.bmm(attn_probs, value_states) 2025-12-04T09:00:27.1096871Z 2025-12-04T09:00:27.1096974Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.1097184Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.1097263Z res = mod(**inputs) 2025-12-04T09:00:27.1097512Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.1097589Z outputs = self.model( 2025-12-04T09:00:27.1097839Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.1097916Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.1098151Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.1098230Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.1098485Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1098555Z return func(*args, **kwargs) 2025-12-04T09:00:27.1098829Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.1098936Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.1099185Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1099255Z return func(*args, **kwargs) 2025-12-04T09:00:27.1099525Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-12-04T09:00:27.1099657Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-12-04T09:00:27.1099660Z 2025-12-04T09:00:27.1099770Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.1099987Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.1100051Z res = mod(**inputs) 2025-12-04T09:00:27.1100308Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.1100376Z outputs = self.model( 2025-12-04T09:00:27.1100634Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.1100708Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.1100927Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.1101015Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.1101253Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1101322Z return func(*args, **kwargs) 2025-12-04T09:00:27.1101572Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.1101666Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.1101916Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1101983Z return func(*args, **kwargs) 2025-12-04T09:00:27.1102243Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 260, in forward 2025-12-04T09:00:27.1102332Z attn_output = self.out_proj(attn_output) 2025-12-04T09:00:27.1102336Z 2025-12-04T09:00:27.1102452Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.1102656Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.1102720Z res = mod(**inputs) 2025-12-04T09:00:27.1102963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.1103036Z outputs = self.model( 2025-12-04T09:00:27.1103283Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.1103371Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.1103607Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.1103685Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.1103933Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1104004Z return func(*args, **kwargs) 2025-12-04T09:00:27.1104250Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-12-04T09:00:27.1104374Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:00:27.1104378Z 2025-12-04T09:00:27.1104479Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.1104682Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.1104771Z res = mod(**inputs) 2025-12-04T09:00:27.1105021Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.1105097Z outputs = self.model( 2025-12-04T09:00:27.1105346Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.1105418Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.1105650Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.1105728Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.1105978Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1106047Z return func(*args, **kwargs) 2025-12-04T09:00:27.1106294Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-12-04T09:00:27.1106420Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:00:27.1106635Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:00:27.1106704Z return self.act(input) 2025-12-04T09:00:27.1106707Z 2025-12-04T09:00:27.1106817Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.1107014Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.1107085Z res = mod(**inputs) 2025-12-04T09:00:27.1107331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.1107396Z outputs = self.model( 2025-12-04T09:00:27.1107647Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.1107721Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.1107940Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.1108042Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.1108283Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1108358Z return func(*args, **kwargs) 2025-12-04T09:00:27.1108639Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 367, in forward 2025-12-04T09:00:27.1108720Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:00:27.1108724Z 2025-12-04T09:00:27.1108832Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.1109029Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.1109104Z res = mod(**inputs) 2025-12-04T09:00:27.1109366Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.1109434Z outputs = self.model( 2025-12-04T09:00:27.1109688Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.1109760Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.1109979Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.1110065Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.1110309Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1110384Z return func(*args, **kwargs) 2025-12-04T09:00:27.1110630Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 369, in forward 2025-12-04T09:00:27.1110728Z hidden_states = residual + hidden_states 2025-12-04T09:00:27.1110731Z 2025-12-04T09:00:27.1110843Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.1111041Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.1111111Z res = mod(**inputs) 2025-12-04T09:00:27.1111358Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.1111427Z outputs = self.model( 2025-12-04T09:00:27.1111687Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.1111760Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.1111985Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.1112069Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.1112329Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1112405Z return func(*args, **kwargs) 2025-12-04T09:00:27.1112659Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.1112760Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.1113015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1113088Z return func(*args, **kwargs) 2025-12-04T09:00:27.1113340Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 157, in forward 2025-12-04T09:00:27.1113460Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:00:27.1113463Z 2025-12-04T09:00:27.1113567Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.1113781Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.1113848Z res = mod(**inputs) 2025-12-04T09:00:27.1114114Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.1114191Z outputs = self.model( 2025-12-04T09:00:27.1114445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.1114543Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.1114768Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.1114848Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.1115103Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1115175Z return func(*args, **kwargs) 2025-12-04T09:00:27.1115444Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.1115554Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.1115812Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1115895Z return func(*args, **kwargs) 2025-12-04T09:00:27.1116165Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 177, in forward 2025-12-04T09:00:27.1116253Z key_states = self.k_proj(current_states) 2025-12-04T09:00:27.1116257Z 2025-12-04T09:00:27.1116375Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.1116592Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.1116669Z res = mod(**inputs) 2025-12-04T09:00:27.1116957Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.1117029Z outputs = self.model( 2025-12-04T09:00:27.1117308Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.1117387Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.1117626Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.1117718Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.1117982Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1118064Z return func(*args, **kwargs) 2025-12-04T09:00:27.1118396Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.1118509Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.1118783Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1118857Z return func(*args, **kwargs) 2025-12-04T09:00:27.1119126Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 157, in forward 2025-12-04T09:00:27.1119254Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:00:27.1119260Z 2025-12-04T09:00:27.1119370Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.1119592Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.1119662Z res = mod(**inputs) 2025-12-04T09:00:27.1119927Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.1120009Z outputs = self.model( 2025-12-04T09:00:27.1120279Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.1120365Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.1120628Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.1120858Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.1121144Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1121262Z return func(*args, **kwargs) 2025-12-04T09:00:27.1121541Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.1121654Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.1121917Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1122000Z return func(*args, **kwargs) 2025-12-04T09:00:27.1122295Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 199, in forward 2025-12-04T09:00:27.1122438Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-12-04T09:00:27.1122443Z 2025-12-04T09:00:27.1122553Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.1122756Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.1122831Z res = mod(**inputs) 2025-12-04T09:00:27.1123086Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.1123156Z outputs = self.model( 2025-12-04T09:00:27.1123421Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.1123521Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.1123759Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.1123846Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.1124096Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1124174Z return func(*args, **kwargs) 2025-12-04T09:00:27.1124427Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.1124528Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.1124784Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1124853Z return func(*args, **kwargs) 2025-12-04T09:00:27.1125108Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 178, in forward 2025-12-04T09:00:27.1125208Z value_states = self.v_proj(current_states) 2025-12-04T09:00:27.1125213Z 2025-12-04T09:00:27.1125323Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.1125544Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.1125613Z res = mod(**inputs) 2025-12-04T09:00:27.1125878Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.1125961Z outputs = self.model( 2025-12-04T09:00:27.1126226Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.1126308Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.1126550Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.1126636Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.1126904Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1126977Z return func(*args, **kwargs) 2025-12-04T09:00:27.1127266Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.1127378Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.1127643Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1127742Z return func(*args, **kwargs) 2025-12-04T09:00:27.1128007Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 245, in forward 2025-12-04T09:00:27.1128112Z attn_output = torch.bmm(attn_probs, value_states) 2025-12-04T09:00:27.1128116Z 2025-12-04T09:00:27.1128230Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.1128463Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.1128542Z res = mod(**inputs) 2025-12-04T09:00:27.1128815Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.1128887Z outputs = self.model( 2025-12-04T09:00:27.1129159Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.1129239Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.1129478Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.1129570Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.1129836Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1129933Z return func(*args, **kwargs) 2025-12-04T09:00:27.1130202Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.1130308Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.1130579Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1130652Z return func(*args, **kwargs) 2025-12-04T09:00:27.1130920Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-12-04T09:00:27.1131068Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-12-04T09:00:27.1131071Z 2025-12-04T09:00:27.1131180Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.1131402Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.1131473Z res = mod(**inputs) 2025-12-04T09:00:27.1131742Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.1131822Z outputs = self.model( 2025-12-04T09:00:27.1132093Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.1132179Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.1132417Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.1132503Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.1132772Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1132847Z return func(*args, **kwargs) 2025-12-04T09:00:27.1133115Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.1133228Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.1133494Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1133576Z return func(*args, **kwargs) 2025-12-04T09:00:27.1133866Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 260, in forward 2025-12-04T09:00:27.1133954Z attn_output = self.out_proj(attn_output) 2025-12-04T09:00:27.1133977Z 2025-12-04T09:00:27.1134102Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.1134318Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.1134394Z res = mod(**inputs) 2025-12-04T09:00:27.1134663Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.1134737Z outputs = self.model( 2025-12-04T09:00:27.1135031Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.1135111Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.1135350Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.1135440Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.1135703Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1135785Z return func(*args, **kwargs) 2025-12-04T09:00:27.1136053Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-12-04T09:00:27.1136181Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:00:27.1136185Z 2025-12-04T09:00:27.1136302Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.1136537Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.1136606Z res = mod(**inputs) 2025-12-04T09:00:27.1136883Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.1136955Z outputs = self.model( 2025-12-04T09:00:27.1137230Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.1137310Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.1137550Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.1137641Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.1137904Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1137987Z return func(*args, **kwargs) 2025-12-04T09:00:27.1138255Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-12-04T09:00:27.1138382Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:00:27.1138623Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:00:27.1138697Z return self.act(input) 2025-12-04T09:00:27.1138700Z 2025-12-04T09:00:27.1138811Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.1139043Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.1139111Z res = mod(**inputs) 2025-12-04T09:00:27.1139386Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.1139458Z outputs = self.model( 2025-12-04T09:00:27.1139724Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.1139814Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.1140053Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.1140153Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.1140425Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1140519Z return func(*args, **kwargs) 2025-12-04T09:00:27.1140803Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 367, in forward 2025-12-04T09:00:27.1140885Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:00:27.1140888Z 2025-12-04T09:00:27.1140993Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.1141200Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.1141267Z res = mod(**inputs) 2025-12-04T09:00:27.1141543Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.1141614Z outputs = self.model( 2025-12-04T09:00:27.1141871Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.1141951Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.1142178Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.1142257Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.1142513Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1142583Z return func(*args, **kwargs) 2025-12-04T09:00:27.1142843Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.1142960Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.1143212Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1143291Z return func(*args, **kwargs) 2025-12-04T09:00:27.1143545Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 157, in forward 2025-12-04T09:00:27.1143668Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:00:27.1143671Z 2025-12-04T09:00:27.1143774Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.1143977Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.1144048Z res = mod(**inputs) 2025-12-04T09:00:27.1144301Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.1144370Z outputs = self.model( 2025-12-04T09:00:27.1144633Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.1144708Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.1144941Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.1145021Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.1145270Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1145348Z return func(*args, **kwargs) 2025-12-04T09:00:27.1145599Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.1145699Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.1145954Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1146025Z return func(*args, **kwargs) 2025-12-04T09:00:27.1146308Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 177, in forward 2025-12-04T09:00:27.1146392Z key_states = self.k_proj(current_states) 2025-12-04T09:00:27.1146396Z 2025-12-04T09:00:27.1146500Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.1146727Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.1146792Z res = mod(**inputs) 2025-12-04T09:00:27.1147056Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.1147125Z outputs = self.model( 2025-12-04T09:00:27.1147386Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.1147470Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.1147717Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.1147799Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.1148056Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1148124Z return func(*args, **kwargs) 2025-12-04T09:00:27.1148383Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.1148479Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.1148723Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1148801Z return func(*args, **kwargs) 2025-12-04T09:00:27.1149069Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 157, in forward 2025-12-04T09:00:27.1149182Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:00:27.1149193Z 2025-12-04T09:00:27.1149298Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.1149499Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.1149572Z res = mod(**inputs) 2025-12-04T09:00:27.1149828Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.1149897Z outputs = self.model( 2025-12-04T09:00:27.1150174Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.1150251Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.1150498Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.1150596Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.1150842Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1150921Z return func(*args, **kwargs) 2025-12-04T09:00:27.1151173Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.1151275Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.1151530Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1151599Z return func(*args, **kwargs) 2025-12-04T09:00:27.1151859Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 199, in forward 2025-12-04T09:00:27.1151999Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-12-04T09:00:27.1152003Z 2025-12-04T09:00:27.1152107Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.1152316Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.1152398Z res = mod(**inputs) 2025-12-04T09:00:27.1152663Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.1152732Z outputs = self.model( 2025-12-04T09:00:27.1153004Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.1153084Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.1153312Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.1153392Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.1153652Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1153737Z return func(*args, **kwargs) 2025-12-04T09:00:27.1154000Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.1154099Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.1154346Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1154427Z return func(*args, **kwargs) 2025-12-04T09:00:27.1154681Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 178, in forward 2025-12-04T09:00:27.1154770Z value_states = self.v_proj(current_states) 2025-12-04T09:00:27.1154781Z 2025-12-04T09:00:27.1154883Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.1155087Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.1155184Z res = mod(**inputs) 2025-12-04T09:00:27.1155439Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.1155509Z outputs = self.model( 2025-12-04T09:00:27.1155771Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.1155843Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.1156078Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.1156157Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.1156401Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1156477Z return func(*args, **kwargs) 2025-12-04T09:00:27.1156729Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.1156827Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.1157079Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1157149Z return func(*args, **kwargs) 2025-12-04T09:00:27.1157466Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 245, in forward 2025-12-04T09:00:27.1157593Z attn_output = torch.bmm(attn_probs, value_states) 2025-12-04T09:00:27.1157598Z 2025-12-04T09:00:27.1157701Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.1157921Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.1157992Z res = mod(**inputs) 2025-12-04T09:00:27.1158413Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.1158500Z outputs = self.model( 2025-12-04T09:00:27.1158777Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.1158898Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.1159146Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.1159234Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.1159572Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1159645Z return func(*args, **kwargs) 2025-12-04T09:00:27.1159920Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.1160021Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.1160302Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1160385Z return func(*args, **kwargs) 2025-12-04T09:00:27.1160656Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-12-04T09:00:27.1160796Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-12-04T09:00:27.1160809Z 2025-12-04T09:00:27.1160920Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.1161137Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.1161214Z res = mod(**inputs) 2025-12-04T09:00:27.1161486Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.1161558Z outputs = self.model( 2025-12-04T09:00:27.1161835Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.1161932Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.1162180Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.1162265Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.1162527Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1162612Z return func(*args, **kwargs) 2025-12-04T09:00:27.1162880Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.1162981Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.1163249Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1163325Z return func(*args, **kwargs) 2025-12-04T09:00:27.1163605Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 260, in forward 2025-12-04T09:00:27.1163693Z attn_output = self.out_proj(attn_output) 2025-12-04T09:00:27.1163696Z 2025-12-04T09:00:27.1163806Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.1164027Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.1164095Z res = mod(**inputs) 2025-12-04T09:00:27.1164373Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.1164445Z outputs = self.model( 2025-12-04T09:00:27.1164711Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.1164797Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.1165032Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.1165120Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.1165388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1165478Z return func(*args, **kwargs) 2025-12-04T09:00:27.1165755Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-12-04T09:00:27.1165899Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:00:27.1165903Z 2025-12-04T09:00:27.1166013Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.1166242Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.1166312Z res = mod(**inputs) 2025-12-04T09:00:27.1166588Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.1166664Z outputs = self.model( 2025-12-04T09:00:27.1166947Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.1167035Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.1167277Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.1167360Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.1167633Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1167708Z return func(*args, **kwargs) 2025-12-04T09:00:27.1167985Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-12-04T09:00:27.1168114Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:00:27.1168367Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:00:27.1168452Z return self.act(input) 2025-12-04T09:00:27.1168456Z 2025-12-04T09:00:27.1168564Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.1168827Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.1168938Z res = mod(**inputs) 2025-12-04T09:00:27.1169229Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.1169310Z outputs = self.model( 2025-12-04T09:00:27.1169582Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.1169711Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.1169972Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.1170057Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.1170338Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1170466Z return func(*args, **kwargs) 2025-12-04T09:00:27.1170796Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 367, in forward 2025-12-04T09:00:27.1170896Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:00:27.1170901Z 2025-12-04T09:00:27.1171011Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.1171227Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.1171303Z res = mod(**inputs) 2025-12-04T09:00:27.1171572Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.1171653Z outputs = self.model( 2025-12-04T09:00:27.1171922Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.1171999Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.1172263Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.1172349Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.1172610Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1172709Z return func(*args, **kwargs) 2025-12-04T09:00:27.1172975Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 369, in forward 2025-12-04T09:00:27.1173068Z hidden_states = residual + hidden_states 2025-12-04T09:00:27.1173071Z 2025-12-04T09:00:27.1173178Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.1173394Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.1173486Z res = mod(**inputs) 2025-12-04T09:00:27.1173757Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.1173840Z outputs = self.model( 2025-12-04T09:00:27.1174110Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.1174189Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.1174434Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.1174518Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.1174782Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1174864Z return func(*args, **kwargs) 2025-12-04T09:00:27.1175156Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.1175268Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.1175538Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1175612Z return func(*args, **kwargs) 2025-12-04T09:00:27.1175888Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 157, in forward 2025-12-04T09:00:27.1176011Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:00:27.1176026Z 2025-12-04T09:00:27.1176134Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.1176332Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.1176396Z res = mod(**inputs) 2025-12-04T09:00:27.1176653Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.1176722Z outputs = self.model( 2025-12-04T09:00:27.1176972Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.1177053Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.1177272Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.1177357Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.1177602Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1177670Z return func(*args, **kwargs) 2025-12-04T09:00:27.1177922Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.1178020Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.1178265Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1178341Z return func(*args, **kwargs) 2025-12-04T09:00:27.1178605Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 177, in forward 2025-12-04T09:00:27.1178692Z key_states = self.k_proj(current_states) 2025-12-04T09:00:27.1178696Z 2025-12-04T09:00:27.1178813Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.1179011Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.1179081Z res = mod(**inputs) 2025-12-04T09:00:27.1179325Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.1179400Z outputs = self.model( 2025-12-04T09:00:27.1179647Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.1179745Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.1179976Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.1180053Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.1180294Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1180373Z return func(*args, **kwargs) 2025-12-04T09:00:27.1180620Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.1180719Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.1180959Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1181047Z return func(*args, **kwargs) 2025-12-04T09:00:27.1181301Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 157, in forward 2025-12-04T09:00:27.1181412Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:00:27.1181415Z 2025-12-04T09:00:27.1181524Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.1181720Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.1181785Z res = mod(**inputs) 2025-12-04T09:00:27.1182041Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.1182109Z outputs = self.model( 2025-12-04T09:00:27.1182353Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.1182434Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.1182655Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.1182741Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.1182982Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1183052Z return func(*args, **kwargs) 2025-12-04T09:00:27.1183300Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.1183396Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.1183638Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1183713Z return func(*args, **kwargs) 2025-12-04T09:00:27.1183958Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 199, in forward 2025-12-04T09:00:27.1184102Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-12-04T09:00:27.1184105Z 2025-12-04T09:00:27.1184208Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.1184419Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.1184492Z res = mod(**inputs) 2025-12-04T09:00:27.1184741Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.1184831Z outputs = self.model( 2025-12-04T09:00:27.1185086Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.1185159Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.1185388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.1185466Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.1185728Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1185803Z return func(*args, **kwargs) 2025-12-04T09:00:27.1186052Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.1186153Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.1186394Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1186465Z return func(*args, **kwargs) 2025-12-04T09:00:27.1186718Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 178, in forward 2025-12-04T09:00:27.1186804Z value_states = self.v_proj(current_states) 2025-12-04T09:00:27.1186808Z 2025-12-04T09:00:27.1186916Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.1187129Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.1187194Z res = mod(**inputs) 2025-12-04T09:00:27.1187448Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.1187514Z outputs = self.model( 2025-12-04T09:00:27.1187759Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.1187838Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.1188059Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.1188142Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.1188384Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1188453Z return func(*args, **kwargs) 2025-12-04T09:00:27.1188706Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.1188803Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.1189051Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1189125Z return func(*args, **kwargs) 2025-12-04T09:00:27.1189372Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 245, in forward 2025-12-04T09:00:27.1189480Z attn_output = torch.bmm(attn_probs, value_states) 2025-12-04T09:00:27.1189483Z 2025-12-04T09:00:27.1189583Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.1189781Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.1189851Z res = mod(**inputs) 2025-12-04T09:00:27.1190100Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.1190175Z outputs = self.model( 2025-12-04T09:00:27.1190438Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.1190513Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.1190743Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.1190838Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.1191076Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1191152Z return func(*args, **kwargs) 2025-12-04T09:00:27.1191394Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.1191497Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.1191752Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1191823Z return func(*args, **kwargs) 2025-12-04T09:00:27.1192079Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-12-04T09:00:27.1192206Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-12-04T09:00:27.1192212Z 2025-12-04T09:00:27.1192319Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.1192518Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.1192581Z res = mod(**inputs) 2025-12-04T09:00:27.1192833Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.1192900Z outputs = self.model( 2025-12-04T09:00:27.1193169Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.1193252Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.1193474Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.1193560Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.1193802Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1193872Z return func(*args, **kwargs) 2025-12-04T09:00:27.1194124Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.1194219Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.1194464Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1194545Z return func(*args, **kwargs) 2025-12-04T09:00:27.1194796Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 260, in forward 2025-12-04T09:00:27.1194896Z attn_output = self.out_proj(attn_output) 2025-12-04T09:00:27.1194901Z 2025-12-04T09:00:27.1195003Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.1195204Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.1195279Z res = mod(**inputs) 2025-12-04T09:00:27.1195529Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.1195604Z outputs = self.model( 2025-12-04T09:00:27.1195861Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.1195934Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.1196168Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.1196247Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.1196525Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1196608Z return func(*args, **kwargs) 2025-12-04T09:00:27.1196877Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-12-04T09:00:27.1197031Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:00:27.1197035Z 2025-12-04T09:00:27.1197144Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.1197359Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.1197436Z res = mod(**inputs) 2025-12-04T09:00:27.1197710Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.1197807Z outputs = self.model( 2025-12-04T09:00:27.1198079Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.1198234Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.1198504Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.1198595Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.1198867Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1198954Z return func(*args, **kwargs) 2025-12-04T09:00:27.1199231Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-12-04T09:00:27.1199369Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:00:27.1199635Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:00:27.1199707Z return self.act(input) 2025-12-04T09:00:27.1199711Z 2025-12-04T09:00:27.1199830Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.1200034Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.1200101Z res = mod(**inputs) 2025-12-04T09:00:27.1200362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.1200432Z outputs = self.model( 2025-12-04T09:00:27.1200692Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.1200766Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.1200992Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.1201083Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.1201334Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1201413Z return func(*args, **kwargs) 2025-12-04T09:00:27.1201663Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 367, in forward 2025-12-04T09:00:27.1201746Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:00:27.1201750Z 2025-12-04T09:00:27.1201859Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.1202060Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.1202124Z res = mod(**inputs) 2025-12-04T09:00:27.1202384Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.1202453Z outputs = self.model( 2025-12-04T09:00:27.1202710Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.1202785Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.1203027Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.1203115Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.1203392Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1203462Z return func(*args, **kwargs) 2025-12-04T09:00:27.1203729Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.1203829Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.1204083Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1204171Z return func(*args, **kwargs) 2025-12-04T09:00:27.1204423Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 157, in forward 2025-12-04T09:00:27.1204547Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:00:27.1204550Z 2025-12-04T09:00:27.1204655Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.1204868Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.1204937Z res = mod(**inputs) 2025-12-04T09:00:27.1205188Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.1205265Z outputs = self.model( 2025-12-04T09:00:27.1205519Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.1205609Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.1205844Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.1205924Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.1206177Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1206248Z return func(*args, **kwargs) 2025-12-04T09:00:27.1206511Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.1206616Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.1206877Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1206954Z return func(*args, **kwargs) 2025-12-04T09:00:27.1207208Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 177, in forward 2025-12-04T09:00:27.1207292Z key_states = self.k_proj(current_states) 2025-12-04T09:00:27.1207296Z 2025-12-04T09:00:27.1207408Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.1207613Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.1207679Z res = mod(**inputs) 2025-12-04T09:00:27.1207947Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.1208016Z outputs = self.model( 2025-12-04T09:00:27.1208268Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.1208340Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.1208562Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.1208651Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.1208895Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1208982Z return func(*args, **kwargs) 2025-12-04T09:00:27.1209252Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.1209345Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.1209609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1209676Z return func(*args, **kwargs) 2025-12-04T09:00:27.1209914Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 157, in forward 2025-12-04T09:00:27.1210030Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:00:27.1210034Z 2025-12-04T09:00:27.1210133Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.1210351Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.1210415Z res = mod(**inputs) 2025-12-04T09:00:27.1210657Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.1210731Z outputs = self.model( 2025-12-04T09:00:27.1210971Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.1211045Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.1211266Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.1211342Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.1211585Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1211667Z return func(*args, **kwargs) 2025-12-04T09:00:27.1211913Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.1212017Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.1212257Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1212326Z return func(*args, **kwargs) 2025-12-04T09:00:27.1212578Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 199, in forward 2025-12-04T09:00:27.1212709Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-12-04T09:00:27.1212712Z 2025-12-04T09:00:27.1212818Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.1213009Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.1213073Z res = mod(**inputs) 2025-12-04T09:00:27.1213325Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.1213391Z outputs = self.model( 2025-12-04T09:00:27.1213643Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.1213715Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.1213929Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.1214014Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.1214254Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1214320Z return func(*args, **kwargs) 2025-12-04T09:00:27.1214568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.1214666Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.1214931Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1214998Z return func(*args, **kwargs) 2025-12-04T09:00:27.1215241Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 178, in forward 2025-12-04T09:00:27.1215349Z value_states = self.v_proj(current_states) 2025-12-04T09:00:27.1215352Z 2025-12-04T09:00:27.1215451Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.1215652Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.1215713Z res = mod(**inputs) 2025-12-04T09:00:27.1215954Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.1216029Z outputs = self.model( 2025-12-04T09:00:27.1216281Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.1216352Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.1216578Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.1216653Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.1216897Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1216965Z return func(*args, **kwargs) 2025-12-04T09:00:27.1217207Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.1217305Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.1217557Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1217625Z return func(*args, **kwargs) 2025-12-04T09:00:27.1217876Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 245, in forward 2025-12-04T09:00:27.1217969Z attn_output = torch.bmm(attn_probs, value_states) 2025-12-04T09:00:27.1217973Z 2025-12-04T09:00:27.1218076Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.1218271Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.1218333Z res = mod(**inputs) 2025-12-04T09:00:27.1218583Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.1218648Z outputs = self.model( 2025-12-04T09:00:27.1218901Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.1218972Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.1219188Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.1219271Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.1219507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1219574Z return func(*args, **kwargs) 2025-12-04T09:00:27.1219823Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.1219915Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.1220158Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1220224Z return func(*args, **kwargs) 2025-12-04T09:00:27.1220465Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-12-04T09:00:27.1220601Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-12-04T09:00:27.1220604Z 2025-12-04T09:00:27.1220890Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.1221101Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.1221166Z res = mod(**inputs) 2025-12-04T09:00:27.1221434Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.1221508Z outputs = self.model( 2025-12-04T09:00:27.1221750Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.1221822Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.1222044Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.1222144Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.1222390Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1222458Z return func(*args, **kwargs) 2025-12-04T09:00:27.1222708Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.1222813Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.1223049Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1223123Z return func(*args, **kwargs) 2025-12-04T09:00:27.1223363Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 260, in forward 2025-12-04T09:00:27.1223441Z attn_output = self.out_proj(attn_output) 2025-12-04T09:00:27.1223468Z 2025-12-04T09:00:27.1223578Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.1223770Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.1223831Z res = mod(**inputs) 2025-12-04T09:00:27.1224080Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.1224146Z outputs = self.model( 2025-12-04T09:00:27.1224398Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.1224479Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.1224689Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.1224771Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.1224998Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1225066Z return func(*args, **kwargs) 2025-12-04T09:00:27.1225316Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-12-04T09:00:27.1225434Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:00:27.1225437Z 2025-12-04T09:00:27.1225542Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.1225741Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.1225805Z res = mod(**inputs) 2025-12-04T09:00:27.1226058Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.1226125Z outputs = self.model( 2025-12-04T09:00:27.1226383Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.1226459Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.1226689Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.1226777Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.1227060Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1227132Z return func(*args, **kwargs) 2025-12-04T09:00:27.1227408Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-12-04T09:00:27.1227528Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:00:27.1227755Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:00:27.1227828Z return self.act(input) 2025-12-04T09:00:27.1227831Z 2025-12-04T09:00:27.1227935Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.1228160Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.1228228Z res = mod(**inputs) 2025-12-04T09:00:27.1228488Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.1228566Z outputs = self.model( 2025-12-04T09:00:27.1228821Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.1228903Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.1229128Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.1229208Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.1229468Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1229558Z return func(*args, **kwargs) 2025-12-04T09:00:27.1229818Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 367, in forward 2025-12-04T09:00:27.1229902Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:00:27.1229907Z 2025-12-04T09:00:27.1230011Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.1230220Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.1230288Z res = mod(**inputs) 2025-12-04T09:00:27.1230541Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.1230616Z outputs = self.model( 2025-12-04T09:00:27.1230880Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.1230958Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.1231181Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.1231258Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.1231506Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1231575Z return func(*args, **kwargs) 2025-12-04T09:00:27.1231821Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 369, in forward 2025-12-04T09:00:27.1231909Z hidden_states = residual + hidden_states 2025-12-04T09:00:27.1231912Z 2025-12-04T09:00:27.1232013Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.1232214Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.1232277Z res = mod(**inputs) 2025-12-04T09:00:27.1232534Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.1232613Z outputs = self.model( 2025-12-04T09:00:27.1232857Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.1232957Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.1233179Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.1233256Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.1233522Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1233591Z return func(*args, **kwargs) 2025-12-04T09:00:27.1233841Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.1233948Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.1234215Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1234295Z return func(*args, **kwargs) 2025-12-04T09:00:27.1234549Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 157, in forward 2025-12-04T09:00:27.1234663Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:00:27.1234666Z 2025-12-04T09:00:27.1234775Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.1234979Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.1235055Z res = mod(**inputs) 2025-12-04T09:00:27.1235311Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.1235380Z outputs = self.model( 2025-12-04T09:00:27.1235640Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.1235732Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.1235960Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.1236048Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.1236294Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1236375Z return func(*args, **kwargs) 2025-12-04T09:00:27.1236625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.1236723Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.1236977Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1237050Z return func(*args, **kwargs) 2025-12-04T09:00:27.1237301Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 177, in forward 2025-12-04T09:00:27.1237391Z key_states = self.k_proj(current_states) 2025-12-04T09:00:27.1237394Z 2025-12-04T09:00:27.1237500Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.1237709Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.1237776Z res = mod(**inputs) 2025-12-04T09:00:27.1238032Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.1238110Z outputs = self.model( 2025-12-04T09:00:27.1238436Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.1238525Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.1238756Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.1238842Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.1239133Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1239209Z return func(*args, **kwargs) 2025-12-04T09:00:27.1239492Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.1239616Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.1239863Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1239943Z return func(*args, **kwargs) 2025-12-04T09:00:27.1240217Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 157, in forward 2025-12-04T09:00:27.1240339Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:00:27.1240343Z 2025-12-04T09:00:27.1240479Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.1240698Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.1240777Z res = mod(**inputs) 2025-12-04T09:00:27.1241048Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.1241124Z outputs = self.model( 2025-12-04T09:00:27.1241401Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.1241480Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.1241718Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.1241809Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.1242090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1242179Z return func(*args, **kwargs) 2025-12-04T09:00:27.1242438Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.1242536Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.1242790Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1242861Z return func(*args, **kwargs) 2025-12-04T09:00:27.1243116Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 199, in forward 2025-12-04T09:00:27.1243261Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-12-04T09:00:27.1243265Z 2025-12-04T09:00:27.1243367Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.1243578Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.1243645Z res = mod(**inputs) 2025-12-04T09:00:27.1243904Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.1243983Z outputs = self.model( 2025-12-04T09:00:27.1244238Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.1244319Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.1244546Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.1244624Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.1244879Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1244948Z return func(*args, **kwargs) 2025-12-04T09:00:27.1245204Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.1245310Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.1245575Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1245654Z return func(*args, **kwargs) 2025-12-04T09:00:27.1245905Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 178, in forward 2025-12-04T09:00:27.1246013Z value_states = self.v_proj(current_states) 2025-12-04T09:00:27.1246017Z 2025-12-04T09:00:27.1246132Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.1246339Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.1246416Z res = mod(**inputs) 2025-12-04T09:00:27.1246674Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.1246759Z outputs = self.model( 2025-12-04T09:00:27.1247023Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.1247096Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.1247330Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.1247422Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.1247677Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1247756Z return func(*args, **kwargs) 2025-12-04T09:00:27.1248011Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.1248108Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.1248402Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1248470Z return func(*args, **kwargs) 2025-12-04T09:00:27.1248726Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 245, in forward 2025-12-04T09:00:27.1248831Z attn_output = torch.bmm(attn_probs, value_states) 2025-12-04T09:00:27.1248834Z 2025-12-04T09:00:27.1248939Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.1249147Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.1249211Z res = mod(**inputs) 2025-12-04T09:00:27.1249470Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.1249548Z outputs = self.model( 2025-12-04T09:00:27.1249804Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.1249885Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.1250113Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.1250195Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.1250450Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1250523Z return func(*args, **kwargs) 2025-12-04T09:00:27.1250776Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.1250879Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.1251127Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1251206Z return func(*args, **kwargs) 2025-12-04T09:00:27.1251463Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-12-04T09:00:27.1251594Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-12-04T09:00:27.1251614Z 2025-12-04T09:00:27.1251728Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.1251932Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.1252024Z res = mod(**inputs) 2025-12-04T09:00:27.1252278Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.1252349Z outputs = self.model( 2025-12-04T09:00:27.1252609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.1252684Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.1252931Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.1253021Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.1253272Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1253350Z return func(*args, **kwargs) 2025-12-04T09:00:27.1253601Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.1253700Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.1253956Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1254026Z return func(*args, **kwargs) 2025-12-04T09:00:27.1254330Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 260, in forward 2025-12-04T09:00:27.1254439Z attn_output = self.out_proj(attn_output) 2025-12-04T09:00:27.1254442Z 2025-12-04T09:00:27.1254548Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.1254779Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.1254844Z res = mod(**inputs) 2025-12-04T09:00:27.1255100Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.1255178Z outputs = self.model( 2025-12-04T09:00:27.1255441Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.1255520Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.1255754Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.1255833Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.1256090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1256159Z return func(*args, **kwargs) 2025-12-04T09:00:27.1256417Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-12-04T09:00:27.1256553Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:00:27.1256559Z 2025-12-04T09:00:27.1256659Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.1256865Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.1256929Z res = mod(**inputs) 2025-12-04T09:00:27.1257175Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.1257251Z outputs = self.model( 2025-12-04T09:00:27.1257497Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.1257579Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.1257799Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.1257920Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.1258178Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1258265Z return func(*args, **kwargs) 2025-12-04T09:00:27.1258513Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-12-04T09:00:27.1258639Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:00:27.1258856Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:00:27.1258934Z return self.act(input) 2025-12-04T09:00:27.1258938Z 2025-12-04T09:00:27.1259040Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.1259256Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.1259330Z res = mod(**inputs) 2025-12-04T09:00:27.1259579Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.1259646Z outputs = self.model( 2025-12-04T09:00:27.1259906Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.1259978Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.1260208Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.1260285Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.1260531Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1260624Z return func(*args, **kwargs) 2025-12-04T09:00:27.1260871Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 367, in forward 2025-12-04T09:00:27.1260959Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:00:27.1260963Z 2025-12-04T09:00:27.1261065Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.1261263Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.1261335Z res = mod(**inputs) 2025-12-04T09:00:27.1261584Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.1261651Z outputs = self.model( 2025-12-04T09:00:27.1261905Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.1261977Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.1274447Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.1274586Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.1274898Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1274986Z return func(*args, **kwargs) 2025-12-04T09:00:27.1275269Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.1275385Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.1275658Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1275730Z return func(*args, **kwargs) 2025-12-04T09:00:27.1275989Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 157, in forward 2025-12-04T09:00:27.1276114Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:00:27.1276121Z 2025-12-04T09:00:27.1276241Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.1276527Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.1276597Z res = mod(**inputs) 2025-12-04T09:00:27.1276859Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.1276966Z outputs = self.model( 2025-12-04T09:00:27.1277237Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.1277326Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.1277569Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.1277663Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.1277961Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1278040Z return func(*args, **kwargs) 2025-12-04T09:00:27.1278438Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.1278559Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.1278841Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1278920Z return func(*args, **kwargs) 2025-12-04T09:00:27.1279200Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 177, in forward 2025-12-04T09:00:27.1279307Z key_states = self.k_proj(current_states) 2025-12-04T09:00:27.1279344Z 2025-12-04T09:00:27.1279465Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.1279692Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.1279785Z res = mod(**inputs) 2025-12-04T09:00:27.1280063Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.1280153Z outputs = self.model( 2025-12-04T09:00:27.1280426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.1280507Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.1280763Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.1280851Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.1281112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1281212Z return func(*args, **kwargs) 2025-12-04T09:00:27.1281470Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.1281582Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.1281834Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1281908Z return func(*args, **kwargs) 2025-12-04T09:00:27.1282177Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 157, in forward 2025-12-04T09:00:27.1282293Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:00:27.1282297Z 2025-12-04T09:00:27.1282412Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.1282619Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.1282688Z res = mod(**inputs) 2025-12-04T09:00:27.1282950Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.1283023Z outputs = self.model( 2025-12-04T09:00:27.1283297Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.1283381Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.1283613Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.1283719Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.1283968Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1284042Z return func(*args, **kwargs) 2025-12-04T09:00:27.1284306Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.1284435Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.1284685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1284767Z return func(*args, **kwargs) 2025-12-04T09:00:27.1285024Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 199, in forward 2025-12-04T09:00:27.1285173Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-12-04T09:00:27.1285178Z 2025-12-04T09:00:27.1285286Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.1285493Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.1285566Z res = mod(**inputs) 2025-12-04T09:00:27.1285823Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.1285913Z outputs = self.model( 2025-12-04T09:00:27.1286171Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.1286249Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.1286489Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.1286572Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.1286823Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1286906Z return func(*args, **kwargs) 2025-12-04T09:00:27.1287160Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.1287264Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.1287513Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1287589Z return func(*args, **kwargs) 2025-12-04T09:00:27.1287855Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 178, in forward 2025-12-04T09:00:27.1287946Z value_states = self.v_proj(current_states) 2025-12-04T09:00:27.1287949Z 2025-12-04T09:00:27.1288060Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.1288265Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.1288333Z res = mod(**inputs) 2025-12-04T09:00:27.1288596Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.1288664Z outputs = self.model( 2025-12-04T09:00:27.1288920Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.1289008Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.1289239Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.1289324Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.1289593Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1289666Z return func(*args, **kwargs) 2025-12-04T09:00:27.1289944Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.1290044Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.1290296Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1290377Z return func(*args, **kwargs) 2025-12-04T09:00:27.1290645Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 245, in forward 2025-12-04T09:00:27.1290763Z attn_output = torch.bmm(attn_probs, value_states) 2025-12-04T09:00:27.1290767Z 2025-12-04T09:00:27.1290875Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.1291081Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.1291152Z res = mod(**inputs) 2025-12-04T09:00:27.1291410Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.1291491Z outputs = self.model( 2025-12-04T09:00:27.1291747Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.1291823Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.1292061Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.1292158Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.1292410Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1292493Z return func(*args, **kwargs) 2025-12-04T09:00:27.1292748Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.1292856Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.1293108Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1293178Z return func(*args, **kwargs) 2025-12-04T09:00:27.1293442Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-12-04T09:00:27.1293574Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-12-04T09:00:27.1293580Z 2025-12-04T09:00:27.1293693Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.1293901Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.1293970Z res = mod(**inputs) 2025-12-04T09:00:27.1294236Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.1294322Z outputs = self.model( 2025-12-04T09:00:27.1294568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.1294648Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.1294871Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.1294959Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.1295203Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1295274Z return func(*args, **kwargs) 2025-12-04T09:00:27.1295532Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.1295648Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.1295902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1295988Z return func(*args, **kwargs) 2025-12-04T09:00:27.1296242Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 260, in forward 2025-12-04T09:00:27.1296335Z attn_output = self.out_proj(attn_output) 2025-12-04T09:00:27.1296338Z 2025-12-04T09:00:27.1296442Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.1296642Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.1296715Z res = mod(**inputs) 2025-12-04T09:00:27.1296988Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.1297069Z outputs = self.model( 2025-12-04T09:00:27.1297321Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.1297393Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.1297622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.1297700Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.1297940Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1298018Z return func(*args, **kwargs) 2025-12-04T09:00:27.1298266Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-12-04T09:00:27.1298415Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:00:27.1298419Z 2025-12-04T09:00:27.1298521Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.1298718Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.1298791Z res = mod(**inputs) 2025-12-04T09:00:27.1299038Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.1299114Z outputs = self.model( 2025-12-04T09:00:27.1299361Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.1299434Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.1299663Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.1299743Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.1299985Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1300063Z return func(*args, **kwargs) 2025-12-04T09:00:27.1300311Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-12-04T09:00:27.1300439Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:00:27.1300658Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:00:27.1300730Z return self.act(input) 2025-12-04T09:00:27.1300734Z 2025-12-04T09:00:27.1300846Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.1301041Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.1301106Z res = mod(**inputs) 2025-12-04T09:00:27.1301367Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.1301434Z outputs = self.model( 2025-12-04T09:00:27.1301705Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.1301779Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.1302002Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.1302106Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.1302353Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1302427Z return func(*args, **kwargs) 2025-12-04T09:00:27.1302676Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 367, in forward 2025-12-04T09:00:27.1302761Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:00:27.1302765Z 2025-12-04T09:00:27.1302887Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.1303087Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.1303154Z res = mod(**inputs) 2025-12-04T09:00:27.1303412Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.1303482Z outputs = self.model( 2025-12-04T09:00:27.1303742Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.1303817Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.1304041Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.1304126Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.1304388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1304458Z return func(*args, **kwargs) 2025-12-04T09:00:27.1304720Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 369, in forward 2025-12-04T09:00:27.1304802Z hidden_states = residual + hidden_states 2025-12-04T09:00:27.1304805Z 2025-12-04T09:00:27.1304913Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.1305109Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.1305174Z res = mod(**inputs) 2025-12-04T09:00:27.1305430Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.1305504Z outputs = self.model( 2025-12-04T09:00:27.1305750Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.1305831Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.1306055Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.1306144Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.1306396Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1306467Z return func(*args, **kwargs) 2025-12-04T09:00:27.1306731Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.1306831Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.1307081Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1307159Z return func(*args, **kwargs) 2025-12-04T09:00:27.1307415Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 157, in forward 2025-12-04T09:00:27.1307535Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:00:27.1307538Z 2025-12-04T09:00:27.1307655Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.1307860Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.1307934Z res = mod(**inputs) 2025-12-04T09:00:27.1308216Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.1308294Z outputs = self.model( 2025-12-04T09:00:27.1308550Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.1308623Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.1308858Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.1308955Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.1309214Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1309295Z return func(*args, **kwargs) 2025-12-04T09:00:27.1309562Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.1309685Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.1309942Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1310013Z return func(*args, **kwargs) 2025-12-04T09:00:27.1310274Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 177, in forward 2025-12-04T09:00:27.1310376Z key_states = self.k_proj(current_states) 2025-12-04T09:00:27.1310380Z 2025-12-04T09:00:27.1310491Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.1310710Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.1310780Z res = mod(**inputs) 2025-12-04T09:00:27.1311061Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.1311133Z outputs = self.model( 2025-12-04T09:00:27.1311388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.1311468Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.1311707Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.1311799Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.1312061Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1312141Z return func(*args, **kwargs) 2025-12-04T09:00:27.1312416Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.1312522Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.1312783Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1312864Z return func(*args, **kwargs) 2025-12-04T09:00:27.1313135Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 157, in forward 2025-12-04T09:00:27.1313263Z query_states = self.q_proj(hidden_states) * self.scaling 2025-12-04T09:00:27.1313267Z 2025-12-04T09:00:27.1313376Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.1313590Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.1313670Z res = mod(**inputs) 2025-12-04T09:00:27.1313937Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.1314026Z outputs = self.model( 2025-12-04T09:00:27.1314300Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.1314377Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.1314651Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.1314735Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.1314998Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1315081Z return func(*args, **kwargs) 2025-12-04T09:00:27.1315357Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.1315484Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.1315753Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1315829Z return func(*args, **kwargs) 2025-12-04T09:00:27.1316105Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 199, in forward 2025-12-04T09:00:27.1316255Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-12-04T09:00:27.1316259Z 2025-12-04T09:00:27.1316367Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.1316591Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.1316661Z res = mod(**inputs) 2025-12-04T09:00:27.1316936Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.1317028Z outputs = self.model( 2025-12-04T09:00:27.1317296Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.1317384Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.1317633Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.1317727Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.1317999Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1318074Z return func(*args, **kwargs) 2025-12-04T09:00:27.1318444Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.1318556Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.1318834Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1318920Z return func(*args, **kwargs) 2025-12-04T09:00:27.1319201Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 178, in forward 2025-12-04T09:00:27.1319304Z value_states = self.v_proj(current_states) 2025-12-04T09:00:27.1319308Z 2025-12-04T09:00:27.1319423Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.1319654Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.1319734Z res = mod(**inputs) 2025-12-04T09:00:27.1320024Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.1320099Z outputs = self.model( 2025-12-04T09:00:27.1320377Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.1320456Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.1320966Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.1321127Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.1321383Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1321495Z return func(*args, **kwargs) 2025-12-04T09:00:27.1321748Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.1321855Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.1322119Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1322194Z return func(*args, **kwargs) 2025-12-04T09:00:27.1322491Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 245, in forward 2025-12-04T09:00:27.1322596Z attn_output = torch.bmm(attn_probs, value_states) 2025-12-04T09:00:27.1322600Z 2025-12-04T09:00:27.1322711Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.1322934Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.1323006Z res = mod(**inputs) 2025-12-04T09:00:27.1323288Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.1323362Z outputs = self.model( 2025-12-04T09:00:27.1323630Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.1323719Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.1323998Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.1324085Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.1324338Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1324407Z return func(*args, **kwargs) 2025-12-04T09:00:27.1324671Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.1324770Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.1325020Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1325094Z return func(*args, **kwargs) 2025-12-04T09:00:27.1325347Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-12-04T09:00:27.1325489Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-12-04T09:00:27.1325493Z 2025-12-04T09:00:27.1325597Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.1325801Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.1325874Z res = mod(**inputs) 2025-12-04T09:00:27.1326131Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.1326202Z outputs = self.model( 2025-12-04T09:00:27.1326465Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.1326539Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.1326777Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.1326858Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.1327111Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1327189Z return func(*args, **kwargs) 2025-12-04T09:00:27.1327457Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 333, in forward 2025-12-04T09:00:27.1327564Z hidden_states, self_attn_weights = self.self_attn( 2025-12-04T09:00:27.1327814Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1327902Z return func(*args, **kwargs) 2025-12-04T09:00:27.1328166Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 260, in forward 2025-12-04T09:00:27.1328249Z attn_output = self.out_proj(attn_output) 2025-12-04T09:00:27.1328252Z 2025-12-04T09:00:27.1328356Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.1328567Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.1328651Z res = mod(**inputs) 2025-12-04T09:00:27.1328914Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.1328985Z outputs = self.model( 2025-12-04T09:00:27.1329240Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.1329323Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.1329553Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.1329639Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.1329962Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1330073Z return func(*args, **kwargs) 2025-12-04T09:00:27.1330368Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-12-04T09:00:27.1330491Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:00:27.1330495Z 2025-12-04T09:00:27.1330601Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.1330819Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.1330885Z res = mod(**inputs) 2025-12-04T09:00:27.1331150Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.1331221Z outputs = self.model( 2025-12-04T09:00:27.1331476Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.1331562Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.1331802Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.1331886Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.1332152Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1332225Z return func(*args, **kwargs) 2025-12-04T09:00:27.1332490Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 365, in forward 2025-12-04T09:00:27.1332612Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-12-04T09:00:27.1332835Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:00:27.1332916Z return self.act(input) 2025-12-04T09:00:27.1332919Z 2025-12-04T09:00:27.1333025Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.1333235Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.1333304Z res = mod(**inputs) 2025-12-04T09:00:27.1333565Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 668, in forward 2025-12-04T09:00:27.1333644Z outputs = self.model( 2025-12-04T09:00:27.1333925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 559, in forward 2025-12-04T09:00:27.1334003Z layer_outputs = decoder_layer( 2025-12-04T09:00:27.1334241Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:00:27.1334341Z return super().__call__(*args, **kwargs) 2025-12-04T09:00:27.1334604Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-12-04T09:00:27.1334676Z return func(*args, **kwargs) 2025-12-04T09:00:27.1334939Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 367, in forward 2025-12-04T09:00:27.1335049Z hidden_states = self.fc2(hidden_states) 2025-12-04T09:00:27.1335053Z 2025-12-04T09:00:27.1335158Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.1335371Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.1335438Z res = mod(**inputs) 2025-12-04T09:00:27.1335691Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 685, in forward 2025-12-04T09:00:27.1335782Z logits = self.lm_head(outputs[0]) 2025-12-04T09:00:27.1335787Z 2025-12-04T09:00:27.1335889Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:00:27.1336087Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:00:27.1336160Z res = mod(**inputs) 2025-12-04T09:00:27.1336424Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 689, in forward 2025-12-04T09:00:27.1336523Z loss = self.loss_function( 2025-12-04T09:00:27.1336774Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/loss/loss_utils.py", line 67, in ForCausalLMLoss 2025-12-04T09:00:27.1336962Z loss = fixed_cross_entropy(logits, shift_labels, num_items_in_batch, ignore_index, **kwargs) 2025-12-04T09:00:27.1337221Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/loss/loss_utils.py", line 36, in fixed_cross_entropy 2025-12-04T09:00:27.1337424Z loss = nn.functional.cross_entropy(source, target, ignore_index=ignore_index, reduction=reduction) 2025-12-04T09:00:27.1337428Z 2025-12-04T09:00:40.2391585Z Compilation time (from dynamo_timed): 26.683917932 2025-12-04T09:00:40.2490929Z pass 2025-12-04T09:00:40.2491309Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:00:40.2492175Z TIMING: _recursive_pre_grad_passes:0.01312 _recursive_joint_graph_passes:0.80335 _recursive_post_grad_passes:0.255 async_compile.wait:0.88332 code_gen:12.62752 inductor_compile:16.0444 backend_compile:21.76051 gc:0.00038 entire_frame_compile:26.68392 total_wall_time:26.68392 2025-12-04T09:00:40.2493142Z STATS: call_* op count: 921 | FakeTensorMode.__torch_dispatch__:17015 | FakeTensor.__torch_dispatch__:9977 | ProxyTorchDispatchMode.__torch_dispatch__:4882 2025-12-04T09:00:40.2493665Z Dynamo produced 1 graphs covering 921 ops with 0 graph breaks (0 unique) 2025-12-04T09:00:43.2515943Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/__init__.py:65: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-12-04T09:00:43.2516884Z import pynvml # type: ignore[import] 2025-12-04T09:00:46.7604898Z 2025-12-04T09:00:50.8045529Z loading model: 0it [00:00, ?it/s] 2025-12-04T09:00:50.8048744Z loading model: 0it [00:04, ?it/s] 2025-12-04T09:00:50.8067330Z cpu eval XLNetLMHeadModel 2025-12-04T09:00:57.5399594Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:01:00.4329529Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:01:03.2484956Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:01:23.9568266Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:23.9568745Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:23.9569105Z res = mod(**inputs) 2025-12-04T09:01:23.9569527Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:23.9569962Z transformer_outputs = self.transformer( 2025-12-04T09:01:23.9570714Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1307, in forward 2025-12-04T09:01:23.9571150Z word_emb_k = self.word_embedding(input_ids) 2025-12-04T09:01:23.9571306Z 2025-12-04T09:01:23.9571434Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:23.9571810Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:23.9572235Z res = mod(**inputs) 2025-12-04T09:01:23.9572622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:23.9573036Z transformer_outputs = self.transformer( 2025-12-04T09:01:23.9573453Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1334, in forward 2025-12-04T09:01:23.9573934Z pos_emb = self.relative_positional_encoding(qlen, klen, bsz=bsz) 2025-12-04T09:01:23.9574528Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1157, in relative_positional_encoding 2025-12-04T09:01:23.9575041Z pos_emb = self.positional_embedding(fwd_pos_seq, inv_freq, bsz) 2025-12-04T09:01:23.9575536Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1115, in positional_embedding 2025-12-04T09:01:23.9576062Z pos_emb = torch.cat([torch.sin(sinusoid_inp), torch.cos(sinusoid_inp)], dim=-1) 2025-12-04T09:01:23.9576282Z 2025-12-04T09:01:23.9576401Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:23.9576774Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:23.9577111Z res = mod(**inputs) 2025-12-04T09:01:23.9577533Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:23.9577955Z transformer_outputs = self.transformer( 2025-12-04T09:01:23.9578363Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1334, in forward 2025-12-04T09:01:23.9578823Z pos_emb = self.relative_positional_encoding(qlen, klen, bsz=bsz) 2025-12-04T09:01:23.9579332Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1157, in relative_positional_encoding 2025-12-04T09:01:23.9579834Z pos_emb = self.positional_embedding(fwd_pos_seq, inv_freq, bsz) 2025-12-04T09:01:23.9580354Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1115, in positional_embedding 2025-12-04T09:01:23.9580903Z pos_emb = torch.cat([torch.sin(sinusoid_inp), torch.cos(sinusoid_inp)], dim=-1) 2025-12-04T09:01:23.9581124Z 2025-12-04T09:01:23.9581243Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:23.9581612Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:23.9581959Z res = mod(**inputs) 2025-12-04T09:01:23.9582342Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:23.9582816Z transformer_outputs = self.transformer( 2025-12-04T09:01:23.9583229Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:23.9583681Z outputs = layer_module( 2025-12-04T09:01:23.9584081Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:23.9584512Z outputs = self.rel_attn( 2025-12-04T09:01:23.9584916Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:01:23.9585395Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:01:23.9585852Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:01:23.9586315Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:01:23.9586500Z 2025-12-04T09:01:23.9586618Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:23.9587012Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:23.9587362Z res = mod(**inputs) 2025-12-04T09:01:23.9587749Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:23.9588188Z transformer_outputs = self.transformer( 2025-12-04T09:01:23.9588624Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:23.9589038Z outputs = layer_module( 2025-12-04T09:01:23.9589459Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:23.9589887Z outputs = self.rel_attn( 2025-12-04T09:01:23.9590294Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:01:23.9590723Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:01:23.9591185Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:01:23.9591689Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:01:23.9591870Z 2025-12-04T09:01:23.9591989Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:23.9592388Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:23.9592738Z res = mod(**inputs) 2025-12-04T09:01:23.9593129Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:23.9593564Z transformer_outputs = self.transformer( 2025-12-04T09:01:23.9594003Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:23.9594424Z outputs = layer_module( 2025-12-04T09:01:23.9594828Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:23.9595245Z outputs = self.rel_attn( 2025-12-04T09:01:23.9595651Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:01:23.9596091Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:01:23.9596549Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:01:23.9597040Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:01:23.9597223Z 2025-12-04T09:01:23.9597337Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:23.9597727Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:23.9598085Z res = mod(**inputs) 2025-12-04T09:01:23.9598767Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:23.9599248Z transformer_outputs = self.transformer( 2025-12-04T09:01:23.9599692Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:23.9600118Z outputs = layer_module( 2025-12-04T09:01:23.9600524Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:23.9600944Z outputs = self.rel_attn( 2025-12-04T09:01:23.9601367Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:01:23.9601812Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:01:23.9602279Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:01:23.9602775Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:01:23.9602954Z 2025-12-04T09:01:23.9603068Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:23.9603475Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:23.9603832Z res = mod(**inputs) 2025-12-04T09:01:23.9604224Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:23.9604634Z transformer_outputs = self.transformer( 2025-12-04T09:01:23.9605067Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:23.9605467Z outputs = layer_module( 2025-12-04T09:01:23.9605843Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:23.9606241Z outputs = self.rel_attn( 2025-12-04T09:01:23.9606622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:01:23.9607047Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:01:23.9607475Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:01:23.9607951Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:01:23.9608141Z 2025-12-04T09:01:23.9608253Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:23.9608652Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:23.9609005Z res = mod(**inputs) 2025-12-04T09:01:23.9609421Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:23.9609877Z transformer_outputs = self.transformer( 2025-12-04T09:01:23.9610318Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:23.9610719Z outputs = layer_module( 2025-12-04T09:01:23.9611099Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:23.9611491Z outputs = self.rel_attn( 2025-12-04T09:01:23.9611906Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:01:23.9612331Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:01:23.9612772Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:01:23.9613220Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:01:23.9613418Z 2025-12-04T09:01:23.9613525Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:23.9613907Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:23.9614254Z res = mod(**inputs) 2025-12-04T09:01:23.9614618Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:23.9615050Z transformer_outputs = self.transformer( 2025-12-04T09:01:23.9615463Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:23.9615854Z outputs = layer_module( 2025-12-04T09:01:23.9616275Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:23.9616669Z outputs = self.rel_attn( 2025-12-04T09:01:23.9617050Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:01:23.9617454Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:01:23.9617889Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:01:23.9618348Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:01:23.9618516Z 2025-12-04T09:01:23.9618627Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:23.9618988Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:23.9619320Z res = mod(**inputs) 2025-12-04T09:01:23.9619747Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:23.9620145Z transformer_outputs = self.transformer( 2025-12-04T09:01:23.9620547Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:23.9621118Z outputs = layer_module( 2025-12-04T09:01:23.9621493Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:23.9621882Z outputs = self.rel_attn( 2025-12-04T09:01:23.9622266Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:01:23.9622695Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:01:23.9623123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:01:23.9623576Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:01:23.9623754Z 2025-12-04T09:01:23.9623864Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:23.9624243Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:23.9624562Z res = mod(**inputs) 2025-12-04T09:01:23.9624929Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:23.9625339Z transformer_outputs = self.transformer( 2025-12-04T09:01:23.9625754Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:23.9626141Z outputs = layer_module( 2025-12-04T09:01:23.9626519Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:23.9626936Z outputs = self.rel_attn( 2025-12-04T09:01:23.9627303Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:01:23.9627712Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:01:23.9628186Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:01:23.9628637Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:01:23.9628834Z 2025-12-04T09:01:23.9628944Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:23.9629320Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:23.9629659Z res = mod(**inputs) 2025-12-04T09:01:23.9630035Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:23.9630447Z transformer_outputs = self.transformer( 2025-12-04T09:01:23.9630884Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:23.9631282Z outputs = layer_module( 2025-12-04T09:01:23.9631657Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:23.9632061Z outputs = self.rel_attn( 2025-12-04T09:01:23.9632442Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:01:23.9632857Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:01:23.9633284Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:01:23.9633746Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:01:23.9633921Z 2025-12-04T09:01:23.9634062Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:23.9634462Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:23.9634814Z res = mod(**inputs) 2025-12-04T09:01:23.9635218Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:23.9635660Z transformer_outputs = self.transformer( 2025-12-04T09:01:23.9636093Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:23.9636516Z outputs = layer_module( 2025-12-04T09:01:23.9636920Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:23.9637347Z outputs = self.rel_attn( 2025-12-04T09:01:23.9637781Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:01:23.9638281Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:01:23.9638759Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:01:23.9639247Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:01:23.9639427Z 2025-12-04T09:01:23.9639541Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:23.9639938Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:23.9640296Z res = mod(**inputs) 2025-12-04T09:01:23.9640687Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:23.9641130Z transformer_outputs = self.transformer( 2025-12-04T09:01:23.9641571Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:23.9642003Z outputs = layer_module( 2025-12-04T09:01:23.9642400Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:23.9642826Z outputs = self.rel_attn( 2025-12-04T09:01:23.9643252Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:01:23.9643696Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:01:23.9644163Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:01:23.9644650Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:01:23.9644830Z 2025-12-04T09:01:23.9644948Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:23.9645327Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:23.9645681Z res = mod(**inputs) 2025-12-04T09:01:23.9646097Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:23.9646532Z transformer_outputs = self.transformer( 2025-12-04T09:01:23.9646978Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:23.9647393Z outputs = layer_module( 2025-12-04T09:01:23.9647793Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:23.9648214Z outputs = self.rel_attn( 2025-12-04T09:01:23.9648587Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:01:23.9648981Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:01:23.9649388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:01:23.9649829Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:01:23.9650000Z 2025-12-04T09:01:23.9650100Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:23.9650449Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:23.9650764Z res = mod(**inputs) 2025-12-04T09:01:23.9651122Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:23.9651531Z transformer_outputs = self.transformer( 2025-12-04T09:01:23.9651930Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:23.9652311Z outputs = layer_module( 2025-12-04T09:01:23.9652685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:23.9653100Z outputs = self.rel_attn( 2025-12-04T09:01:23.9653463Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:01:23.9653861Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:01:23.9654284Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:01:23.9654734Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:01:23.9654913Z 2025-12-04T09:01:23.9655021Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:23.9655366Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:23.9655691Z res = mod(**inputs) 2025-12-04T09:01:23.9656050Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:23.9656451Z transformer_outputs = self.transformer( 2025-12-04T09:01:23.9656852Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:23.9657244Z outputs = layer_module( 2025-12-04T09:01:23.9657628Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:23.9658007Z outputs = self.rel_attn( 2025-12-04T09:01:23.9658407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:01:23.9658802Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:01:23.9659216Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:01:23.9659648Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:01:23.9659819Z 2025-12-04T09:01:23.9659922Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:23.9660304Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:23.9660619Z res = mod(**inputs) 2025-12-04T09:01:23.9660992Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:23.9661396Z transformer_outputs = self.transformer( 2025-12-04T09:01:23.9661798Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:23.9662179Z outputs = layer_module( 2025-12-04T09:01:23.9662547Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:23.9662932Z outputs = self.rel_attn( 2025-12-04T09:01:23.9663297Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:01:23.9663723Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:01:23.9664140Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:01:23.9664591Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:01:23.9664760Z 2025-12-04T09:01:23.9664868Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:23.9665240Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:23.9665600Z res = mod(**inputs) 2025-12-04T09:01:23.9665972Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:23.9666377Z transformer_outputs = self.transformer( 2025-12-04T09:01:23.9666797Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:23.9667201Z outputs = layer_module( 2025-12-04T09:01:23.9667573Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:23.9667974Z outputs = self.rel_attn( 2025-12-04T09:01:23.9668360Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:01:23.9668778Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:01:23.9669204Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:01:23.9669664Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:01:23.9669841Z 2025-12-04T09:01:23.9669947Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:23.9670330Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:23.9670658Z res = mod(**inputs) 2025-12-04T09:01:23.9671033Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:23.9671454Z transformer_outputs = self.transformer( 2025-12-04T09:01:23.9671873Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:23.9672286Z outputs = layer_module( 2025-12-04T09:01:23.9672687Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:23.9673088Z outputs = self.rel_attn( 2025-12-04T09:01:23.9673466Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:01:23.9673887Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:01:23.9674338Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:01:23.9674858Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:01:23.9675043Z 2025-12-04T09:01:23.9675157Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:23.9675553Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:23.9675906Z res = mod(**inputs) 2025-12-04T09:01:23.9676297Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:23.9676748Z transformer_outputs = self.transformer( 2025-12-04T09:01:23.9677182Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:23.9677602Z outputs = layer_module( 2025-12-04T09:01:23.9677999Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:23.9678548Z outputs = self.rel_attn( 2025-12-04T09:01:23.9678968Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:01:23.9679426Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:01:23.9679896Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:01:23.9680359Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:01:23.9680538Z 2025-12-04T09:01:23.9680649Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:23.9681005Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:23.9681330Z res = mod(**inputs) 2025-12-04T09:01:23.9681695Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:23.9682107Z transformer_outputs = self.transformer( 2025-12-04T09:01:23.9682511Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:23.9682912Z outputs = layer_module( 2025-12-04T09:01:23.9683291Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:23.9683700Z outputs = self.rel_attn( 2025-12-04T09:01:23.9684066Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:01:23.9684469Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:01:23.9684895Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:01:23.9685344Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:01:23.9685522Z 2025-12-04T09:01:23.9685631Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:23.9686000Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:23.9686332Z res = mod(**inputs) 2025-12-04T09:01:23.9686718Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:23.9687133Z transformer_outputs = self.transformer( 2025-12-04T09:01:23.9687566Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:23.9687960Z outputs = layer_module( 2025-12-04T09:01:23.9688343Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:23.9688744Z outputs = self.rel_attn( 2025-12-04T09:01:23.9689133Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:01:23.9689557Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:01:23.9689995Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:01:23.9691140Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:01:23.9691311Z 2025-12-04T09:01:23.9691424Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:23.9691803Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:23.9692133Z res = mod(**inputs) 2025-12-04T09:01:23.9692506Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:23.9692915Z transformer_outputs = self.transformer( 2025-12-04T09:01:23.9693330Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:23.9693745Z outputs = layer_module( 2025-12-04T09:01:23.9694126Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:23.9694521Z outputs = self.rel_attn( 2025-12-04T09:01:23.9694909Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:01:23.9695326Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:01:23.9695760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:01:23.9696228Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:01:23.9696402Z 2025-12-04T09:01:23.9696508Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:23.9696877Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:23.9697200Z res = mod(**inputs) 2025-12-04T09:01:23.9697579Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:23.9697982Z transformer_outputs = self.transformer( 2025-12-04T09:01:23.9698377Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:23.9698757Z outputs = layer_module( 2025-12-04T09:01:23.9699123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:23.9699510Z outputs = self.rel_attn( 2025-12-04T09:01:23.9699879Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:01:23.9700282Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:01:23.9700704Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:01:23.9701151Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:01:23.9701315Z 2025-12-04T09:01:23.9701436Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:23.9701798Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:23.9702125Z res = mod(**inputs) 2025-12-04T09:01:23.9702519Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:23.9702928Z transformer_outputs = self.transformer( 2025-12-04T09:01:23.9703350Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:23.9703760Z outputs = layer_module( 2025-12-04T09:01:23.9704136Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:23.9704586Z outputs = self.rel_attn( 2025-12-04T09:01:23.9704974Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:01:23.9705387Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:01:23.9705815Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:01:23.9706280Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:01:23.9706449Z 2025-12-04T09:01:23.9706572Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:23.9706931Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:23.9707250Z res = mod(**inputs) 2025-12-04T09:01:23.9707616Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:23.9708077Z transformer_outputs = self.transformer( 2025-12-04T09:01:23.9708505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:23.9708928Z outputs = layer_module( 2025-12-04T09:01:23.9709333Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:23.9709776Z outputs = self.rel_attn( 2025-12-04T09:01:23.9710175Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-12-04T09:01:23.9710631Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-12-04T09:01:23.9710796Z 2025-12-04T09:01:23.9710919Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:23.9711310Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:23.9711658Z res = mod(**inputs) 2025-12-04T09:01:23.9712065Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:23.9712507Z transformer_outputs = self.transformer( 2025-12-04T09:01:23.9712946Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:23.9713350Z outputs = layer_module( 2025-12-04T09:01:23.9713734Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:23.9714130Z outputs = self.rel_attn( 2025-12-04T09:01:23.9714507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-12-04T09:01:23.9714966Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-12-04T09:01:23.9715134Z 2025-12-04T09:01:23.9715253Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:23.9715640Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:23.9715992Z res = mod(**inputs) 2025-12-04T09:01:23.9716403Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:23.9716838Z transformer_outputs = self.transformer( 2025-12-04T09:01:23.9717285Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:23.9717710Z outputs = layer_module( 2025-12-04T09:01:23.9718113Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:23.9718610Z outputs = self.rel_attn( 2025-12-04T09:01:23.9719016Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:01:23.9719490Z attn_vec = self.rel_attn_core( 2025-12-04T09:01:23.9719948Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-12-04T09:01:23.9720492Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-12-04T09:01:23.9720832Z 2025-12-04T09:01:23.9720952Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:23.9721351Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:23.9721731Z res = mod(**inputs) 2025-12-04T09:01:23.9722127Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:23.9722579Z transformer_outputs = self.transformer( 2025-12-04T09:01:23.9723031Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1334, in forward 2025-12-04T09:01:23.9723568Z pos_emb = self.relative_positional_encoding(qlen, klen, bsz=bsz) 2025-12-04T09:01:23.9724118Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1157, in relative_positional_encoding 2025-12-04T09:01:23.9724668Z pos_emb = self.positional_embedding(fwd_pos_seq, inv_freq, bsz) 2025-12-04T09:01:23.9725200Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1115, in positional_embedding 2025-12-04T09:01:23.9725758Z pos_emb = torch.cat([torch.sin(sinusoid_inp), torch.cos(sinusoid_inp)], dim=-1) 2025-12-04T09:01:23.9725995Z 2025-12-04T09:01:23.9726111Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:23.9726508Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:23.9726866Z res = mod(**inputs) 2025-12-04T09:01:23.9727265Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:23.9727712Z transformer_outputs = self.transformer( 2025-12-04T09:01:23.9728157Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:23.9728581Z outputs = layer_module( 2025-12-04T09:01:23.9728982Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:23.9729402Z outputs = self.rel_attn( 2025-12-04T09:01:23.9729798Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-12-04T09:01:23.9730252Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-12-04T09:01:23.9730451Z 2025-12-04T09:01:23.9730557Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:23.9730928Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:23.9731252Z res = mod(**inputs) 2025-12-04T09:01:23.9731635Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:23.9732040Z transformer_outputs = self.transformer( 2025-12-04T09:01:23.9732439Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:23.9732858Z outputs = layer_module( 2025-12-04T09:01:23.9733236Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:23.9733625Z outputs = self.rel_attn( 2025-12-04T09:01:23.9733998Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:01:23.9734387Z attn_vec = self.rel_attn_core( 2025-12-04T09:01:23.9734821Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-12-04T09:01:23.9735288Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-12-04T09:01:23.9735476Z 2025-12-04T09:01:23.9735585Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:23.9735948Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:23.9736283Z res = mod(**inputs) 2025-12-04T09:01:23.9736653Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:23.9737064Z transformer_outputs = self.transformer( 2025-12-04T09:01:23.9737473Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:23.9737896Z outputs = layer_module( 2025-12-04T09:01:23.9738278Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:23.9738677Z outputs = self.rel_attn( 2025-12-04T09:01:23.9739081Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-12-04T09:01:23.9739539Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-12-04T09:01:23.9739699Z 2025-12-04T09:01:23.9739815Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:23.9740183Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:23.9740516Z res = mod(**inputs) 2025-12-04T09:01:23.9740894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:23.9741356Z transformer_outputs = self.transformer( 2025-12-04T09:01:23.9741762Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:23.9742150Z outputs = layer_module( 2025-12-04T09:01:23.9742523Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:23.9742904Z outputs = self.rel_attn( 2025-12-04T09:01:23.9743274Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:01:23.9743671Z attn_vec = self.rel_attn_core( 2025-12-04T09:01:23.9744067Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-12-04T09:01:23.9744542Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-12-04T09:01:23.9744726Z 2025-12-04T09:01:23.9744830Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:23.9745193Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:23.9745511Z res = mod(**inputs) 2025-12-04T09:01:23.9745895Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:23.9746296Z transformer_outputs = self.transformer( 2025-12-04T09:01:23.9746688Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:23.9747095Z outputs = layer_module( 2025-12-04T09:01:23.9747467Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:23.9747856Z outputs = self.rel_attn( 2025-12-04T09:01:23.9748222Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:01:23.9748634Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:01:23.9749077Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:01:23.9749533Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:01:23.9749703Z 2025-12-04T09:01:23.9749807Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:23.9750173Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:23.9750497Z res = mod(**inputs) 2025-12-04T09:01:23.9750853Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:23.9751274Z transformer_outputs = self.transformer( 2025-12-04T09:01:23.9751697Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:23.9752111Z outputs = layer_module( 2025-12-04T09:01:23.9752501Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:23.9752900Z outputs = self.rel_attn( 2025-12-04T09:01:23.9753285Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:01:23.9753701Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:01:23.9754132Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:01:23.9754595Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:01:23.9754765Z 2025-12-04T09:01:23.9754878Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:23.9755244Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:23.9755578Z res = mod(**inputs) 2025-12-04T09:01:23.9755959Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:23.9756377Z transformer_outputs = self.transformer( 2025-12-04T09:01:23.9756783Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:23.9757184Z outputs = layer_module( 2025-12-04T09:01:23.9757566Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:01:23.9758116Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:01:23.9758774Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:01:23.9759232Z return forward_fn(*input_tensors) 2025-12-04T09:01:23.9759690Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:01:23.9760119Z output_x = self.ff(output_x) 2025-12-04T09:01:23.9760539Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-12-04T09:01:23.9760998Z output = self.layer_1(output) 2025-12-04T09:01:23.9761144Z 2025-12-04T09:01:23.9761261Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:23.9761630Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:23.9761984Z res = mod(**inputs) 2025-12-04T09:01:23.9762363Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:23.9762773Z transformer_outputs = self.transformer( 2025-12-04T09:01:23.9763183Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:23.9763586Z outputs = layer_module( 2025-12-04T09:01:23.9763979Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:01:23.9764516Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:01:23.9765064Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:01:23.9765484Z return forward_fn(*input_tensors) 2025-12-04T09:01:23.9765890Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:01:23.9766294Z output_x = self.ff(output_x) 2025-12-04T09:01:23.9766686Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-12-04T09:01:23.9767126Z output = self.activation_function(output) 2025-12-04T09:01:23.9767497Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:01:23.9767858Z return self.act(input) 2025-12-04T09:01:23.9767979Z 2025-12-04T09:01:23.9768090Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:23.9768465Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:23.9768791Z res = mod(**inputs) 2025-12-04T09:01:23.9769167Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:23.9769582Z transformer_outputs = self.transformer( 2025-12-04T09:01:23.9769995Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:23.9770394Z outputs = layer_module( 2025-12-04T09:01:23.9770762Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:01:23.9771299Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:01:23.9771845Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:01:23.9772395Z return forward_fn(*input_tensors) 2025-12-04T09:01:23.9772794Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:01:23.9773229Z output_x = self.ff(output_x) 2025-12-04T09:01:23.9773640Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-12-04T09:01:23.9774073Z output = self.layer_2(output) 2025-12-04T09:01:23.9774209Z 2025-12-04T09:01:23.9774334Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:23.9774747Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:23.9775121Z res = mod(**inputs) 2025-12-04T09:01:23.9775553Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:23.9775981Z transformer_outputs = self.transformer( 2025-12-04T09:01:23.9776390Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:23.9776808Z outputs = layer_module( 2025-12-04T09:01:23.9777191Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:23.9777601Z outputs = self.rel_attn( 2025-12-04T09:01:23.9778004Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-12-04T09:01:23.9778459Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-12-04T09:01:23.9778623Z 2025-12-04T09:01:23.9778764Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:23.9779151Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:23.9779503Z res = mod(**inputs) 2025-12-04T09:01:23.9779897Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:23.9780330Z transformer_outputs = self.transformer( 2025-12-04T09:01:23.9780732Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:23.9781132Z outputs = layer_module( 2025-12-04T09:01:23.9781512Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:23.9781906Z outputs = self.rel_attn( 2025-12-04T09:01:23.9782313Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-12-04T09:01:23.9782744Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-12-04T09:01:23.9782906Z 2025-12-04T09:01:23.9783020Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:23.9783382Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:23.9783718Z res = mod(**inputs) 2025-12-04T09:01:23.9784097Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:23.9784511Z transformer_outputs = self.transformer( 2025-12-04T09:01:23.9784918Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:23.9785322Z outputs = layer_module( 2025-12-04T09:01:23.9785707Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:23.9786106Z outputs = self.rel_attn( 2025-12-04T09:01:23.9786499Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:01:23.9786910Z attn_vec = self.rel_attn_core( 2025-12-04T09:01:23.9787351Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-12-04T09:01:23.9787858Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-12-04T09:01:23.9788070Z 2025-12-04T09:01:23.9788183Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:23.9788577Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:23.9788924Z res = mod(**inputs) 2025-12-04T09:01:23.9789313Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:23.9789755Z transformer_outputs = self.transformer( 2025-12-04T09:01:23.9790189Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:23.9790641Z outputs = layer_module( 2025-12-04T09:01:23.9791036Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:23.9791447Z outputs = self.rel_attn( 2025-12-04T09:01:23.9791827Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-12-04T09:01:23.9792281Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-12-04T09:01:23.9792479Z 2025-12-04T09:01:23.9792586Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:23.9792977Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:23.9793324Z res = mod(**inputs) 2025-12-04T09:01:23.9793736Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:23.9794184Z transformer_outputs = self.transformer( 2025-12-04T09:01:23.9794631Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:23.9795052Z outputs = layer_module( 2025-12-04T09:01:23.9795462Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:23.9795889Z outputs = self.rel_attn( 2025-12-04T09:01:23.9796300Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:01:23.9796725Z attn_vec = self.rel_attn_core( 2025-12-04T09:01:23.9797191Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-12-04T09:01:23.9797712Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-12-04T09:01:23.9797918Z 2025-12-04T09:01:23.9798034Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:23.9798503Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:23.9798871Z res = mod(**inputs) 2025-12-04T09:01:23.9799287Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:23.9799735Z transformer_outputs = self.transformer( 2025-12-04T09:01:23.9800161Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:23.9800560Z outputs = layer_module( 2025-12-04T09:01:23.9800942Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:23.9801336Z outputs = self.rel_attn( 2025-12-04T09:01:23.9801721Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-12-04T09:01:23.9802153Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-12-04T09:01:23.9802313Z 2025-12-04T09:01:23.9802421Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:23.9802799Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:23.9803317Z res = mod(**inputs) 2025-12-04T09:01:23.9803722Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:23.9804153Z transformer_outputs = self.transformer( 2025-12-04T09:01:23.9804590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:23.9805018Z outputs = layer_module( 2025-12-04T09:01:23.9805414Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:23.9805854Z outputs = self.rel_attn( 2025-12-04T09:01:23.9806261Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:01:23.9806687Z attn_vec = self.rel_attn_core( 2025-12-04T09:01:23.9807151Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-12-04T09:01:23.9807665Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-12-04T09:01:23.9807861Z 2025-12-04T09:01:23.9807986Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:23.9808383Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:23.9808732Z res = mod(**inputs) 2025-12-04T09:01:23.9809149Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:23.9809586Z transformer_outputs = self.transformer( 2025-12-04T09:01:23.9810015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:23.9810435Z outputs = layer_module( 2025-12-04T09:01:23.9810838Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:23.9811260Z outputs = self.rel_attn( 2025-12-04T09:01:23.9811654Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:01:23.9812097Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:01:23.9812581Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:01:23.9813072Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:01:23.9813254Z 2025-12-04T09:01:23.9813367Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:23.9813758Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:23.9814111Z res = mod(**inputs) 2025-12-04T09:01:23.9814494Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:23.9814931Z transformer_outputs = self.transformer( 2025-12-04T09:01:23.9815369Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:23.9815788Z outputs = layer_module( 2025-12-04T09:01:23.9816181Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:23.9816604Z outputs = self.rel_attn( 2025-12-04T09:01:23.9817009Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:01:23.9817444Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:01:23.9817909Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:01:23.9818404Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:01:23.9818584Z 2025-12-04T09:01:23.9818704Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:23.9819088Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:23.9819442Z res = mod(**inputs) 2025-12-04T09:01:23.9819836Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:23.9820285Z transformer_outputs = self.transformer( 2025-12-04T09:01:23.9823397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:23.9823843Z outputs = layer_module( 2025-12-04T09:01:23.9824230Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:01:23.9824783Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:01:23.9825376Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:01:23.9825792Z return forward_fn(*input_tensors) 2025-12-04T09:01:23.9826193Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:01:23.9826608Z output_x = self.ff(output_x) 2025-12-04T09:01:23.9827032Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-12-04T09:01:23.9827462Z output = self.layer_1(output) 2025-12-04T09:01:23.9827600Z 2025-12-04T09:01:23.9827714Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:23.9828111Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:23.9828465Z res = mod(**inputs) 2025-12-04T09:01:23.9828855Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:23.9829315Z transformer_outputs = self.transformer( 2025-12-04T09:01:23.9829757Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:23.9830181Z outputs = layer_module( 2025-12-04T09:01:23.9830612Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:01:23.9831183Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:01:23.9831753Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:01:23.9832156Z return forward_fn(*input_tensors) 2025-12-04T09:01:23.9832558Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:01:23.9832964Z output_x = self.ff(output_x) 2025-12-04T09:01:23.9833366Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-12-04T09:01:23.9833778Z output = self.activation_function(output) 2025-12-04T09:01:23.9834152Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:01:23.9834518Z return self.act(input) 2025-12-04T09:01:23.9834639Z 2025-12-04T09:01:23.9834758Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:23.9835147Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:23.9835497Z res = mod(**inputs) 2025-12-04T09:01:23.9835895Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:23.9836326Z transformer_outputs = self.transformer( 2025-12-04T09:01:23.9836761Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:23.9837198Z outputs = layer_module( 2025-12-04T09:01:23.9837600Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:01:23.9838162Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:01:23.9838905Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:01:23.9839362Z return forward_fn(*input_tensors) 2025-12-04T09:01:23.9839804Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:01:23.9840246Z output_x = self.ff(output_x) 2025-12-04T09:01:23.9840666Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-12-04T09:01:23.9841092Z output = self.layer_2(output) 2025-12-04T09:01:23.9841228Z 2025-12-04T09:01:23.9841343Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:23.9841739Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:23.9842094Z res = mod(**inputs) 2025-12-04T09:01:23.9842505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:23.9842947Z transformer_outputs = self.transformer( 2025-12-04T09:01:23.9843385Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:23.9843808Z outputs = layer_module( 2025-12-04T09:01:23.9844205Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:23.9844632Z outputs = self.rel_attn( 2025-12-04T09:01:23.9845042Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-12-04T09:01:23.9845493Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-12-04T09:01:23.9845700Z 2025-12-04T09:01:23.9845814Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:23.9846207Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:23.9846560Z res = mod(**inputs) 2025-12-04T09:01:23.9846954Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:23.9847386Z transformer_outputs = self.transformer( 2025-12-04T09:01:23.9847822Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:23.9848248Z outputs = layer_module( 2025-12-04T09:01:23.9848645Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:23.9849068Z outputs = self.rel_attn( 2025-12-04T09:01:23.9849472Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-12-04T09:01:23.9849936Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-12-04T09:01:23.9850103Z 2025-12-04T09:01:23.9850217Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:23.9850616Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:23.9850944Z res = mod(**inputs) 2025-12-04T09:01:23.9851306Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:23.9851717Z transformer_outputs = self.transformer( 2025-12-04T09:01:23.9852128Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:23.9852528Z outputs = layer_module( 2025-12-04T09:01:23.9852902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:23.9853300Z outputs = self.rel_attn( 2025-12-04T09:01:23.9853689Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:01:23.9854138Z attn_vec = self.rel_attn_core( 2025-12-04T09:01:23.9854551Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-12-04T09:01:23.9855056Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-12-04T09:01:23.9855269Z 2025-12-04T09:01:23.9855385Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:23.9855752Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:23.9856084Z res = mod(**inputs) 2025-12-04T09:01:23.9856460Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:23.9856879Z transformer_outputs = self.transformer( 2025-12-04T09:01:23.9857304Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:23.9857705Z outputs = layer_module( 2025-12-04T09:01:23.9858090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:23.9858480Z outputs = self.rel_attn( 2025-12-04T09:01:23.9858864Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-12-04T09:01:23.9859327Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-12-04T09:01:23.9859519Z 2025-12-04T09:01:23.9859634Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:23.9860000Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:23.9860364Z res = mod(**inputs) 2025-12-04T09:01:23.9860740Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:23.9861155Z transformer_outputs = self.transformer( 2025-12-04T09:01:23.9861561Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:23.9861960Z outputs = layer_module( 2025-12-04T09:01:23.9862340Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:23.9862730Z outputs = self.rel_attn( 2025-12-04T09:01:23.9863114Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:01:23.9863518Z attn_vec = self.rel_attn_core( 2025-12-04T09:01:23.9863941Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-12-04T09:01:23.9864415Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-12-04T09:01:23.9864610Z 2025-12-04T09:01:23.9864718Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:23.9865094Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:23.9865428Z res = mod(**inputs) 2025-12-04T09:01:23.9865788Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:23.9866195Z transformer_outputs = self.transformer( 2025-12-04T09:01:23.9866593Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:23.9866972Z outputs = layer_module( 2025-12-04T09:01:23.9867341Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:23.9867727Z outputs = self.rel_attn( 2025-12-04T09:01:23.9868101Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-12-04T09:01:23.9868537Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-12-04T09:01:23.9868701Z 2025-12-04T09:01:23.9868805Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:23.9869167Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:23.9869497Z res = mod(**inputs) 2025-12-04T09:01:23.9869862Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:23.9870281Z transformer_outputs = self.transformer( 2025-12-04T09:01:23.9870691Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:23.9871083Z outputs = layer_module( 2025-12-04T09:01:23.9871479Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:23.9871880Z outputs = self.rel_attn( 2025-12-04T09:01:23.9872280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:01:23.9872668Z attn_vec = self.rel_attn_core( 2025-12-04T09:01:23.9873075Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-12-04T09:01:23.9873538Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-12-04T09:01:23.9873714Z 2025-12-04T09:01:23.9873820Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:23.9874179Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:23.9874521Z res = mod(**inputs) 2025-12-04T09:01:23.9874887Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:23.9875284Z transformer_outputs = self.transformer( 2025-12-04T09:01:23.9875684Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:23.9876075Z outputs = layer_module( 2025-12-04T09:01:23.9876449Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:23.9876851Z outputs = self.rel_attn( 2025-12-04T09:01:23.9877235Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:01:23.9877656Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:01:23.9878084Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:01:23.9878643Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:01:23.9878842Z 2025-12-04T09:01:23.9878960Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:23.9879372Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:23.9879730Z res = mod(**inputs) 2025-12-04T09:01:23.9880141Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:23.9880594Z transformer_outputs = self.transformer( 2025-12-04T09:01:23.9881054Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:23.9881447Z outputs = layer_module( 2025-12-04T09:01:23.9881825Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:23.9882227Z outputs = self.rel_attn( 2025-12-04T09:01:23.9882603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:01:23.9883050Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:01:23.9883493Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:01:23.9883956Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:01:23.9884142Z 2025-12-04T09:01:23.9884248Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:23.9884621Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:23.9884957Z res = mod(**inputs) 2025-12-04T09:01:23.9885321Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:23.9885736Z transformer_outputs = self.transformer( 2025-12-04T09:01:23.9886164Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:23.9886564Z outputs = layer_module( 2025-12-04T09:01:23.9886951Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:01:23.9887494Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:01:23.9888047Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:01:23.9888459Z return forward_fn(*input_tensors) 2025-12-04T09:01:23.9888856Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:01:23.9889264Z output_x = self.ff(output_x) 2025-12-04T09:01:23.9889676Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-12-04T09:01:23.9890078Z output = self.layer_1(output) 2025-12-04T09:01:23.9890215Z 2025-12-04T09:01:23.9890324Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:23.9890705Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:23.9891036Z res = mod(**inputs) 2025-12-04T09:01:23.9891422Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:23.9891835Z transformer_outputs = self.transformer( 2025-12-04T09:01:23.9892244Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:23.9892640Z outputs = layer_module( 2025-12-04T09:01:23.9893022Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:01:23.9893562Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:01:23.9894121Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:01:23.9894524Z return forward_fn(*input_tensors) 2025-12-04T09:01:23.9894925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:01:23.9895329Z output_x = self.ff(output_x) 2025-12-04T09:01:23.9895719Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-12-04T09:01:23.9896132Z output = self.activation_function(output) 2025-12-04T09:01:23.9896505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:01:23.9896866Z return self.act(input) 2025-12-04T09:01:23.9896981Z 2025-12-04T09:01:23.9897088Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:23.9897482Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:23.9897819Z res = mod(**inputs) 2025-12-04T09:01:23.9898201Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:23.9898626Z transformer_outputs = self.transformer( 2025-12-04T09:01:23.9899028Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:23.9899417Z outputs = layer_module( 2025-12-04T09:01:23.9899789Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:01:23.9900305Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:01:23.9900865Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:01:23.9901270Z return forward_fn(*input_tensors) 2025-12-04T09:01:23.9901658Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:01:23.9902060Z output_x = self.ff(output_x) 2025-12-04T09:01:23.9902459Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-12-04T09:01:23.9902863Z output = self.layer_2(output) 2025-12-04T09:01:23.9902989Z 2025-12-04T09:01:23.9903098Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:23.9903480Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:23.9903838Z res = mod(**inputs) 2025-12-04T09:01:23.9904214Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:23.9904626Z transformer_outputs = self.transformer( 2025-12-04T09:01:23.9905048Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:23.9905437Z outputs = layer_module( 2025-12-04T09:01:23.9905802Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:23.9906194Z outputs = self.rel_attn( 2025-12-04T09:01:23.9906569Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-12-04T09:01:23.9906986Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-12-04T09:01:23.9907138Z 2025-12-04T09:01:23.9907243Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:23.9907612Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:23.9907940Z res = mod(**inputs) 2025-12-04T09:01:23.9908306Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:23.9908719Z transformer_outputs = self.transformer( 2025-12-04T09:01:23.9909130Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:23.9909531Z outputs = layer_module( 2025-12-04T09:01:23.9909908Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:23.9910312Z outputs = self.rel_attn( 2025-12-04T09:01:23.9910703Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-12-04T09:01:23.9911131Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-12-04T09:01:23.9911294Z 2025-12-04T09:01:23.9911404Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:23.9911811Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:23.9912135Z res = mod(**inputs) 2025-12-04T09:01:23.9912494Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:23.9912964Z transformer_outputs = self.transformer( 2025-12-04T09:01:23.9913362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:23.9913750Z outputs = layer_module( 2025-12-04T09:01:23.9914118Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:23.9914511Z outputs = self.rel_attn( 2025-12-04T09:01:23.9914902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:01:23.9915293Z attn_vec = self.rel_attn_core( 2025-12-04T09:01:23.9915705Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-12-04T09:01:23.9916178Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-12-04T09:01:23.9916366Z 2025-12-04T09:01:23.9916480Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:23.9916836Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:23.9917162Z res = mod(**inputs) 2025-12-04T09:01:23.9917527Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:23.9917933Z transformer_outputs = self.transformer( 2025-12-04T09:01:23.9918470Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:23.9918922Z outputs = layer_module( 2025-12-04T09:01:23.9919364Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:23.9919781Z outputs = self.rel_attn( 2025-12-04T09:01:23.9920169Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-12-04T09:01:23.9920612Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-12-04T09:01:23.9920997Z 2025-12-04T09:01:23.9921113Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:23.9921464Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:23.9921787Z res = mod(**inputs) 2025-12-04T09:01:23.9922150Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:23.9922544Z transformer_outputs = self.transformer( 2025-12-04T09:01:23.9922937Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:23.9923321Z outputs = layer_module( 2025-12-04T09:01:23.9923684Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:23.9924060Z outputs = self.rel_attn( 2025-12-04T09:01:23.9924422Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:01:23.9924802Z attn_vec = self.rel_attn_core( 2025-12-04T09:01:23.9925190Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-12-04T09:01:23.9925635Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-12-04T09:01:23.9925819Z 2025-12-04T09:01:23.9925921Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:23.9926327Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:23.9926634Z res = mod(**inputs) 2025-12-04T09:01:23.9926989Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:23.9927411Z transformer_outputs = self.transformer( 2025-12-04T09:01:23.9927811Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:23.9928244Z outputs = layer_module( 2025-12-04T09:01:23.9928615Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:23.9929003Z outputs = self.rel_attn( 2025-12-04T09:01:23.9929397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-12-04T09:01:23.9929804Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-12-04T09:01:23.9929963Z 2025-12-04T09:01:23.9930067Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:23.9930427Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:23.9930737Z res = mod(**inputs) 2025-12-04T09:01:23.9931096Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:23.9931492Z transformer_outputs = self.transformer( 2025-12-04T09:01:23.9931885Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:23.9932258Z outputs = layer_module( 2025-12-04T09:01:23.9932646Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:23.9933027Z outputs = self.rel_attn( 2025-12-04T09:01:23.9933385Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:01:23.9933766Z attn_vec = self.rel_attn_core( 2025-12-04T09:01:23.9934165Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-12-04T09:01:23.9934626Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-12-04T09:01:23.9934800Z 2025-12-04T09:01:23.9934904Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:23.9935261Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:23.9935582Z res = mod(**inputs) 2025-12-04T09:01:23.9935942Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:23.9936340Z transformer_outputs = self.transformer( 2025-12-04T09:01:23.9936744Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:23.9937122Z outputs = layer_module( 2025-12-04T09:01:23.9937474Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:23.9937859Z outputs = self.rel_attn( 2025-12-04T09:01:23.9938234Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:01:23.9938653Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:01:23.9939085Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:01:23.9939547Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:01:23.9939726Z 2025-12-04T09:01:23.9939839Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:23.9940219Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:23.9940540Z res = mod(**inputs) 2025-12-04T09:01:23.9940904Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:23.9941354Z transformer_outputs = self.transformer( 2025-12-04T09:01:23.9941754Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:23.9942152Z outputs = layer_module( 2025-12-04T09:01:23.9942532Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:23.9942930Z outputs = self.rel_attn( 2025-12-04T09:01:23.9943324Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:01:23.9943744Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:01:23.9944189Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:01:23.9944649Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:01:23.9944836Z 2025-12-04T09:01:23.9944940Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:23.9945307Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:23.9945636Z res = mod(**inputs) 2025-12-04T09:01:23.9945997Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:23.9946402Z transformer_outputs = self.transformer( 2025-12-04T09:01:23.9946827Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:23.9947219Z outputs = layer_module( 2025-12-04T09:01:23.9947588Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:01:23.9948119Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:01:23.9948656Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:01:23.9949061Z return forward_fn(*input_tensors) 2025-12-04T09:01:23.9949461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:01:23.9949862Z output_x = self.ff(output_x) 2025-12-04T09:01:23.9950246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-12-04T09:01:23.9950640Z output = self.layer_1(output) 2025-12-04T09:01:23.9950774Z 2025-12-04T09:01:23.9950882Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:23.9951251Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:23.9951582Z res = mod(**inputs) 2025-12-04T09:01:23.9951944Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:23.9952353Z transformer_outputs = self.transformer( 2025-12-04T09:01:23.9952756Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:23.9953141Z outputs = layer_module( 2025-12-04T09:01:23.9953523Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:01:23.9954039Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:01:23.9954577Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:01:23.9954961Z return forward_fn(*input_tensors) 2025-12-04T09:01:23.9955343Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:01:23.9955758Z output_x = self.ff(output_x) 2025-12-04T09:01:23.9956133Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-12-04T09:01:23.9956525Z output = self.activation_function(output) 2025-12-04T09:01:23.9956880Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:01:23.9957228Z return self.act(input) 2025-12-04T09:01:23.9957341Z 2025-12-04T09:01:23.9957447Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:23.9957831Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:23.9958151Z res = mod(**inputs) 2025-12-04T09:01:23.9958583Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:23.9959037Z transformer_outputs = self.transformer( 2025-12-04T09:01:23.9959509Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:23.9959948Z outputs = layer_module( 2025-12-04T09:01:23.9960351Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:01:23.9960888Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:01:23.9961427Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:01:23.9961818Z return forward_fn(*input_tensors) 2025-12-04T09:01:23.9962192Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:01:23.9962574Z output_x = self.ff(output_x) 2025-12-04T09:01:23.9962949Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-12-04T09:01:23.9963329Z output = self.layer_2(output) 2025-12-04T09:01:23.9963449Z 2025-12-04T09:01:23.9963549Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:23.9963902Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:23.9964224Z res = mod(**inputs) 2025-12-04T09:01:23.9964564Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:23.9964950Z transformer_outputs = self.transformer( 2025-12-04T09:01:23.9965333Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:23.9965713Z outputs = layer_module( 2025-12-04T09:01:23.9966068Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:23.9966458Z outputs = self.rel_attn( 2025-12-04T09:01:23.9966813Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-12-04T09:01:23.9967209Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-12-04T09:01:23.9967356Z 2025-12-04T09:01:23.9967454Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:23.9967798Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:23.9968107Z res = mod(**inputs) 2025-12-04T09:01:23.9968446Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:23.9968847Z transformer_outputs = self.transformer( 2025-12-04T09:01:23.9969236Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:23.9969625Z outputs = layer_module( 2025-12-04T09:01:23.9969974Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:23.9970351Z outputs = self.rel_attn( 2025-12-04T09:01:23.9970709Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-12-04T09:01:23.9971102Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-12-04T09:01:23.9971260Z 2025-12-04T09:01:23.9971360Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:23.9971720Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:23.9972031Z res = mod(**inputs) 2025-12-04T09:01:23.9972375Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:23.9972762Z transformer_outputs = self.transformer( 2025-12-04T09:01:23.9973150Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:23.9973523Z outputs = layer_module( 2025-12-04T09:01:23.9973874Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:23.9974248Z outputs = self.rel_attn( 2025-12-04T09:01:23.9974612Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:01:23.9974999Z attn_vec = self.rel_attn_core( 2025-12-04T09:01:23.9975386Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-12-04T09:01:23.9975830Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-12-04T09:01:23.9976006Z 2025-12-04T09:01:23.9976111Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:23.9976449Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:23.9976755Z res = mod(**inputs) 2025-12-04T09:01:23.9977097Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:23.9977480Z transformer_outputs = self.transformer( 2025-12-04T09:01:23.9977866Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:23.9978248Z outputs = layer_module( 2025-12-04T09:01:23.9978611Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:23.9978983Z outputs = self.rel_attn( 2025-12-04T09:01:23.9979351Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-12-04T09:01:23.9979788Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-12-04T09:01:23.9979969Z 2025-12-04T09:01:23.9980075Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:23.9980418Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:23.9980729Z res = mod(**inputs) 2025-12-04T09:01:23.9980981Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:23.9981070Z transformer_outputs = self.transformer( 2025-12-04T09:01:23.9981322Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:23.9981415Z outputs = layer_module( 2025-12-04T09:01:23.9981666Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:23.9981733Z outputs = self.rel_attn( 2025-12-04T09:01:23.9982006Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:01:23.9982081Z attn_vec = self.rel_attn_core( 2025-12-04T09:01:23.9982348Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-12-04T09:01:23.9982482Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-12-04T09:01:23.9982487Z 2025-12-04T09:01:23.9982589Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:23.9982807Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:23.9982873Z res = mod(**inputs) 2025-12-04T09:01:23.9983125Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:23.9983213Z transformer_outputs = self.transformer( 2025-12-04T09:01:23.9983490Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:23.9983565Z outputs = layer_module( 2025-12-04T09:01:23.9983820Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:23.9983888Z outputs = self.rel_attn( 2025-12-04T09:01:23.9984149Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-12-04T09:01:23.9984270Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-12-04T09:01:23.9984274Z 2025-12-04T09:01:23.9984378Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:23.9984586Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:23.9984650Z res = mod(**inputs) 2025-12-04T09:01:23.9984914Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:23.9984997Z transformer_outputs = self.transformer( 2025-12-04T09:01:23.9985253Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:23.9985328Z outputs = layer_module( 2025-12-04T09:01:23.9985582Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:23.9985651Z outputs = self.rel_attn( 2025-12-04T09:01:23.9985919Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:01:23.9985992Z attn_vec = self.rel_attn_core( 2025-12-04T09:01:23.9986270Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-12-04T09:01:23.9986395Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-12-04T09:01:23.9986400Z 2025-12-04T09:01:23.9986499Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:23.9986706Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:23.9986769Z res = mod(**inputs) 2025-12-04T09:01:23.9987031Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:23.9987113Z transformer_outputs = self.transformer( 2025-12-04T09:01:23.9987385Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:23.9987474Z outputs = layer_module( 2025-12-04T09:01:23.9987739Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:23.9987807Z outputs = self.rel_attn( 2025-12-04T09:01:23.9988114Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:01:23.9988204Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:01:23.9988495Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:01:23.9988608Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:01:23.9988613Z 2025-12-04T09:01:23.9988714Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:23.9988955Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:23.9989020Z res = mod(**inputs) 2025-12-04T09:01:23.9989296Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:23.9989378Z transformer_outputs = self.transformer( 2025-12-04T09:01:23.9989653Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:23.9989731Z outputs = layer_module( 2025-12-04T09:01:23.9989994Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:23.9990062Z outputs = self.rel_attn( 2025-12-04T09:01:23.9990334Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:01:23.9990441Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:01:23.9990726Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:01:23.9990847Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:01:23.9990851Z 2025-12-04T09:01:23.9990950Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:23.9991161Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:23.9991223Z res = mod(**inputs) 2025-12-04T09:01:23.9991484Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:23.9991565Z transformer_outputs = self.transformer( 2025-12-04T09:01:23.9991860Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:23.9991935Z outputs = layer_module( 2025-12-04T09:01:23.9992182Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:01:23.9992384Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:01:23.9992654Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:01:23.9992734Z return forward_fn(*input_tensors) 2025-12-04T09:01:23.9992997Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:01:23.9993069Z output_x = self.ff(output_x) 2025-12-04T09:01:23.9993321Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-12-04T09:01:23.9993403Z output = self.layer_1(output) 2025-12-04T09:01:23.9993406Z 2025-12-04T09:01:23.9993511Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:23.9993738Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:23.9993802Z res = mod(**inputs) 2025-12-04T09:01:23.9994067Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:23.9994170Z transformer_outputs = self.transformer( 2025-12-04T09:01:23.9994415Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:23.9994479Z outputs = layer_module( 2025-12-04T09:01:23.9994734Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:01:23.9994934Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:01:23.9995225Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:01:23.9995308Z return forward_fn(*input_tensors) 2025-12-04T09:01:23.9995569Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:01:23.9995651Z output_x = self.ff(output_x) 2025-12-04T09:01:23.9995916Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-12-04T09:01:23.9996013Z output = self.activation_function(output) 2025-12-04T09:01:23.9996234Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:01:23.9996306Z return self.act(input) 2025-12-04T09:01:23.9996310Z 2025-12-04T09:01:23.9996442Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:23.9996646Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:23.9996712Z res = mod(**inputs) 2025-12-04T09:01:23.9996985Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:23.9997069Z transformer_outputs = self.transformer( 2025-12-04T09:01:23.9997338Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:23.9997410Z outputs = layer_module( 2025-12-04T09:01:23.9997673Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:01:23.9997892Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:01:23.9998229Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:01:23.9998334Z return forward_fn(*input_tensors) 2025-12-04T09:01:23.9998609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:01:23.9998690Z output_x = self.ff(output_x) 2025-12-04T09:01:23.9998991Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-12-04T09:01:23.9999075Z output = self.layer_2(output) 2025-12-04T09:01:23.9999080Z 2025-12-04T09:01:23.9999196Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:23.9999431Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:23.9999502Z res = mod(**inputs) 2025-12-04T09:01:23.9999817Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:23.9999922Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0000206Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0000312Z outputs = layer_module( 2025-12-04T09:01:24.0000602Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0000683Z outputs = self.rel_attn( 2025-12-04T09:01:24.0000972Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-12-04T09:01:24.0001076Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-12-04T09:01:24.0001080Z 2025-12-04T09:01:24.0001196Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0001409Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0001478Z res = mod(**inputs) 2025-12-04T09:01:24.0001774Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0001861Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0002144Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0002214Z outputs = layer_module( 2025-12-04T09:01:24.0002485Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0002568Z outputs = self.rel_attn( 2025-12-04T09:01:24.0002839Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-12-04T09:01:24.0002946Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-12-04T09:01:24.0002956Z 2025-12-04T09:01:24.0003081Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0003291Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0003364Z res = mod(**inputs) 2025-12-04T09:01:24.0003640Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0003726Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0004005Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0004077Z outputs = layer_module( 2025-12-04T09:01:24.0004355Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0004427Z outputs = self.rel_attn( 2025-12-04T09:01:24.0004701Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:01:24.0004786Z attn_vec = self.rel_attn_core( 2025-12-04T09:01:24.0005077Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-12-04T09:01:24.0005221Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-12-04T09:01:24.0005232Z 2025-12-04T09:01:24.0005343Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0005557Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0005633Z res = mod(**inputs) 2025-12-04T09:01:24.0005912Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0006000Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0006289Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0006359Z outputs = layer_module( 2025-12-04T09:01:24.0006627Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0006697Z outputs = self.rel_attn( 2025-12-04T09:01:24.0006980Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-12-04T09:01:24.0007124Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-12-04T09:01:24.0007143Z 2025-12-04T09:01:24.0007248Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0007450Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0007526Z res = mod(**inputs) 2025-12-04T09:01:24.0007791Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0007885Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0008166Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0008240Z outputs = layer_module( 2025-12-04T09:01:24.0008532Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0008607Z outputs = self.rel_attn( 2025-12-04T09:01:24.0008894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:01:24.0008976Z attn_vec = self.rel_attn_core( 2025-12-04T09:01:24.0009273Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-12-04T09:01:24.0009429Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-12-04T09:01:24.0009433Z 2025-12-04T09:01:24.0009552Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0009754Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0009826Z res = mod(**inputs) 2025-12-04T09:01:24.0010088Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0010180Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0010443Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0010512Z outputs = layer_module( 2025-12-04T09:01:24.0010778Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0010845Z outputs = self.rel_attn( 2025-12-04T09:01:24.0011103Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-12-04T09:01:24.0011213Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-12-04T09:01:24.0011217Z 2025-12-04T09:01:24.0011320Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0011532Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0011596Z res = mod(**inputs) 2025-12-04T09:01:24.0011858Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0011950Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0012211Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0012286Z outputs = layer_module( 2025-12-04T09:01:24.0012544Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0012614Z outputs = self.rel_attn( 2025-12-04T09:01:24.0012884Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:01:24.0012956Z attn_vec = self.rel_attn_core( 2025-12-04T09:01:24.0013250Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-12-04T09:01:24.0013386Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-12-04T09:01:24.0013404Z 2025-12-04T09:01:24.0013509Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0013720Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0013787Z res = mod(**inputs) 2025-12-04T09:01:24.0014055Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0014151Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0014434Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0014511Z outputs = layer_module( 2025-12-04T09:01:24.0014773Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0014843Z outputs = self.rel_attn( 2025-12-04T09:01:24.0015113Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:01:24.0015207Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:01:24.0015494Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:01:24.0015622Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:01:24.0015625Z 2025-12-04T09:01:24.0015729Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0015965Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0016030Z res = mod(**inputs) 2025-12-04T09:01:24.0016293Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0016385Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0016648Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0016725Z outputs = layer_module( 2025-12-04T09:01:24.0016986Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0017056Z outputs = self.rel_attn( 2025-12-04T09:01:24.0017322Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:01:24.0017415Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:01:24.0017698Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:01:24.0017821Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:01:24.0017824Z 2025-12-04T09:01:24.0017927Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0018153Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0018225Z res = mod(**inputs) 2025-12-04T09:01:24.0018505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0018611Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0018873Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0018949Z outputs = layer_module( 2025-12-04T09:01:24.0019210Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:01:24.0019463Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:01:24.0019746Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:01:24.0019829Z return forward_fn(*input_tensors) 2025-12-04T09:01:24.0020124Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:01:24.0020210Z output_x = self.ff(output_x) 2025-12-04T09:01:24.0020486Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-12-04T09:01:24.0020572Z output = self.layer_1(output) 2025-12-04T09:01:24.0020578Z 2025-12-04T09:01:24.0020859Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0021138Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0021218Z res = mod(**inputs) 2025-12-04T09:01:24.0021498Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0021596Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0021872Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0021948Z outputs = layer_module( 2025-12-04T09:01:24.0022232Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:01:24.0022460Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:01:24.0022776Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:01:24.0022868Z return forward_fn(*input_tensors) 2025-12-04T09:01:24.0023148Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:01:24.0023237Z output_x = self.ff(output_x) 2025-12-04T09:01:24.0023516Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-12-04T09:01:24.0023613Z output = self.activation_function(output) 2025-12-04T09:01:24.0023857Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:01:24.0023928Z return self.act(input) 2025-12-04T09:01:24.0023932Z 2025-12-04T09:01:24.0024046Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0024253Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0024321Z res = mod(**inputs) 2025-12-04T09:01:24.0024594Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0024676Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0024939Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0025019Z outputs = layer_module( 2025-12-04T09:01:24.0025296Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:01:24.0025524Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:01:24.0025806Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:01:24.0025889Z return forward_fn(*input_tensors) 2025-12-04T09:01:24.0026180Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:01:24.0026287Z output_x = self.ff(output_x) 2025-12-04T09:01:24.0026581Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-12-04T09:01:24.0026659Z output = self.layer_2(output) 2025-12-04T09:01:24.0026696Z 2025-12-04T09:01:24.0026809Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0027048Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0027117Z res = mod(**inputs) 2025-12-04T09:01:24.0027415Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0027512Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0027827Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0027908Z outputs = layer_module( 2025-12-04T09:01:24.0028190Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0028276Z outputs = self.rel_attn( 2025-12-04T09:01:24.0028551Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-12-04T09:01:24.0028653Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-12-04T09:01:24.0028657Z 2025-12-04T09:01:24.0028768Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0028974Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0029039Z res = mod(**inputs) 2025-12-04T09:01:24.0029329Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0029416Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0029683Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0029760Z outputs = layer_module( 2025-12-04T09:01:24.0030053Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0030135Z outputs = self.rel_attn( 2025-12-04T09:01:24.0030409Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-12-04T09:01:24.0030519Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-12-04T09:01:24.0030523Z 2025-12-04T09:01:24.0030642Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0030859Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0030929Z res = mod(**inputs) 2025-12-04T09:01:24.0031217Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0031306Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0031593Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0031667Z outputs = layer_module( 2025-12-04T09:01:24.0031944Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0032027Z outputs = self.rel_attn( 2025-12-04T09:01:24.0032306Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:01:24.0032394Z attn_vec = self.rel_attn_core( 2025-12-04T09:01:24.0032696Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-12-04T09:01:24.0032838Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-12-04T09:01:24.0032860Z 2025-12-04T09:01:24.0032981Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0033199Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0033286Z res = mod(**inputs) 2025-12-04T09:01:24.0033574Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0033663Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0033987Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0034061Z outputs = layer_module( 2025-12-04T09:01:24.0034364Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0034450Z outputs = self.rel_attn( 2025-12-04T09:01:24.0034737Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-12-04T09:01:24.0034891Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-12-04T09:01:24.0034895Z 2025-12-04T09:01:24.0035008Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0035240Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0035319Z res = mod(**inputs) 2025-12-04T09:01:24.0035610Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0035701Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0036018Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0036095Z outputs = layer_module( 2025-12-04T09:01:24.0036389Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0036465Z outputs = self.rel_attn( 2025-12-04T09:01:24.0036746Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:01:24.0036836Z attn_vec = self.rel_attn_core( 2025-12-04T09:01:24.0037139Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-12-04T09:01:24.0037291Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-12-04T09:01:24.0037294Z 2025-12-04T09:01:24.0037408Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0037630Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0037709Z res = mod(**inputs) 2025-12-04T09:01:24.0038006Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0038099Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0038459Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0038544Z outputs = layer_module( 2025-12-04T09:01:24.0038844Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0038919Z outputs = self.rel_attn( 2025-12-04T09:01:24.0039205Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-12-04T09:01:24.0039326Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-12-04T09:01:24.0039332Z 2025-12-04T09:01:24.0039449Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0039686Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0039780Z res = mod(**inputs) 2025-12-04T09:01:24.0040074Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0040166Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0040451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0040523Z outputs = layer_module( 2025-12-04T09:01:24.0040791Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0040860Z outputs = self.rel_attn( 2025-12-04T09:01:24.0041131Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:01:24.0041230Z attn_vec = self.rel_attn_core( 2025-12-04T09:01:24.0041512Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-12-04T09:01:24.0041649Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-12-04T09:01:24.0041653Z 2025-12-04T09:01:24.0041759Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0041977Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0042042Z res = mod(**inputs) 2025-12-04T09:01:24.0042307Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0042397Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0042660Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0042749Z outputs = layer_module( 2025-12-04T09:01:24.0043028Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0043098Z outputs = self.rel_attn( 2025-12-04T09:01:24.0043375Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:01:24.0043468Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:01:24.0043749Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:01:24.0043873Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:01:24.0043876Z 2025-12-04T09:01:24.0043982Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0044185Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0044259Z res = mod(**inputs) 2025-12-04T09:01:24.0044523Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0044615Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0044879Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0044950Z outputs = layer_module( 2025-12-04T09:01:24.0045217Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0045288Z outputs = self.rel_attn( 2025-12-04T09:01:24.0045554Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:01:24.0045645Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:01:24.0045930Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:01:24.0046049Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:01:24.0046053Z 2025-12-04T09:01:24.0046173Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0046377Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0046450Z res = mod(**inputs) 2025-12-04T09:01:24.0046729Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0046820Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0047082Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0047149Z outputs = layer_module( 2025-12-04T09:01:24.0047422Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:01:24.0047649Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:01:24.0047930Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:01:24.0048010Z return forward_fn(*input_tensors) 2025-12-04T09:01:24.0048275Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:01:24.0048361Z output_x = self.ff(output_x) 2025-12-04T09:01:24.0048631Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-12-04T09:01:24.0048706Z output = self.layer_1(output) 2025-12-04T09:01:24.0048717Z 2025-12-04T09:01:24.0048822Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0049036Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0049111Z res = mod(**inputs) 2025-12-04T09:01:24.0049371Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0049453Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0049717Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0049786Z outputs = layer_module( 2025-12-04T09:01:24.0050045Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:01:24.0050251Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:01:24.0050513Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:01:24.0050601Z return forward_fn(*input_tensors) 2025-12-04T09:01:24.0050855Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:01:24.0050926Z output_x = self.ff(output_x) 2025-12-04T09:01:24.0051186Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-12-04T09:01:24.0051274Z output = self.activation_function(output) 2025-12-04T09:01:24.0051495Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:01:24.0051565Z return self.act(input) 2025-12-04T09:01:24.0051568Z 2025-12-04T09:01:24.0051669Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0051873Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0051937Z res = mod(**inputs) 2025-12-04T09:01:24.0052199Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0052300Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0052554Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0052630Z outputs = layer_module( 2025-12-04T09:01:24.0052906Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:01:24.0053112Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:01:24.0053386Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:01:24.0053462Z return forward_fn(*input_tensors) 2025-12-04T09:01:24.0053745Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:01:24.0053817Z output_x = self.ff(output_x) 2025-12-04T09:01:24.0054075Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-12-04T09:01:24.0054154Z output = self.layer_2(output) 2025-12-04T09:01:24.0054158Z 2025-12-04T09:01:24.0054259Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0054467Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0054532Z res = mod(**inputs) 2025-12-04T09:01:24.0054797Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0054887Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0055168Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0055239Z outputs = layer_module( 2025-12-04T09:01:24.0055514Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0055585Z outputs = self.rel_attn( 2025-12-04T09:01:24.0055856Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-12-04T09:01:24.0055959Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-12-04T09:01:24.0055963Z 2025-12-04T09:01:24.0056068Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0056278Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0056344Z res = mod(**inputs) 2025-12-04T09:01:24.0056616Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0056702Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0056968Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0057045Z outputs = layer_module( 2025-12-04T09:01:24.0057308Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0057377Z outputs = self.rel_attn( 2025-12-04T09:01:24.0057648Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-12-04T09:01:24.0057752Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-12-04T09:01:24.0057755Z 2025-12-04T09:01:24.0057864Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0058068Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0058135Z res = mod(**inputs) 2025-12-04T09:01:24.0058418Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0058513Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0058766Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0058840Z outputs = layer_module( 2025-12-04T09:01:24.0059118Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0059194Z outputs = self.rel_attn( 2025-12-04T09:01:24.0059454Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:01:24.0059528Z attn_vec = self.rel_attn_core( 2025-12-04T09:01:24.0059819Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-12-04T09:01:24.0059969Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-12-04T09:01:24.0059973Z 2025-12-04T09:01:24.0060087Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0060298Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0060362Z res = mod(**inputs) 2025-12-04T09:01:24.0060626Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0060709Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0060960Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0061035Z outputs = layer_module( 2025-12-04T09:01:24.0061288Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0061380Z outputs = self.rel_attn( 2025-12-04T09:01:24.0061636Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-12-04T09:01:24.0061771Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-12-04T09:01:24.0061774Z 2025-12-04T09:01:24.0061884Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0062079Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0062152Z res = mod(**inputs) 2025-12-04T09:01:24.0062406Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0062487Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0062747Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0062814Z outputs = layer_module( 2025-12-04T09:01:24.0063068Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0063145Z outputs = self.rel_attn( 2025-12-04T09:01:24.0063395Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:01:24.0063475Z attn_vec = self.rel_attn_core( 2025-12-04T09:01:24.0063749Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-12-04T09:01:24.0063876Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-12-04T09:01:24.0063880Z 2025-12-04T09:01:24.0063990Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0064189Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0064262Z res = mod(**inputs) 2025-12-04T09:01:24.0064517Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0064652Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0064921Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0064987Z outputs = layer_module( 2025-12-04T09:01:24.0065270Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0065346Z outputs = self.rel_attn( 2025-12-04T09:01:24.0065602Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-12-04T09:01:24.0065709Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-12-04T09:01:24.0065712Z 2025-12-04T09:01:24.0065815Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0066027Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0066102Z res = mod(**inputs) 2025-12-04T09:01:24.0066362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0066451Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0066707Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0066775Z outputs = layer_module( 2025-12-04T09:01:24.0067034Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0067103Z outputs = self.rel_attn( 2025-12-04T09:01:24.0067358Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:01:24.0067456Z attn_vec = self.rel_attn_core( 2025-12-04T09:01:24.0067737Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-12-04T09:01:24.0067869Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-12-04T09:01:24.0067873Z 2025-12-04T09:01:24.0067974Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0068172Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0068244Z res = mod(**inputs) 2025-12-04T09:01:24.0068506Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0068588Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0068852Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0068921Z outputs = layer_module( 2025-12-04T09:01:24.0069185Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0069252Z outputs = self.rel_attn( 2025-12-04T09:01:24.0069510Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:01:24.0069607Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:01:24.0069885Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:01:24.0070003Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:01:24.0070006Z 2025-12-04T09:01:24.0070109Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0070305Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0070380Z res = mod(**inputs) 2025-12-04T09:01:24.0070639Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0070721Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0070998Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0071065Z outputs = layer_module( 2025-12-04T09:01:24.0071328Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0071411Z outputs = self.rel_attn( 2025-12-04T09:01:24.0071665Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:01:24.0071762Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:01:24.0072038Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:01:24.0072172Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:01:24.0072176Z 2025-12-04T09:01:24.0072278Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0072475Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0072545Z res = mod(**inputs) 2025-12-04T09:01:24.0072800Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0072884Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0073146Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0073213Z outputs = layer_module( 2025-12-04T09:01:24.0073471Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:01:24.0073695Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:01:24.0073962Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:01:24.0074045Z return forward_fn(*input_tensors) 2025-12-04T09:01:24.0074301Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:01:24.0074381Z output_x = self.ff(output_x) 2025-12-04T09:01:24.0074640Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-12-04T09:01:24.0074713Z output = self.layer_1(output) 2025-12-04T09:01:24.0074717Z 2025-12-04T09:01:24.0074827Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0075029Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0075097Z res = mod(**inputs) 2025-12-04T09:01:24.0075370Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0075456Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0075725Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0075795Z outputs = layer_module( 2025-12-04T09:01:24.0076059Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:01:24.0076280Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:01:24.0076551Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:01:24.0076639Z return forward_fn(*input_tensors) 2025-12-04T09:01:24.0076906Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:01:24.0076993Z output_x = self.ff(output_x) 2025-12-04T09:01:24.0077264Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-12-04T09:01:24.0077356Z output = self.activation_function(output) 2025-12-04T09:01:24.0077600Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:01:24.0077680Z return self.act(input) 2025-12-04T09:01:24.0077683Z 2025-12-04T09:01:24.0077789Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0077998Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0078063Z res = mod(**inputs) 2025-12-04T09:01:24.0078434Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0078542Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0078834Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0078917Z outputs = layer_module( 2025-12-04T09:01:24.0079201Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:01:24.0079435Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:01:24.0079741Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:01:24.0079825Z return forward_fn(*input_tensors) 2025-12-04T09:01:24.0080117Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:01:24.0080222Z output_x = self.ff(output_x) 2025-12-04T09:01:24.0080501Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-12-04T09:01:24.0080598Z output = self.layer_2(output) 2025-12-04T09:01:24.0080602Z 2025-12-04T09:01:24.0080708Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0080911Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0080988Z res = mod(**inputs) 2025-12-04T09:01:24.0081249Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0081341Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0081601Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0081673Z outputs = layer_module( 2025-12-04T09:01:24.0081945Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0082017Z outputs = self.rel_attn( 2025-12-04T09:01:24.0082275Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-12-04T09:01:24.0082383Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-12-04T09:01:24.0082388Z 2025-12-04T09:01:24.0082492Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0082699Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0082764Z res = mod(**inputs) 2025-12-04T09:01:24.0083025Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0083119Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0083383Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0083458Z outputs = layer_module( 2025-12-04T09:01:24.0083734Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0083805Z outputs = self.rel_attn( 2025-12-04T09:01:24.0084072Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-12-04T09:01:24.0084194Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-12-04T09:01:24.0084198Z 2025-12-04T09:01:24.0084303Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0084515Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0084581Z res = mod(**inputs) 2025-12-04T09:01:24.0084869Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0084952Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0085215Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0085293Z outputs = layer_module( 2025-12-04T09:01:24.0085555Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0085627Z outputs = self.rel_attn( 2025-12-04T09:01:24.0085893Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:01:24.0085968Z attn_vec = self.rel_attn_core( 2025-12-04T09:01:24.0086260Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-12-04T09:01:24.0086410Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-12-04T09:01:24.0086415Z 2025-12-04T09:01:24.0086520Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0086775Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0086843Z res = mod(**inputs) 2025-12-04T09:01:24.0087131Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0087221Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0087498Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0087579Z outputs = layer_module( 2025-12-04T09:01:24.0087856Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0087930Z outputs = self.rel_attn( 2025-12-04T09:01:24.0088214Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-12-04T09:01:24.0088358Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-12-04T09:01:24.0088362Z 2025-12-04T09:01:24.0088477Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0088689Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0088758Z res = mod(**inputs) 2025-12-04T09:01:24.0089046Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0089132Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0089419Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0089489Z outputs = layer_module( 2025-12-04T09:01:24.0089749Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0089827Z outputs = self.rel_attn( 2025-12-04T09:01:24.0090123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:01:24.0090198Z attn_vec = self.rel_attn_core( 2025-12-04T09:01:24.0090488Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-12-04T09:01:24.0090634Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-12-04T09:01:24.0090637Z 2025-12-04T09:01:24.0090748Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0090952Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0091016Z res = mod(**inputs) 2025-12-04T09:01:24.0091300Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0091384Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0091658Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0091726Z outputs = layer_module( 2025-12-04T09:01:24.0091989Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0092066Z outputs = self.rel_attn( 2025-12-04T09:01:24.0092335Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-12-04T09:01:24.0092437Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-12-04T09:01:24.0092440Z 2025-12-04T09:01:24.0092549Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0092777Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0092851Z res = mod(**inputs) 2025-12-04T09:01:24.0093115Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0093200Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0093469Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0093538Z outputs = layer_module( 2025-12-04T09:01:24.0093808Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0093877Z outputs = self.rel_attn( 2025-12-04T09:01:24.0094150Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:01:24.0094233Z attn_vec = self.rel_attn_core( 2025-12-04T09:01:24.0094517Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-12-04T09:01:24.0094643Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-12-04T09:01:24.0094648Z 2025-12-04T09:01:24.0094759Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0094962Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0095038Z res = mod(**inputs) 2025-12-04T09:01:24.0095302Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0095387Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0095657Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0095727Z outputs = layer_module( 2025-12-04T09:01:24.0096006Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0096088Z outputs = self.rel_attn( 2025-12-04T09:01:24.0096382Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:01:24.0096490Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:01:24.0096792Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:01:24.0096933Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:01:24.0096937Z 2025-12-04T09:01:24.0097055Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0097273Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0097349Z res = mod(**inputs) 2025-12-04T09:01:24.0097631Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0097736Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0098030Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0098102Z outputs = layer_module( 2025-12-04T09:01:24.0098381Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0098476Z outputs = self.rel_attn( 2025-12-04T09:01:24.0098758Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:01:24.0098861Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:01:24.0099167Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:01:24.0099305Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:01:24.0099308Z 2025-12-04T09:01:24.0099429Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0099644Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0099720Z res = mod(**inputs) 2025-12-04T09:01:24.0100003Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0100094Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0100381Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0100452Z outputs = layer_module( 2025-12-04T09:01:24.0100730Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:01:24.0100973Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:01:24.0101246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:01:24.0101333Z return forward_fn(*input_tensors) 2025-12-04T09:01:24.0101601Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:01:24.0101678Z output_x = self.ff(output_x) 2025-12-04T09:01:24.0101965Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-12-04T09:01:24.0102042Z output = self.layer_1(output) 2025-12-04T09:01:24.0102046Z 2025-12-04T09:01:24.0102172Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0102373Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0102441Z res = mod(**inputs) 2025-12-04T09:01:24.0102713Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0102796Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0103086Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0103164Z outputs = layer_module( 2025-12-04T09:01:24.0103425Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:01:24.0103660Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:01:24.0103931Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:01:24.0104011Z return forward_fn(*input_tensors) 2025-12-04T09:01:24.0104317Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:01:24.0104396Z output_x = self.ff(output_x) 2025-12-04T09:01:24.0104690Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-12-04T09:01:24.0104785Z output = self.activation_function(output) 2025-12-04T09:01:24.0105021Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:01:24.0105108Z return self.act(input) 2025-12-04T09:01:24.0105112Z 2025-12-04T09:01:24.0105221Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0105436Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0105510Z res = mod(**inputs) 2025-12-04T09:01:24.0105793Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0105918Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0106197Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0106268Z outputs = layer_module( 2025-12-04T09:01:24.0106549Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:01:24.0106772Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:01:24.0107061Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:01:24.0107141Z return forward_fn(*input_tensors) 2025-12-04T09:01:24.0107420Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:01:24.0107506Z output_x = self.ff(output_x) 2025-12-04T09:01:24.0107783Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-12-04T09:01:24.0107865Z output = self.layer_2(output) 2025-12-04T09:01:24.0107876Z 2025-12-04T09:01:24.0107987Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0108202Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0108279Z res = mod(**inputs) 2025-12-04T09:01:24.0108555Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0108642Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0108929Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0109002Z outputs = layer_module( 2025-12-04T09:01:24.0109285Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0109358Z outputs = self.rel_attn( 2025-12-04T09:01:24.0109653Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-12-04T09:01:24.0109768Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-12-04T09:01:24.0109771Z 2025-12-04T09:01:24.0109882Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0110114Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0110190Z res = mod(**inputs) 2025-12-04T09:01:24.0110468Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0110562Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0110844Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0110933Z outputs = layer_module( 2025-12-04T09:01:24.0111222Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0111295Z outputs = self.rel_attn( 2025-12-04T09:01:24.0111578Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-12-04T09:01:24.0111689Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-12-04T09:01:24.0111693Z 2025-12-04T09:01:24.0111802Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0112023Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0112093Z res = mod(**inputs) 2025-12-04T09:01:24.0112375Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0112499Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0112780Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0112859Z outputs = layer_module( 2025-12-04T09:01:24.0113134Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0113209Z outputs = self.rel_attn( 2025-12-04T09:01:24.0113493Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:01:24.0113571Z attn_vec = self.rel_attn_core( 2025-12-04T09:01:24.0113874Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-12-04T09:01:24.0114023Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-12-04T09:01:24.0114028Z 2025-12-04T09:01:24.0114141Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0114361Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0114430Z res = mod(**inputs) 2025-12-04T09:01:24.0114709Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0114804Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0115085Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0115163Z outputs = layer_module( 2025-12-04T09:01:24.0115438Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0115512Z outputs = self.rel_attn( 2025-12-04T09:01:24.0115797Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-12-04T09:01:24.0115943Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-12-04T09:01:24.0115947Z 2025-12-04T09:01:24.0116079Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0116296Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0116364Z res = mod(**inputs) 2025-12-04T09:01:24.0116670Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0116757Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0117037Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0117117Z outputs = layer_module( 2025-12-04T09:01:24.0117394Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0117489Z outputs = self.rel_attn( 2025-12-04T09:01:24.0117767Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:01:24.0117844Z attn_vec = self.rel_attn_core( 2025-12-04T09:01:24.0118149Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-12-04T09:01:24.0118361Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-12-04T09:01:24.0118367Z 2025-12-04T09:01:24.0118492Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0118715Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0118786Z res = mod(**inputs) 2025-12-04T09:01:24.0119083Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0119198Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0119488Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0119573Z outputs = layer_module( 2025-12-04T09:01:24.0119863Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0119948Z outputs = self.rel_attn( 2025-12-04T09:01:24.0120228Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-12-04T09:01:24.0120336Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-12-04T09:01:24.0120340Z 2025-12-04T09:01:24.0120458Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0120675Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0120902Z res = mod(**inputs) 2025-12-04T09:01:24.0121201Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0121297Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0121581Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0121654Z outputs = layer_module( 2025-12-04T09:01:24.0121934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0122017Z outputs = self.rel_attn( 2025-12-04T09:01:24.0122294Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:01:24.0122379Z attn_vec = self.rel_attn_core( 2025-12-04T09:01:24.0122679Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-12-04T09:01:24.0122816Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-12-04T09:01:24.0122820Z 2025-12-04T09:01:24.0122991Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0123205Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0123275Z res = mod(**inputs) 2025-12-04T09:01:24.0123562Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0123680Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0123975Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0124046Z outputs = layer_module( 2025-12-04T09:01:24.0124320Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0124403Z outputs = self.rel_attn( 2025-12-04T09:01:24.0124704Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:01:24.0124805Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:01:24.0125112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:01:24.0125235Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:01:24.0125239Z 2025-12-04T09:01:24.0125356Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0125572Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0125640Z res = mod(**inputs) 2025-12-04T09:01:24.0125925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0126043Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0126333Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0126408Z outputs = layer_module( 2025-12-04T09:01:24.0126685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0126765Z outputs = self.rel_attn( 2025-12-04T09:01:24.0127044Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:01:24.0127150Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:01:24.0127440Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:01:24.0127551Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:01:24.0127557Z 2025-12-04T09:01:24.0127667Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0127872Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0127940Z res = mod(**inputs) 2025-12-04T09:01:24.0128208Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0128292Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0128562Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0128631Z outputs = layer_module( 2025-12-04T09:01:24.0128893Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:01:24.0129117Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:01:24.0129393Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:01:24.0129480Z return forward_fn(*input_tensors) 2025-12-04T09:01:24.0129760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:01:24.0129837Z output_x = self.ff(output_x) 2025-12-04T09:01:24.0130107Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-12-04T09:01:24.0130201Z output = self.layer_1(output) 2025-12-04T09:01:24.0130205Z 2025-12-04T09:01:24.0130311Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0130521Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0130587Z res = mod(**inputs) 2025-12-04T09:01:24.0130866Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0130964Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0131230Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0131306Z outputs = layer_module( 2025-12-04T09:01:24.0131566Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:01:24.0131781Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:01:24.0132057Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:01:24.0132136Z return forward_fn(*input_tensors) 2025-12-04T09:01:24.0132406Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:01:24.0132497Z output_x = self.ff(output_x) 2025-12-04T09:01:24.0132762Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-12-04T09:01:24.0132860Z output = self.activation_function(output) 2025-12-04T09:01:24.0133084Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:01:24.0133164Z return self.act(input) 2025-12-04T09:01:24.0133167Z 2025-12-04T09:01:24.0133272Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0133482Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0133557Z res = mod(**inputs) 2025-12-04T09:01:24.0133876Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0133967Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0134258Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0134332Z outputs = layer_module( 2025-12-04T09:01:24.0134620Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:01:24.0134844Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:01:24.0135145Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:01:24.0135230Z return forward_fn(*input_tensors) 2025-12-04T09:01:24.0135497Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:01:24.0135575Z output_x = self.ff(output_x) 2025-12-04T09:01:24.0135841Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-12-04T09:01:24.0135915Z output = self.layer_2(output) 2025-12-04T09:01:24.0135919Z 2025-12-04T09:01:24.0136049Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0136254Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0136319Z res = mod(**inputs) 2025-12-04T09:01:24.0136618Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0136707Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0136995Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0137067Z outputs = layer_module( 2025-12-04T09:01:24.0137342Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0137452Z outputs = self.rel_attn( 2025-12-04T09:01:24.0137740Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-12-04T09:01:24.0137854Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-12-04T09:01:24.0137858Z 2025-12-04T09:01:24.0137969Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0138193Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0138269Z res = mod(**inputs) 2025-12-04T09:01:24.0138545Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0138633Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0138925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0139009Z outputs = layer_module( 2025-12-04T09:01:24.0139277Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0139349Z outputs = self.rel_attn( 2025-12-04T09:01:24.0139606Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-12-04T09:01:24.0139715Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-12-04T09:01:24.0139720Z 2025-12-04T09:01:24.0139823Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0140030Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0140096Z res = mod(**inputs) 2025-12-04T09:01:24.0140359Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0140451Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0140713Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0140781Z outputs = layer_module( 2025-12-04T09:01:24.0141049Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0141117Z outputs = self.rel_attn( 2025-12-04T09:01:24.0141381Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:01:24.0141456Z attn_vec = self.rel_attn_core( 2025-12-04T09:01:24.0141735Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-12-04T09:01:24.0141875Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-12-04T09:01:24.0141881Z 2025-12-04T09:01:24.0141985Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0142194Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0142262Z res = mod(**inputs) 2025-12-04T09:01:24.0142543Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0142634Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0142899Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0142986Z outputs = layer_module( 2025-12-04T09:01:24.0143254Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0143325Z outputs = self.rel_attn( 2025-12-04T09:01:24.0143603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-12-04T09:01:24.0143758Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-12-04T09:01:24.0143761Z 2025-12-04T09:01:24.0143865Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0144080Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0144144Z res = mod(**inputs) 2025-12-04T09:01:24.0144405Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0144498Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0144758Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0144835Z outputs = layer_module( 2025-12-04T09:01:24.0145119Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0145209Z outputs = self.rel_attn( 2025-12-04T09:01:24.0145493Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:01:24.0145573Z attn_vec = self.rel_attn_core( 2025-12-04T09:01:24.0145874Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-12-04T09:01:24.0146006Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-12-04T09:01:24.0146012Z 2025-12-04T09:01:24.0146114Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0146325Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0146390Z res = mod(**inputs) 2025-12-04T09:01:24.0146652Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0146745Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0147004Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0147082Z outputs = layer_module( 2025-12-04T09:01:24.0147344Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0147415Z outputs = self.rel_attn( 2025-12-04T09:01:24.0147685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-12-04T09:01:24.0147789Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-12-04T09:01:24.0147792Z 2025-12-04T09:01:24.0147901Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0148102Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0148169Z res = mod(**inputs) 2025-12-04T09:01:24.0148440Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0148524Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0148803Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0148880Z outputs = layer_module( 2025-12-04T09:01:24.0149140Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0149236Z outputs = self.rel_attn( 2025-12-04T09:01:24.0149494Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:01:24.0149567Z attn_vec = self.rel_attn_core( 2025-12-04T09:01:24.0149854Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-12-04T09:01:24.0149981Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-12-04T09:01:24.0150000Z 2025-12-04T09:01:24.0150112Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0150317Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0150382Z res = mod(**inputs) 2025-12-04T09:01:24.0150659Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0150742Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0151010Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0151086Z outputs = layer_module( 2025-12-04T09:01:24.0151352Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0151445Z outputs = self.rel_attn( 2025-12-04T09:01:24.0151709Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:01:24.0151801Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:01:24.0152098Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:01:24.0152213Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:01:24.0152218Z 2025-12-04T09:01:24.0152329Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0152536Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0152601Z res = mod(**inputs) 2025-12-04T09:01:24.0152884Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0152969Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0153236Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0153311Z outputs = layer_module( 2025-12-04T09:01:24.0153576Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0153650Z outputs = self.rel_attn( 2025-12-04T09:01:24.0153912Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:01:24.0154003Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:01:24.0154294Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:01:24.0154411Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:01:24.0154415Z 2025-12-04T09:01:24.0154534Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0154750Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0154820Z res = mod(**inputs) 2025-12-04T09:01:24.0155126Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0155217Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0155496Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0155595Z outputs = layer_module( 2025-12-04T09:01:24.0155872Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:01:24.0156106Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:01:24.0156398Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:01:24.0156498Z return forward_fn(*input_tensors) 2025-12-04T09:01:24.0156787Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:01:24.0156865Z output_x = self.ff(output_x) 2025-12-04T09:01:24.0157151Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-12-04T09:01:24.0157232Z output = self.layer_1(output) 2025-12-04T09:01:24.0157236Z 2025-12-04T09:01:24.0157346Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0157570Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0157638Z res = mod(**inputs) 2025-12-04T09:01:24.0157917Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0158034Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0158550Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0158643Z outputs = layer_module( 2025-12-04T09:01:24.0158929Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:01:24.0159160Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:01:24.0159470Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:01:24.0159553Z return forward_fn(*input_tensors) 2025-12-04T09:01:24.0159840Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:01:24.0159919Z output_x = self.ff(output_x) 2025-12-04T09:01:24.0160213Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-12-04T09:01:24.0160312Z output = self.activation_function(output) 2025-12-04T09:01:24.0160533Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:01:24.0160605Z return self.act(input) 2025-12-04T09:01:24.0160609Z 2025-12-04T09:01:24.0160732Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0160930Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0161002Z res = mod(**inputs) 2025-12-04T09:01:24.0161258Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0161339Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0161607Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0161674Z outputs = layer_module( 2025-12-04T09:01:24.0161954Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:01:24.0162175Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:01:24.0162446Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:01:24.0162557Z return forward_fn(*input_tensors) 2025-12-04T09:01:24.0162823Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:01:24.0162896Z output_x = self.ff(output_x) 2025-12-04T09:01:24.0163164Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-12-04T09:01:24.0163260Z output = self.layer_2(output) 2025-12-04T09:01:24.0163264Z 2025-12-04T09:01:24.0163378Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0163585Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0163651Z res = mod(**inputs) 2025-12-04T09:01:24.0163923Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0164017Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0164278Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0164356Z outputs = layer_module( 2025-12-04T09:01:24.0164615Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0164746Z outputs = self.rel_attn( 2025-12-04T09:01:24.0165012Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-12-04T09:01:24.0165113Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-12-04T09:01:24.0165116Z 2025-12-04T09:01:24.0165226Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0165428Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0165513Z res = mod(**inputs) 2025-12-04T09:01:24.0165770Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0165851Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0166127Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0166196Z outputs = layer_module( 2025-12-04T09:01:24.0166463Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0166538Z outputs = self.rel_attn( 2025-12-04T09:01:24.0166794Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-12-04T09:01:24.0166901Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-12-04T09:01:24.0166905Z 2025-12-04T09:01:24.0167007Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0167205Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0167277Z res = mod(**inputs) 2025-12-04T09:01:24.0167543Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0167633Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0167912Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0167979Z outputs = layer_module( 2025-12-04T09:01:24.0168257Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0168325Z outputs = self.rel_attn( 2025-12-04T09:01:24.0168580Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:01:24.0168675Z attn_vec = self.rel_attn_core( 2025-12-04T09:01:24.0168944Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-12-04T09:01:24.0169087Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-12-04T09:01:24.0169091Z 2025-12-04T09:01:24.0169192Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0169390Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0169476Z res = mod(**inputs) 2025-12-04T09:01:24.0169740Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0169832Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0170095Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0170166Z outputs = layer_module( 2025-12-04T09:01:24.0170433Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0170503Z outputs = self.rel_attn( 2025-12-04T09:01:24.0170764Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-12-04T09:01:24.0170949Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-12-04T09:01:24.0170953Z 2025-12-04T09:01:24.0171057Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0171266Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0171330Z res = mod(**inputs) 2025-12-04T09:01:24.0171593Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0171685Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0171949Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0172017Z outputs = layer_module( 2025-12-04T09:01:24.0172280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0172349Z outputs = self.rel_attn( 2025-12-04T09:01:24.0172617Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:01:24.0172690Z attn_vec = self.rel_attn_core( 2025-12-04T09:01:24.0172969Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-12-04T09:01:24.0173108Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-12-04T09:01:24.0173112Z 2025-12-04T09:01:24.0173214Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0173426Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0173492Z res = mod(**inputs) 2025-12-04T09:01:24.0173753Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0173842Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0174107Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0174175Z outputs = layer_module( 2025-12-04T09:01:24.0174460Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0174531Z outputs = self.rel_attn( 2025-12-04T09:01:24.0174799Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-12-04T09:01:24.0174917Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-12-04T09:01:24.0174921Z 2025-12-04T09:01:24.0175024Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0175237Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0175302Z res = mod(**inputs) 2025-12-04T09:01:24.0175571Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0175673Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0175942Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0176018Z outputs = layer_module( 2025-12-04T09:01:24.0176284Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0176355Z outputs = self.rel_attn( 2025-12-04T09:01:24.0176629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:01:24.0176702Z attn_vec = self.rel_attn_core( 2025-12-04T09:01:24.0176992Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-12-04T09:01:24.0177120Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-12-04T09:01:24.0177139Z 2025-12-04T09:01:24.0177246Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0177457Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0177523Z res = mod(**inputs) 2025-12-04T09:01:24.0177796Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0177879Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0178148Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0178224Z outputs = layer_module( 2025-12-04T09:01:24.0178485Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0178555Z outputs = self.rel_attn( 2025-12-04T09:01:24.0178829Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:01:24.0178920Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:01:24.0179211Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:01:24.0179325Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:01:24.0179329Z 2025-12-04T09:01:24.0179432Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0179645Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0179709Z res = mod(**inputs) 2025-12-04T09:01:24.0179980Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0180064Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0180333Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0180411Z outputs = layer_module( 2025-12-04T09:01:24.0180693Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0180767Z outputs = self.rel_attn( 2025-12-04T09:01:24.0181052Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:01:24.0181165Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:01:24.0181473Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:01:24.0181593Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:01:24.0181596Z 2025-12-04T09:01:24.0181707Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0181932Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0182027Z res = mod(**inputs) 2025-12-04T09:01:24.0182301Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0182386Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0182651Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0182728Z outputs = layer_module( 2025-12-04T09:01:24.0182988Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:01:24.0183202Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:01:24.0183482Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:01:24.0183579Z return forward_fn(*input_tensors) 2025-12-04T09:01:24.0183855Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:01:24.0183931Z output_x = self.ff(output_x) 2025-12-04T09:01:24.0184193Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-12-04T09:01:24.0184275Z output = self.layer_1(output) 2025-12-04T09:01:24.0184280Z 2025-12-04T09:01:24.0184383Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0184595Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0184658Z res = mod(**inputs) 2025-12-04T09:01:24.0184921Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0185013Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0185278Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0185347Z outputs = layer_module( 2025-12-04T09:01:24.0185617Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:01:24.0185827Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:01:24.0186105Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:01:24.0186184Z return forward_fn(*input_tensors) 2025-12-04T09:01:24.0186444Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:01:24.0186527Z output_x = self.ff(output_x) 2025-12-04T09:01:24.0186790Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-12-04T09:01:24.0186885Z output = self.activation_function(output) 2025-12-04T09:01:24.0187130Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:01:24.0187205Z return self.act(input) 2025-12-04T09:01:24.0187209Z 2025-12-04T09:01:24.0187321Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0187537Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0187602Z res = mod(**inputs) 2025-12-04T09:01:24.0187871Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0187955Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0188227Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0188299Z outputs = layer_module( 2025-12-04T09:01:24.0188577Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:01:24.0188805Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:01:24.0189082Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:01:24.0189169Z return forward_fn(*input_tensors) 2025-12-04T09:01:24.0189440Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:01:24.0189512Z output_x = self.ff(output_x) 2025-12-04T09:01:24.0189785Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-12-04T09:01:24.0189877Z output = self.layer_2(output) 2025-12-04T09:01:24.0189881Z 2025-12-04T09:01:24.0189988Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0190199Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0190264Z res = mod(**inputs) 2025-12-04T09:01:24.0190532Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0190617Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0190878Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0190955Z outputs = layer_module( 2025-12-04T09:01:24.0191214Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0191284Z outputs = self.rel_attn( 2025-12-04T09:01:24.0191563Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-12-04T09:01:24.0191663Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-12-04T09:01:24.0191666Z 2025-12-04T09:01:24.0191780Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0191983Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0192049Z res = mod(**inputs) 2025-12-04T09:01:24.0192323Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0192407Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0192679Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0192747Z outputs = layer_module( 2025-12-04T09:01:24.0193011Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0193091Z outputs = self.rel_attn( 2025-12-04T09:01:24.0193366Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-12-04T09:01:24.0193471Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-12-04T09:01:24.0193482Z 2025-12-04T09:01:24.0193587Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0193806Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0193880Z res = mod(**inputs) 2025-12-04T09:01:24.0194148Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0194231Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0194501Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0194571Z outputs = layer_module( 2025-12-04T09:01:24.0194853Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0194924Z outputs = self.rel_attn( 2025-12-04T09:01:24.0195183Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:01:24.0195267Z attn_vec = self.rel_attn_core( 2025-12-04T09:01:24.0195564Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-12-04T09:01:24.0195707Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-12-04T09:01:24.0195712Z 2025-12-04T09:01:24.0195832Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0196044Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0196136Z res = mod(**inputs) 2025-12-04T09:01:24.0196418Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0196508Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0196793Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0196865Z outputs = layer_module( 2025-12-04T09:01:24.0197154Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0197228Z outputs = self.rel_attn( 2025-12-04T09:01:24.0197500Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-12-04T09:01:24.0197649Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-12-04T09:01:24.0197655Z 2025-12-04T09:01:24.0197763Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0197981Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0198058Z res = mod(**inputs) 2025-12-04T09:01:24.0198437Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0198543Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0198828Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0198901Z outputs = layer_module( 2025-12-04T09:01:24.0199192Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0199266Z outputs = self.rel_attn( 2025-12-04T09:01:24.0199555Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:01:24.0199641Z attn_vec = self.rel_attn_core( 2025-12-04T09:01:24.0199974Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-12-04T09:01:24.0200124Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-12-04T09:01:24.0200129Z 2025-12-04T09:01:24.0200240Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0200470Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0200548Z res = mod(**inputs) 2025-12-04T09:01:24.0200828Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0200925Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0201206Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0201281Z outputs = layer_module( 2025-12-04T09:01:24.0201587Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0201664Z outputs = self.rel_attn( 2025-12-04T09:01:24.0201936Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-12-04T09:01:24.0202053Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-12-04T09:01:24.0202058Z 2025-12-04T09:01:24.0202166Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0202388Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0202453Z res = mod(**inputs) 2025-12-04T09:01:24.0202731Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0202846Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0203137Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0203217Z outputs = layer_module( 2025-12-04T09:01:24.0203494Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0203568Z outputs = self.rel_attn( 2025-12-04T09:01:24.0203855Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:01:24.0203931Z attn_vec = self.rel_attn_core( 2025-12-04T09:01:24.0204225Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-12-04T09:01:24.0204364Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-12-04T09:01:24.0204370Z 2025-12-04T09:01:24.0204480Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0204698Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0204764Z res = mod(**inputs) 2025-12-04T09:01:24.0205040Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0205133Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0205411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0205489Z outputs = layer_module( 2025-12-04T09:01:24.0205766Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0205835Z outputs = self.rel_attn( 2025-12-04T09:01:24.0206114Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:01:24.0206215Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:01:24.0206533Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:01:24.0206663Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:01:24.0206666Z 2025-12-04T09:01:24.0206776Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0206998Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0207084Z res = mod(**inputs) 2025-12-04T09:01:24.0207368Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0207461Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0207740Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0207820Z outputs = layer_module( 2025-12-04T09:01:24.0208112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0208190Z outputs = self.rel_attn( 2025-12-04T09:01:24.0208469Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:01:24.0208564Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:01:24.0208867Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:01:24.0208991Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:01:24.0208995Z 2025-12-04T09:01:24.0209104Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0209324Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0209422Z res = mod(**inputs) 2025-12-04T09:01:24.0209703Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0209801Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0210079Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0210158Z outputs = layer_module( 2025-12-04T09:01:24.0210435Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:01:24.0210663Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:01:24.0210955Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:01:24.0211039Z return forward_fn(*input_tensors) 2025-12-04T09:01:24.0211316Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:01:24.0211401Z output_x = self.ff(output_x) 2025-12-04T09:01:24.0211676Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-12-04T09:01:24.0211762Z output = self.layer_1(output) 2025-12-04T09:01:24.0211766Z 2025-12-04T09:01:24.0211874Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0212089Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0212165Z res = mod(**inputs) 2025-12-04T09:01:24.0212440Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0212537Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0212815Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0212889Z outputs = layer_module( 2025-12-04T09:01:24.0213204Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:01:24.0213433Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:01:24.0213719Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:01:24.0213824Z return forward_fn(*input_tensors) 2025-12-04T09:01:24.0214101Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:01:24.0214188Z output_x = self.ff(output_x) 2025-12-04T09:01:24.0214464Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-12-04T09:01:24.0214575Z output = self.activation_function(output) 2025-12-04T09:01:24.0214820Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:01:24.0214893Z return self.act(input) 2025-12-04T09:01:24.0214896Z 2025-12-04T09:01:24.0215007Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0215209Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0215276Z res = mod(**inputs) 2025-12-04T09:01:24.0215543Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0215628Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0215888Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0215983Z outputs = layer_module( 2025-12-04T09:01:24.0216244Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:01:24.0216462Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:01:24.0216732Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:01:24.0216812Z return forward_fn(*input_tensors) 2025-12-04T09:01:24.0217086Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:01:24.0217158Z output_x = self.ff(output_x) 2025-12-04T09:01:24.0217423Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-12-04T09:01:24.0217497Z output = self.layer_2(output) 2025-12-04T09:01:24.0217502Z 2025-12-04T09:01:24.0217607Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0217817Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0217882Z res = mod(**inputs) 2025-12-04T09:01:24.0218148Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0218240Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0218503Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0218578Z outputs = layer_module( 2025-12-04T09:01:24.0218838Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0218907Z outputs = self.rel_attn( 2025-12-04T09:01:24.0219173Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-12-04T09:01:24.0219274Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-12-04T09:01:24.0219278Z 2025-12-04T09:01:24.0219387Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0219620Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0219690Z res = mod(**inputs) 2025-12-04T09:01:24.0219986Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0220093Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0220372Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0220447Z outputs = layer_module( 2025-12-04T09:01:24.0220857Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0220948Z outputs = self.rel_attn( 2025-12-04T09:01:24.0221257Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-12-04T09:01:24.0221364Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-12-04T09:01:24.0221368Z 2025-12-04T09:01:24.0221483Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0221698Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0221769Z res = mod(**inputs) 2025-12-04T09:01:24.0222059Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0222150Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0222435Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0222536Z outputs = layer_module( 2025-12-04T09:01:24.0222826Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0222904Z outputs = self.rel_attn( 2025-12-04T09:01:24.0223167Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:01:24.0223252Z attn_vec = self.rel_attn_core( 2025-12-04T09:01:24.0223534Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-12-04T09:01:24.0223674Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-12-04T09:01:24.0223677Z 2025-12-04T09:01:24.0223791Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0223993Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0224060Z res = mod(**inputs) 2025-12-04T09:01:24.0224335Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0224419Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0224687Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0224756Z outputs = layer_module( 2025-12-04T09:01:24.0225017Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0225097Z outputs = self.rel_attn( 2025-12-04T09:01:24.0225364Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-12-04T09:01:24.0225502Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-12-04T09:01:24.0225505Z 2025-12-04T09:01:24.0225607Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0225812Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0225883Z res = mod(**inputs) 2025-12-04T09:01:24.0226172Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0226257Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0226537Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0226636Z outputs = layer_module( 2025-12-04T09:01:24.0226925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0226999Z outputs = self.rel_attn( 2025-12-04T09:01:24.0227273Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:01:24.0227357Z attn_vec = self.rel_attn_core( 2025-12-04T09:01:24.0227651Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-12-04T09:01:24.0227789Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-12-04T09:01:24.0227800Z 2025-12-04T09:01:24.0227902Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0228105Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0228191Z res = mod(**inputs) 2025-12-04T09:01:24.0228466Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0228556Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0228845Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0228939Z outputs = layer_module( 2025-12-04T09:01:24.0229233Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0229305Z outputs = self.rel_attn( 2025-12-04T09:01:24.0229584Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-12-04T09:01:24.0229702Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-12-04T09:01:24.0229707Z 2025-12-04T09:01:24.0229815Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0230037Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0230104Z res = mod(**inputs) 2025-12-04T09:01:24.0230384Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0230481Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0230763Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0230844Z outputs = layer_module( 2025-12-04T09:01:24.0231117Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0231186Z outputs = self.rel_attn( 2025-12-04T09:01:24.0231462Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:01:24.0231540Z attn_vec = self.rel_attn_core( 2025-12-04T09:01:24.0231838Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-12-04T09:01:24.0231978Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-12-04T09:01:24.0231983Z 2025-12-04T09:01:24.0232093Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0232313Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0232391Z res = mod(**inputs) 2025-12-04T09:01:24.0232694Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0232791Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0233076Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0233166Z outputs = layer_module( 2025-12-04T09:01:24.0233454Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0233525Z outputs = self.rel_attn( 2025-12-04T09:01:24.0233809Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:01:24.0233906Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:01:24.0234256Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:01:24.0234386Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:01:24.0234390Z 2025-12-04T09:01:24.0234500Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0234717Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0234796Z res = mod(**inputs) 2025-12-04T09:01:24.0235076Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0235170Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0235446Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0235535Z outputs = layer_module( 2025-12-04T09:01:24.0235823Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0235895Z outputs = self.rel_attn( 2025-12-04T09:01:24.0236177Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:01:24.0236274Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:01:24.0236572Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:01:24.0236702Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:01:24.0236705Z 2025-12-04T09:01:24.0236814Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0237028Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0237102Z res = mod(**inputs) 2025-12-04T09:01:24.0237382Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0237478Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0237758Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0237829Z outputs = layer_module( 2025-12-04T09:01:24.0238112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:01:24.0238403Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:01:24.0238711Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:01:24.0238797Z return forward_fn(*input_tensors) 2025-12-04T09:01:24.0239084Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:01:24.0239178Z output_x = self.ff(output_x) 2025-12-04T09:01:24.0239497Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-12-04T09:01:24.0239578Z output = self.layer_1(output) 2025-12-04T09:01:24.0239582Z 2025-12-04T09:01:24.0239704Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0239920Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0240013Z res = mod(**inputs) 2025-12-04T09:01:24.0240297Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0240390Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0240683Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0240760Z outputs = layer_module( 2025-12-04T09:01:24.0241062Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:01:24.0241292Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:01:24.0241580Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:01:24.0241672Z return forward_fn(*input_tensors) 2025-12-04T09:01:24.0241954Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:01:24.0242030Z output_x = self.ff(output_x) 2025-12-04T09:01:24.0242320Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-12-04T09:01:24.0242433Z output = self.activation_function(output) 2025-12-04T09:01:24.0242676Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:01:24.0242753Z return self.act(input) 2025-12-04T09:01:24.0242757Z 2025-12-04T09:01:24.0242870Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0243091Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0243160Z res = mod(**inputs) 2025-12-04T09:01:24.0243462Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0243551Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0243829Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0243909Z outputs = layer_module( 2025-12-04T09:01:24.0244194Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:01:24.0244420Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:01:24.0244716Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:01:24.0244797Z return forward_fn(*input_tensors) 2025-12-04T09:01:24.0245082Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:01:24.0245162Z output_x = self.ff(output_x) 2025-12-04T09:01:24.0245438Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-12-04T09:01:24.0245522Z output = self.layer_2(output) 2025-12-04T09:01:24.0245526Z 2025-12-04T09:01:24.0245645Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0245856Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0245921Z res = mod(**inputs) 2025-12-04T09:01:24.0246202Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0246296Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0246569Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0246654Z outputs = layer_module( 2025-12-04T09:01:24.0246915Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0246985Z outputs = self.rel_attn( 2025-12-04T09:01:24.0247246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-12-04T09:01:24.0247346Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-12-04T09:01:24.0247349Z 2025-12-04T09:01:24.0247465Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0247673Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0247737Z res = mod(**inputs) 2025-12-04T09:01:24.0247988Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0248078Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0248338Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0248413Z outputs = layer_module( 2025-12-04T09:01:24.0248672Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0248757Z outputs = self.rel_attn( 2025-12-04T09:01:24.0249028Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-12-04T09:01:24.0249132Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-12-04T09:01:24.0249136Z 2025-12-04T09:01:24.0249248Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0249450Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0249514Z res = mod(**inputs) 2025-12-04T09:01:24.0249794Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0249876Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0250138Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0250215Z outputs = layer_module( 2025-12-04T09:01:24.0250475Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0250552Z outputs = self.rel_attn( 2025-12-04T09:01:24.0250813Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:01:24.0250887Z attn_vec = self.rel_attn_core( 2025-12-04T09:01:24.0251172Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-12-04T09:01:24.0251307Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-12-04T09:01:24.0251310Z 2025-12-04T09:01:24.0251421Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0251621Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0251686Z res = mod(**inputs) 2025-12-04T09:01:24.0251965Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0252055Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0252347Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0252437Z outputs = layer_module( 2025-12-04T09:01:24.0252700Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0252793Z outputs = self.rel_attn( 2025-12-04T09:01:24.0253053Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-12-04T09:01:24.0253188Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-12-04T09:01:24.0253191Z 2025-12-04T09:01:24.0253302Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0253506Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0253578Z res = mod(**inputs) 2025-12-04T09:01:24.0253859Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0253944Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0254217Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0254291Z outputs = layer_module( 2025-12-04T09:01:24.0254569Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0254650Z outputs = self.rel_attn( 2025-12-04T09:01:24.0254925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:01:24.0255009Z attn_vec = self.rel_attn_core( 2025-12-04T09:01:24.0255329Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-12-04T09:01:24.0255468Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-12-04T09:01:24.0255472Z 2025-12-04T09:01:24.0255598Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0255800Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0255869Z res = mod(**inputs) 2025-12-04T09:01:24.0256132Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0256215Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0256488Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0256557Z outputs = layer_module( 2025-12-04T09:01:24.0256829Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0256908Z outputs = self.rel_attn( 2025-12-04T09:01:24.0257172Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-12-04T09:01:24.0257282Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-12-04T09:01:24.0257285Z 2025-12-04T09:01:24.0257388Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0257592Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0257666Z res = mod(**inputs) 2025-12-04T09:01:24.0257929Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0258015Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0258297Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0258370Z outputs = layer_module( 2025-12-04T09:01:24.0258661Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0258734Z outputs = self.rel_attn( 2025-12-04T09:01:24.0259010Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:01:24.0259125Z attn_vec = self.rel_attn_core( 2025-12-04T09:01:24.0259421Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-12-04T09:01:24.0259562Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-12-04T09:01:24.0259565Z 2025-12-04T09:01:24.0259674Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0259892Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0259970Z res = mod(**inputs) 2025-12-04T09:01:24.0260267Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0260360Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0260645Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0260719Z outputs = layer_module( 2025-12-04T09:01:24.0261003Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0261077Z outputs = self.rel_attn( 2025-12-04T09:01:24.0261365Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:01:24.0261464Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:01:24.0261765Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:01:24.0261884Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:01:24.0261887Z 2025-12-04T09:01:24.0261992Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0262199Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0262273Z res = mod(**inputs) 2025-12-04T09:01:24.0262540Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0262630Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0262916Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0262988Z outputs = layer_module( 2025-12-04T09:01:24.0263272Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0263348Z outputs = self.rel_attn( 2025-12-04T09:01:24.0263624Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:01:24.0263728Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:01:24.0264026Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:01:24.0264154Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:01:24.0264158Z 2025-12-04T09:01:24.0264267Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0264481Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0264558Z res = mod(**inputs) 2025-12-04T09:01:24.0264837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0264930Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0265246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0265317Z outputs = layer_module( 2025-12-04T09:01:24.0265582Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:01:24.0265813Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:01:24.0266087Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:01:24.0266172Z return forward_fn(*input_tensors) 2025-12-04T09:01:24.0266436Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:01:24.0266517Z output_x = self.ff(output_x) 2025-12-04T09:01:24.0266832Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-12-04T09:01:24.0266909Z output = self.layer_1(output) 2025-12-04T09:01:24.0266913Z 2025-12-04T09:01:24.0267024Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0267233Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0267301Z res = mod(**inputs) 2025-12-04T09:01:24.0267572Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0267655Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0267923Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0268007Z outputs = layer_module( 2025-12-04T09:01:24.0268275Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:01:24.0268492Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:01:24.0268762Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:01:24.0268845Z return forward_fn(*input_tensors) 2025-12-04T09:01:24.0269114Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:01:24.0269187Z output_x = self.ff(output_x) 2025-12-04T09:01:24.0269455Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-12-04T09:01:24.0269544Z output = self.activation_function(output) 2025-12-04T09:01:24.0269772Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:01:24.0269850Z return self.act(input) 2025-12-04T09:01:24.0269854Z 2025-12-04T09:01:24.0269960Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0270171Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0270236Z res = mod(**inputs) 2025-12-04T09:01:24.0270505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0270597Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0270862Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0270938Z outputs = layer_module( 2025-12-04T09:01:24.0271201Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:01:24.0271414Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:01:24.0271710Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:01:24.0271789Z return forward_fn(*input_tensors) 2025-12-04T09:01:24.0272057Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:01:24.0272160Z output_x = self.ff(output_x) 2025-12-04T09:01:24.0272426Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-12-04T09:01:24.0272509Z output = self.layer_2(output) 2025-12-04T09:01:24.0272513Z 2025-12-04T09:01:24.0272626Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0272841Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0272920Z res = mod(**inputs) 2025-12-04T09:01:24.0273221Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0273317Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0273598Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0273673Z outputs = layer_module( 2025-12-04T09:01:24.0273959Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0274034Z outputs = self.rel_attn( 2025-12-04T09:01:24.0274314Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-12-04T09:01:24.0274427Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-12-04T09:01:24.0274445Z 2025-12-04T09:01:24.0274557Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0274786Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0274859Z res = mod(**inputs) 2025-12-04T09:01:24.0275144Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0275241Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0275523Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0275595Z outputs = layer_module( 2025-12-04T09:01:24.0275885Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0275958Z outputs = self.rel_attn( 2025-12-04T09:01:24.0276249Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-12-04T09:01:24.0276362Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-12-04T09:01:24.0276366Z 2025-12-04T09:01:24.0276477Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0276703Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0276772Z res = mod(**inputs) 2025-12-04T09:01:24.0277065Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0277155Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0277449Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0277528Z outputs = layer_module( 2025-12-04T09:01:24.0277809Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0277884Z outputs = self.rel_attn( 2025-12-04T09:01:24.0278246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:01:24.0278359Z attn_vec = self.rel_attn_core( 2025-12-04T09:01:24.0278675Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-12-04T09:01:24.0278824Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-12-04T09:01:24.0278845Z 2025-12-04T09:01:24.0278964Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0279195Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0279268Z res = mod(**inputs) 2025-12-04T09:01:24.0279565Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0279671Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0279969Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0280053Z outputs = layer_module( 2025-12-04T09:01:24.0280336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0280414Z outputs = self.rel_attn( 2025-12-04T09:01:24.0280720Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-12-04T09:01:24.0280869Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-12-04T09:01:24.0280873Z 2025-12-04T09:01:24.0280995Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0281219Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0281308Z res = mod(**inputs) 2025-12-04T09:01:24.0281611Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0281702Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0282012Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0282088Z outputs = layer_module( 2025-12-04T09:01:24.0282377Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0282462Z outputs = self.rel_attn( 2025-12-04T09:01:24.0282757Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:01:24.0282836Z attn_vec = self.rel_attn_core( 2025-12-04T09:01:24.0283154Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-12-04T09:01:24.0283301Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-12-04T09:01:24.0283304Z 2025-12-04T09:01:24.0283427Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0283648Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0283719Z res = mod(**inputs) 2025-12-04T09:01:24.0284014Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0284106Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0284408Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0284483Z outputs = layer_module( 2025-12-04T09:01:24.0284772Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0284853Z outputs = self.rel_attn( 2025-12-04T09:01:24.0285152Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-12-04T09:01:24.0285295Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-12-04T09:01:24.0285299Z 2025-12-04T09:01:24.0285422Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0285650Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0285744Z res = mod(**inputs) 2025-12-04T09:01:24.0286032Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0286122Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0286428Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0286505Z outputs = layer_module( 2025-12-04T09:01:24.0286809Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0286894Z outputs = self.rel_attn( 2025-12-04T09:01:24.0287181Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:01:24.0287268Z attn_vec = self.rel_attn_core( 2025-12-04T09:01:24.0287574Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-12-04T09:01:24.0287713Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-12-04T09:01:24.0287717Z 2025-12-04T09:01:24.0287838Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0288058Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0288156Z res = mod(**inputs) 2025-12-04T09:01:24.0288448Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0288539Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0288836Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0288911Z outputs = layer_module( 2025-12-04T09:01:24.0289197Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0289282Z outputs = self.rel_attn( 2025-12-04T09:01:24.0289566Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:01:24.0289673Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:01:24.0289984Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:01:24.0290111Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:01:24.0290115Z 2025-12-04T09:01:24.0290238Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0290467Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0290546Z res = mod(**inputs) 2025-12-04T09:01:24.0290832Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0290925Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0291218Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0291293Z outputs = layer_module( 2025-12-04T09:01:24.0291598Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0291681Z outputs = self.rel_attn( 2025-12-04T09:01:24.0291977Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:01:24.0292107Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:01:24.0292423Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:01:24.0292547Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:01:24.0292568Z 2025-12-04T09:01:24.0292690Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0292912Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0292990Z res = mod(**inputs) 2025-12-04T09:01:24.0293285Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0293378Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0293683Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0293761Z outputs = layer_module( 2025-12-04T09:01:24.0294042Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:01:24.0294291Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:01:24.0294586Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:01:24.0294675Z return forward_fn(*input_tensors) 2025-12-04T09:01:24.0294959Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:01:24.0295054Z output_x = self.ff(output_x) 2025-12-04T09:01:24.0295342Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-12-04T09:01:24.0295419Z output = self.layer_1(output) 2025-12-04T09:01:24.0295423Z 2025-12-04T09:01:24.0295541Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0295755Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0295824Z res = mod(**inputs) 2025-12-04T09:01:24.0296109Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0296202Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0296489Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0296566Z outputs = layer_module( 2025-12-04T09:01:24.0296844Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:01:24.0297078Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:01:24.0297364Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:01:24.0297446Z return forward_fn(*input_tensors) 2025-12-04T09:01:24.0297732Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:01:24.0297812Z output_x = self.ff(output_x) 2025-12-04T09:01:24.0298096Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-12-04T09:01:24.0298194Z output = self.activation_function(output) 2025-12-04T09:01:24.0298428Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:01:24.0298513Z return self.act(input) 2025-12-04T09:01:24.0298516Z 2025-12-04T09:01:24.0298629Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0298863Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0298943Z res = mod(**inputs) 2025-12-04T09:01:24.0299221Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0299337Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0299611Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0299681Z outputs = layer_module( 2025-12-04T09:01:24.0299962Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:01:24.0300188Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:01:24.0300495Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:01:24.0300579Z return forward_fn(*input_tensors) 2025-12-04T09:01:24.0300858Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:01:24.0300941Z output_x = self.ff(output_x) 2025-12-04T09:01:24.0301219Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-12-04T09:01:24.0301297Z output = self.layer_2(output) 2025-12-04T09:01:24.0301300Z 2025-12-04T09:01:24.0301418Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0301634Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0301729Z res = mod(**inputs) 2025-12-04T09:01:24.0302019Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0302107Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0302399Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0302470Z outputs = layer_module( 2025-12-04T09:01:24.0302762Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0302838Z outputs = self.rel_attn( 2025-12-04T09:01:24.0303120Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-12-04T09:01:24.0303233Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-12-04T09:01:24.0303237Z 2025-12-04T09:01:24.0303351Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0303569Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0303643Z res = mod(**inputs) 2025-12-04T09:01:24.0303932Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0304027Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0304320Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0304394Z outputs = layer_module( 2025-12-04T09:01:24.0304687Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0304759Z outputs = self.rel_attn( 2025-12-04T09:01:24.0305042Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-12-04T09:01:24.0305155Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-12-04T09:01:24.0305160Z 2025-12-04T09:01:24.0305270Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0305522Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0305592Z res = mod(**inputs) 2025-12-04T09:01:24.0305875Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0305988Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0306265Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0306344Z outputs = layer_module( 2025-12-04T09:01:24.0306621Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0306696Z outputs = self.rel_attn( 2025-12-04T09:01:24.0306995Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:01:24.0307073Z attn_vec = self.rel_attn_core( 2025-12-04T09:01:24.0307371Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-12-04T09:01:24.0307524Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-12-04T09:01:24.0307530Z 2025-12-04T09:01:24.0307641Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0307862Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0307933Z res = mod(**inputs) 2025-12-04T09:01:24.0308217Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0308316Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0308641Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0308720Z outputs = layer_module( 2025-12-04T09:01:24.0308998Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0309073Z outputs = self.rel_attn( 2025-12-04T09:01:24.0309365Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-12-04T09:01:24.0309515Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-12-04T09:01:24.0309519Z 2025-12-04T09:01:24.0309632Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0309859Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0309930Z res = mod(**inputs) 2025-12-04T09:01:24.0310228Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0310319Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0310603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0310686Z outputs = layer_module( 2025-12-04T09:01:24.0310976Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0311059Z outputs = self.rel_attn( 2025-12-04T09:01:24.0311334Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:01:24.0311410Z attn_vec = self.rel_attn_core( 2025-12-04T09:01:24.0311713Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-12-04T09:01:24.0311855Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-12-04T09:01:24.0311860Z 2025-12-04T09:01:24.0311971Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0312215Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0312286Z res = mod(**inputs) 2025-12-04T09:01:24.0312570Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0312677Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0312954Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0313032Z outputs = layer_module( 2025-12-04T09:01:24.0313309Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0313384Z outputs = self.rel_attn( 2025-12-04T09:01:24.0313683Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-12-04T09:01:24.0313791Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-12-04T09:01:24.0313796Z 2025-12-04T09:01:24.0313913Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0314127Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0314198Z res = mod(**inputs) 2025-12-04T09:01:24.0314487Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0314576Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0314868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0314940Z outputs = layer_module( 2025-12-04T09:01:24.0315238Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0315322Z outputs = self.rel_attn( 2025-12-04T09:01:24.0315605Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:01:24.0315683Z attn_vec = self.rel_attn_core( 2025-12-04T09:01:24.0315987Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-12-04T09:01:24.0316121Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-12-04T09:01:24.0316125Z 2025-12-04T09:01:24.0316244Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0316459Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0316528Z res = mod(**inputs) 2025-12-04T09:01:24.0316817Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0316910Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0317201Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0317273Z outputs = layer_module( 2025-12-04T09:01:24.0317553Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0317635Z outputs = self.rel_attn( 2025-12-04T09:01:24.0317914Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:01:24.0318013Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:01:24.0318406Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:01:24.0318539Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:01:24.0318544Z 2025-12-04T09:01:24.0318667Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0318927Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0319000Z res = mod(**inputs) 2025-12-04T09:01:24.0319299Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0319410Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0319711Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0319786Z outputs = layer_module( 2025-12-04T09:01:24.0320073Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0320156Z outputs = self.rel_attn( 2025-12-04T09:01:24.0320447Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:01:24.0320544Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:01:24.0321012Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:01:24.0321138Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:01:24.0321142Z 2025-12-04T09:01:24.0321264Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0321479Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0321549Z res = mod(**inputs) 2025-12-04T09:01:24.0321852Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0321942Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0322274Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0322347Z outputs = layer_module( 2025-12-04T09:01:24.0322625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:01:24.0322869Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:01:24.0323155Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:01:24.0323239Z return forward_fn(*input_tensors) 2025-12-04T09:01:24.0323525Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:01:24.0323606Z output_x = self.ff(output_x) 2025-12-04T09:01:24.0323890Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-12-04T09:01:24.0323972Z output = self.layer_1(output) 2025-12-04T09:01:24.0323976Z 2025-12-04T09:01:24.0324086Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0324311Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0324378Z res = mod(**inputs) 2025-12-04T09:01:24.0324704Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0324793Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0325055Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0325132Z outputs = layer_module( 2025-12-04T09:01:24.0325395Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:01:24.0325614Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:01:24.0325924Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:01:24.0326004Z return forward_fn(*input_tensors) 2025-12-04T09:01:24.0326282Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:01:24.0326383Z output_x = self.ff(output_x) 2025-12-04T09:01:24.0326657Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-12-04T09:01:24.0326759Z output = self.activation_function(output) 2025-12-04T09:01:24.0326980Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:01:24.0327060Z return self.act(input) 2025-12-04T09:01:24.0327064Z 2025-12-04T09:01:24.0327190Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0327394Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0327467Z res = mod(**inputs) 2025-12-04T09:01:24.0327727Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0327811Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0328086Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0328155Z outputs = layer_module( 2025-12-04T09:01:24.0328424Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:01:24.0328634Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:01:24.0328925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:01:24.0329010Z return forward_fn(*input_tensors) 2025-12-04T09:01:24.0329275Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:01:24.0329356Z output_x = self.ff(output_x) 2025-12-04T09:01:24.0329619Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-12-04T09:01:24.0329695Z output = self.layer_2(output) 2025-12-04T09:01:24.0329699Z 2025-12-04T09:01:24.0329811Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0330014Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0330078Z res = mod(**inputs) 2025-12-04T09:01:24.0330350Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0330435Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0330707Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0330777Z outputs = layer_module( 2025-12-04T09:01:24.0331040Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0331120Z outputs = self.rel_attn( 2025-12-04T09:01:24.0331382Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-12-04T09:01:24.0331484Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-12-04T09:01:24.0331495Z 2025-12-04T09:01:24.0331599Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0331806Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0331881Z res = mod(**inputs) 2025-12-04T09:01:24.0332182Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0332266Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0332532Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0332617Z outputs = layer_module( 2025-12-04T09:01:24.0332885Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0332955Z outputs = self.rel_attn( 2025-12-04T09:01:24.0333213Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-12-04T09:01:24.0333322Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-12-04T09:01:24.0333327Z 2025-12-04T09:01:24.0333457Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0333675Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0333741Z res = mod(**inputs) 2025-12-04T09:01:24.0334005Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0334095Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0334361Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0334430Z outputs = layer_module( 2025-12-04T09:01:24.0334701Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0334769Z outputs = self.rel_attn( 2025-12-04T09:01:24.0335055Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:01:24.0335128Z attn_vec = self.rel_attn_core( 2025-12-04T09:01:24.0335410Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-12-04T09:01:24.0335552Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-12-04T09:01:24.0335555Z 2025-12-04T09:01:24.0335660Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0335863Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0335936Z res = mod(**inputs) 2025-12-04T09:01:24.0336199Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0336289Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0336551Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0336621Z outputs = layer_module( 2025-12-04T09:01:24.0336891Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0336962Z outputs = self.rel_attn( 2025-12-04T09:01:24.0337225Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-12-04T09:01:24.0337360Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-12-04T09:01:24.0337364Z 2025-12-04T09:01:24.0337467Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0337674Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0337740Z res = mod(**inputs) 2025-12-04T09:01:24.0338003Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0338095Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0338376Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0338453Z outputs = layer_module( 2025-12-04T09:01:24.0338718Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0338787Z outputs = self.rel_attn( 2025-12-04T09:01:24.0339084Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:01:24.0339157Z attn_vec = self.rel_attn_core( 2025-12-04T09:01:24.0339443Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-12-04T09:01:24.0339574Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-12-04T09:01:24.0339579Z 2025-12-04T09:01:24.0339696Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0339905Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0339971Z res = mod(**inputs) 2025-12-04T09:01:24.0340235Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0340328Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0340591Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0340666Z outputs = layer_module( 2025-12-04T09:01:24.0340931Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0341001Z outputs = self.rel_attn( 2025-12-04T09:01:24.0341288Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-12-04T09:01:24.0341392Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-12-04T09:01:24.0341396Z 2025-12-04T09:01:24.0341508Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0341712Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0341777Z res = mod(**inputs) 2025-12-04T09:01:24.0342052Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0342137Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0342402Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0342477Z outputs = layer_module( 2025-12-04T09:01:24.0342759Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0342836Z outputs = self.rel_attn( 2025-12-04T09:01:24.0343099Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:01:24.0343173Z attn_vec = self.rel_attn_core( 2025-12-04T09:01:24.0343459Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-12-04T09:01:24.0343584Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-12-04T09:01:24.0343588Z 2025-12-04T09:01:24.0343697Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0343903Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0343968Z res = mod(**inputs) 2025-12-04T09:01:24.0344242Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0344327Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0344595Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0344690Z outputs = layer_module( 2025-12-04T09:01:24.0344951Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0345028Z outputs = self.rel_attn( 2025-12-04T09:01:24.0345306Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:01:24.0345400Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:01:24.0345694Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:01:24.0345806Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:01:24.0345811Z 2025-12-04T09:01:24.0345915Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0346154Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0346227Z res = mod(**inputs) 2025-12-04T09:01:24.0346517Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0346606Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0346888Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0346968Z outputs = layer_module( 2025-12-04T09:01:24.0347245Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0347324Z outputs = self.rel_attn( 2025-12-04T09:01:24.0347600Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:01:24.0347715Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:01:24.0348026Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:01:24.0348144Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:01:24.0348148Z 2025-12-04T09:01:24.0348259Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0348483Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0348551Z res = mod(**inputs) 2025-12-04T09:01:24.0348836Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0348925Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0349200Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0349284Z outputs = layer_module( 2025-12-04T09:01:24.0349564Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:01:24.0349796Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:01:24.0350086Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:01:24.0350171Z return forward_fn(*input_tensors) 2025-12-04T09:01:24.0350456Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:01:24.0350532Z output_x = self.ff(output_x) 2025-12-04T09:01:24.0350805Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-12-04T09:01:24.0350892Z output = self.layer_1(output) 2025-12-04T09:01:24.0350896Z 2025-12-04T09:01:24.0351009Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0351250Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0351320Z res = mod(**inputs) 2025-12-04T09:01:24.0351596Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0351711Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0351987Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0352065Z outputs = layer_module( 2025-12-04T09:01:24.0352338Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:01:24.0352561Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:01:24.0352866Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:01:24.0352952Z return forward_fn(*input_tensors) 2025-12-04T09:01:24.0353228Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:01:24.0353312Z output_x = self.ff(output_x) 2025-12-04T09:01:24.0353588Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-12-04T09:01:24.0353691Z output = self.activation_function(output) 2025-12-04T09:01:24.0353921Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:01:24.0353996Z return self.act(input) 2025-12-04T09:01:24.0354000Z 2025-12-04T09:01:24.0354133Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0354348Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0354422Z res = mod(**inputs) 2025-12-04T09:01:24.0354715Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0354804Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0355092Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0355167Z outputs = layer_module( 2025-12-04T09:01:24.0355444Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:01:24.0355677Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:01:24.0355966Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:01:24.0356057Z return forward_fn(*input_tensors) 2025-12-04T09:01:24.0356339Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:01:24.0356416Z output_x = self.ff(output_x) 2025-12-04T09:01:24.0356701Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-12-04T09:01:24.0356782Z output = self.layer_2(output) 2025-12-04T09:01:24.0356786Z 2025-12-04T09:01:24.0356904Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0357120Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0357190Z res = mod(**inputs) 2025-12-04T09:01:24.0357477Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0357566Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0357851Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0357948Z outputs = layer_module( 2025-12-04T09:01:24.0358293Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0358382Z outputs = self.rel_attn( 2025-12-04T09:01:24.0358698Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-12-04T09:01:24.0358808Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-12-04T09:01:24.0358812Z 2025-12-04T09:01:24.0358935Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0359158Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0359240Z res = mod(**inputs) 2025-12-04T09:01:24.0359547Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0359642Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0359952Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0360038Z outputs = layer_module( 2025-12-04T09:01:24.0360328Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0360413Z outputs = self.rel_attn( 2025-12-04T09:01:24.0360690Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-12-04T09:01:24.0360803Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-12-04T09:01:24.0360806Z 2025-12-04T09:01:24.0360931Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0361146Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0361224Z res = mod(**inputs) 2025-12-04T09:01:24.0361506Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0361595Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0361879Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0361954Z outputs = layer_module( 2025-12-04T09:01:24.0362236Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0362310Z outputs = self.rel_attn( 2025-12-04T09:01:24.0362584Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:01:24.0362671Z attn_vec = self.rel_attn_core( 2025-12-04T09:01:24.0362985Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-12-04T09:01:24.0363128Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-12-04T09:01:24.0363131Z 2025-12-04T09:01:24.0363236Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0363438Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0363512Z res = mod(**inputs) 2025-12-04T09:01:24.0363789Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0363877Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0364162Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0364235Z outputs = layer_module( 2025-12-04T09:01:24.0364520Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0364612Z outputs = self.rel_attn( 2025-12-04T09:01:24.0364887Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-12-04T09:01:24.0365036Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-12-04T09:01:24.0365060Z 2025-12-04T09:01:24.0365171Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0365393Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0365462Z res = mod(**inputs) 2025-12-04T09:01:24.0365752Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0365851Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0366147Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0366220Z outputs = layer_module( 2025-12-04T09:01:24.0366505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0366573Z outputs = self.rel_attn( 2025-12-04T09:01:24.0366841Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:01:24.0366917Z attn_vec = self.rel_attn_core( 2025-12-04T09:01:24.0367199Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-12-04T09:01:24.0367339Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-12-04T09:01:24.0367342Z 2025-12-04T09:01:24.0367463Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0367686Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0367755Z res = mod(**inputs) 2025-12-04T09:01:24.0368038Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0368133Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0368416Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0386595Z outputs = layer_module( 2025-12-04T09:01:24.0387126Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0387219Z outputs = self.rel_attn( 2025-12-04T09:01:24.0387541Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-12-04T09:01:24.0387695Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-12-04T09:01:24.0387708Z 2025-12-04T09:01:24.0387834Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0388080Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0388157Z res = mod(**inputs) 2025-12-04T09:01:24.0388459Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0388566Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0388857Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0388945Z outputs = layer_module( 2025-12-04T09:01:24.0389230Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0389311Z outputs = self.rel_attn( 2025-12-04T09:01:24.0389599Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:01:24.0389683Z attn_vec = self.rel_attn_core( 2025-12-04T09:01:24.0390109Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-12-04T09:01:24.0390254Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-12-04T09:01:24.0390339Z 2025-12-04T09:01:24.0390462Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0390698Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0390771Z res = mod(**inputs) 2025-12-04T09:01:24.0391063Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0391162Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0391474Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0391562Z outputs = layer_module( 2025-12-04T09:01:24.0391850Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0391927Z outputs = self.rel_attn( 2025-12-04T09:01:24.0392213Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:01:24.0392317Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:01:24.0392629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:01:24.0392756Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:01:24.0392760Z 2025-12-04T09:01:24.0392909Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0393142Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0393211Z res = mod(**inputs) 2025-12-04T09:01:24.0393501Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0393596Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0393873Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0393959Z outputs = layer_module( 2025-12-04T09:01:24.0394238Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0394311Z outputs = self.rel_attn( 2025-12-04T09:01:24.0394595Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:01:24.0394697Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:01:24.0395006Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:01:24.0395131Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:01:24.0395135Z 2025-12-04T09:01:24.0395246Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0395473Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0395544Z res = mod(**inputs) 2025-12-04T09:01:24.0395830Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0395922Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0396196Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0396277Z outputs = layer_module( 2025-12-04T09:01:24.0396553Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:01:24.0396802Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:01:24.0397104Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:01:24.0397209Z return forward_fn(*input_tensors) 2025-12-04T09:01:24.0397494Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:01:24.0397576Z output_x = self.ff(output_x) 2025-12-04T09:01:24.0397854Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-12-04T09:01:24.0397944Z output = self.layer_1(output) 2025-12-04T09:01:24.0397950Z 2025-12-04T09:01:24.0398067Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0398419Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0398502Z res = mod(**inputs) 2025-12-04T09:01:24.0398790Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0398894Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0399183Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0399260Z outputs = layer_module( 2025-12-04T09:01:24.0399552Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:01:24.0399786Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:01:24.0400121Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:01:24.0400208Z return forward_fn(*input_tensors) 2025-12-04T09:01:24.0400489Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:01:24.0400580Z output_x = self.ff(output_x) 2025-12-04T09:01:24.0400857Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-12-04T09:01:24.0400963Z output = self.activation_function(output) 2025-12-04T09:01:24.0401200Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:01:24.0401278Z return self.act(input) 2025-12-04T09:01:24.0401282Z 2025-12-04T09:01:24.0401405Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0401625Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0401699Z res = mod(**inputs) 2025-12-04T09:01:24.0401992Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0402083Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0402417Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0402495Z outputs = layer_module( 2025-12-04T09:01:24.0402768Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:01:24.0402999Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:01:24.0403287Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:01:24.0403380Z return forward_fn(*input_tensors) 2025-12-04T09:01:24.0403661Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:01:24.0403767Z output_x = self.ff(output_x) 2025-12-04T09:01:24.0404050Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-12-04T09:01:24.0404130Z output = self.layer_2(output) 2025-12-04T09:01:24.0404152Z 2025-12-04T09:01:24.0404267Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0404492Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0404562Z res = mod(**inputs) 2025-12-04T09:01:24.0404847Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0404940Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0405236Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0405319Z outputs = layer_module( 2025-12-04T09:01:24.0405594Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0405671Z outputs = self.rel_attn( 2025-12-04T09:01:24.0405956Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-12-04T09:01:24.0406069Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-12-04T09:01:24.0406072Z 2025-12-04T09:01:24.0406191Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0406405Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0406476Z res = mod(**inputs) 2025-12-04T09:01:24.0406789Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0406877Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0407166Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0407238Z outputs = layer_module( 2025-12-04T09:01:24.0407516Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0407603Z outputs = self.rel_attn( 2025-12-04T09:01:24.0407881Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-12-04T09:01:24.0407993Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-12-04T09:01:24.0408004Z 2025-12-04T09:01:24.0408114Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0408329Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0408407Z res = mod(**inputs) 2025-12-04T09:01:24.0408688Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0408776Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0409065Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0409140Z outputs = layer_module( 2025-12-04T09:01:24.0409439Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0409515Z outputs = self.rel_attn( 2025-12-04T09:01:24.0409793Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:01:24.0409884Z attn_vec = self.rel_attn_core( 2025-12-04T09:01:24.0410197Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-12-04T09:01:24.0410367Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-12-04T09:01:24.0410378Z 2025-12-04T09:01:24.0410492Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0410705Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0410800Z res = mod(**inputs) 2025-12-04T09:01:24.0411081Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0411172Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0411459Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0411533Z outputs = layer_module( 2025-12-04T09:01:24.0411838Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0411913Z outputs = self.rel_attn( 2025-12-04T09:01:24.0412189Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-12-04T09:01:24.0412341Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-12-04T09:01:24.0412345Z 2025-12-04T09:01:24.0412455Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0412667Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0412744Z res = mod(**inputs) 2025-12-04T09:01:24.0413020Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0413115Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0413410Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0413483Z outputs = layer_module( 2025-12-04T09:01:24.0413768Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0413843Z outputs = self.rel_attn( 2025-12-04T09:01:24.0414125Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:01:24.0414206Z attn_vec = self.rel_attn_core( 2025-12-04T09:01:24.0414503Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-12-04T09:01:24.0414652Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-12-04T09:01:24.0414656Z 2025-12-04T09:01:24.0414767Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0414981Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0415059Z res = mod(**inputs) 2025-12-04T09:01:24.0415347Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0415448Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0415738Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0415813Z outputs = layer_module( 2025-12-04T09:01:24.0416096Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0416172Z outputs = self.rel_attn( 2025-12-04T09:01:24.0416459Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-12-04T09:01:24.0416579Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-12-04T09:01:24.0416583Z 2025-12-04T09:01:24.0416692Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0416914Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0417002Z res = mod(**inputs) 2025-12-04T09:01:24.0417282Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0417379Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0417674Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0417755Z outputs = layer_module( 2025-12-04T09:01:24.0418031Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0418106Z outputs = self.rel_attn( 2025-12-04T09:01:24.0418393Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:01:24.0418488Z attn_vec = self.rel_attn_core( 2025-12-04T09:01:24.0418790Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-12-04T09:01:24.0418939Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-12-04T09:01:24.0418943Z 2025-12-04T09:01:24.0419057Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0419286Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0419359Z res = mod(**inputs) 2025-12-04T09:01:24.0419645Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0419747Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0420053Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0420138Z outputs = layer_module( 2025-12-04T09:01:24.0420422Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0420499Z outputs = self.rel_attn( 2025-12-04T09:01:24.0420945Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:01:24.0421058Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:01:24.0421367Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:01:24.0421515Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:01:24.0421519Z 2025-12-04T09:01:24.0421636Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0421849Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0421933Z res = mod(**inputs) 2025-12-04T09:01:24.0422211Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0422309Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0422582Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0422658Z outputs = layer_module( 2025-12-04T09:01:24.0422943Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0423017Z outputs = self.rel_attn( 2025-12-04T09:01:24.0423292Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:01:24.0423399Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:01:24.0423701Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:01:24.0423829Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:01:24.0423891Z 2025-12-04T09:01:24.0424002Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0424215Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0424344Z res = mod(**inputs) 2025-12-04T09:01:24.0424628Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0424726Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0425011Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0425084Z outputs = layer_module( 2025-12-04T09:01:24.0425395Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:01:24.0425623Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:01:24.0425916Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:01:24.0426001Z return forward_fn(*input_tensors) 2025-12-04T09:01:24.0426281Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:01:24.0426368Z output_x = self.ff(output_x) 2025-12-04T09:01:24.0426643Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-12-04T09:01:24.0426721Z output = self.layer_1(output) 2025-12-04T09:01:24.0426725Z 2025-12-04T09:01:24.0426878Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0427095Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0427173Z res = mod(**inputs) 2025-12-04T09:01:24.0427454Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0427548Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0427841Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0427917Z outputs = layer_module( 2025-12-04T09:01:24.0428201Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:01:24.0428449Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:01:24.0428732Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:01:24.0428829Z return forward_fn(*input_tensors) 2025-12-04T09:01:24.0429108Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:01:24.0429185Z output_x = self.ff(output_x) 2025-12-04T09:01:24.0429469Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-12-04T09:01:24.0429568Z output = self.activation_function(output) 2025-12-04T09:01:24.0429815Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:01:24.0429894Z return self.act(input) 2025-12-04T09:01:24.0429898Z 2025-12-04T09:01:24.0430013Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0430241Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0430315Z res = mod(**inputs) 2025-12-04T09:01:24.0430604Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0430724Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0431013Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0431096Z outputs = layer_module( 2025-12-04T09:01:24.0431396Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:01:24.0431625Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:01:24.0431924Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:01:24.0432010Z return forward_fn(*input_tensors) 2025-12-04T09:01:24.0432317Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:01:24.0432397Z output_x = self.ff(output_x) 2025-12-04T09:01:24.0432685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-12-04T09:01:24.0432773Z output = self.layer_2(output) 2025-12-04T09:01:24.0432777Z 2025-12-04T09:01:24.0432895Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0433124Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0433194Z res = mod(**inputs) 2025-12-04T09:01:24.0433481Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0433577Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0433883Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0433959Z outputs = layer_module( 2025-12-04T09:01:24.0434257Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0434335Z outputs = self.rel_attn( 2025-12-04T09:01:24.0434628Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-12-04T09:01:24.0434741Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-12-04T09:01:24.0434744Z 2025-12-04T09:01:24.0434856Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0435088Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0435159Z res = mod(**inputs) 2025-12-04T09:01:24.0435447Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0435548Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0435835Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0435918Z outputs = layer_module( 2025-12-04T09:01:24.0436205Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0436285Z outputs = self.rel_attn( 2025-12-04T09:01:24.0436576Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-12-04T09:01:24.0436687Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-12-04T09:01:24.0436691Z 2025-12-04T09:01:24.0436811Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0437032Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0437105Z res = mod(**inputs) 2025-12-04T09:01:24.0437399Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0437508Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0437797Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0437879Z outputs = layer_module( 2025-12-04T09:01:24.0438245Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0438337Z outputs = self.rel_attn( 2025-12-04T09:01:24.0438635Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:01:24.0438715Z attn_vec = self.rel_attn_core( 2025-12-04T09:01:24.0439027Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-12-04T09:01:24.0439197Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-12-04T09:01:24.0439201Z 2025-12-04T09:01:24.0439324Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0439544Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0439616Z res = mod(**inputs) 2025-12-04T09:01:24.0439909Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0440003Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0440284Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0440368Z outputs = layer_module( 2025-12-04T09:01:24.0440651Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0440762Z outputs = self.rel_attn( 2025-12-04T09:01:24.0441051Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-12-04T09:01:24.0441199Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-12-04T09:01:24.0441203Z 2025-12-04T09:01:24.0441323Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0441546Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0441626Z res = mod(**inputs) 2025-12-04T09:01:24.0441909Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0442000Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0442289Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0442359Z outputs = layer_module( 2025-12-04T09:01:24.0442619Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0442697Z outputs = self.rel_attn( 2025-12-04T09:01:24.0442957Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:01:24.0443038Z attn_vec = self.rel_attn_core( 2025-12-04T09:01:24.0443319Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-12-04T09:01:24.0443452Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-12-04T09:01:24.0443455Z 2025-12-04T09:01:24.0443565Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0443766Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0443833Z res = mod(**inputs) 2025-12-04T09:01:24.0444102Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0444202Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0444473Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0444542Z outputs = layer_module( 2025-12-04T09:01:24.0444821Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0444898Z outputs = self.rel_attn( 2025-12-04T09:01:24.0445157Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-12-04T09:01:24.0445267Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-12-04T09:01:24.0445273Z 2025-12-04T09:01:24.0445377Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0445596Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0445671Z res = mod(**inputs) 2025-12-04T09:01:24.0445933Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0446018Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0446285Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0446356Z outputs = layer_module( 2025-12-04T09:01:24.0446623Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0446692Z outputs = self.rel_attn( 2025-12-04T09:01:24.0446950Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:01:24.0447048Z attn_vec = self.rel_attn_core( 2025-12-04T09:01:24.0447329Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-12-04T09:01:24.0447462Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-12-04T09:01:24.0447466Z 2025-12-04T09:01:24.0447570Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0447771Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0447847Z res = mod(**inputs) 2025-12-04T09:01:24.0448109Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0448192Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0448465Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0448536Z outputs = layer_module( 2025-12-04T09:01:24.0448805Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0448876Z outputs = self.rel_attn( 2025-12-04T09:01:24.0449137Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:01:24.0449236Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:01:24.0449522Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:01:24.0449639Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:01:24.0449643Z 2025-12-04T09:01:24.0449746Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0449947Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0450021Z res = mod(**inputs) 2025-12-04T09:01:24.0450288Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0450388Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0450657Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0450727Z outputs = layer_module( 2025-12-04T09:01:24.0451008Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0451077Z outputs = self.rel_attn( 2025-12-04T09:01:24.0451334Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:01:24.0451434Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:01:24.0451717Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:01:24.0451860Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:01:24.0451864Z 2025-12-04T09:01:24.0451971Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0452173Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0452245Z res = mod(**inputs) 2025-12-04T09:01:24.0452514Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0452604Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0452887Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0452959Z outputs = layer_module( 2025-12-04T09:01:24.0453238Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:01:24.0453485Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:01:24.0453784Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:01:24.0453872Z return forward_fn(*input_tensors) 2025-12-04T09:01:24.0454132Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:01:24.0454226Z output_x = self.ff(output_x) 2025-12-04T09:01:24.0454488Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-12-04T09:01:24.0454562Z output = self.layer_1(output) 2025-12-04T09:01:24.0454573Z 2025-12-04T09:01:24.0454675Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0454876Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0454950Z res = mod(**inputs) 2025-12-04T09:01:24.0455212Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0455296Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0455568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0455640Z outputs = layer_module( 2025-12-04T09:01:24.0455907Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:01:24.0456118Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:01:24.0456385Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:01:24.0456473Z return forward_fn(*input_tensors) 2025-12-04T09:01:24.0456737Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:01:24.0456832Z output_x = self.ff(output_x) 2025-12-04T09:01:24.0457101Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-12-04T09:01:24.0457190Z output = self.activation_function(output) 2025-12-04T09:01:24.0457436Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:01:24.0457509Z return self.act(input) 2025-12-04T09:01:24.0457513Z 2025-12-04T09:01:24.0457618Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0457837Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0457904Z res = mod(**inputs) 2025-12-04T09:01:24.0458190Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0458275Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0458540Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0458614Z outputs = layer_module( 2025-12-04T09:01:24.0458872Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:01:24.0459086Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:01:24.0459362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:01:24.0459439Z return forward_fn(*input_tensors) 2025-12-04T09:01:24.0459723Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:01:24.0459797Z output_x = self.ff(output_x) 2025-12-04T09:01:24.0460063Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-12-04T09:01:24.0460145Z output = self.layer_2(output) 2025-12-04T09:01:24.0460148Z 2025-12-04T09:01:24.0460251Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0460464Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0460529Z res = mod(**inputs) 2025-12-04T09:01:24.0460794Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0460885Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0461152Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0461221Z outputs = layer_module( 2025-12-04T09:01:24.0461502Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0461572Z outputs = self.rel_attn( 2025-12-04T09:01:24.0461833Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-12-04T09:01:24.0461931Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-12-04T09:01:24.0461937Z 2025-12-04T09:01:24.0462036Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0462245Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0462308Z res = mod(**inputs) 2025-12-04T09:01:24.0462572Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0462656Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0462913Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0462985Z outputs = layer_module( 2025-12-04T09:01:24.0463260Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0463331Z outputs = self.rel_attn( 2025-12-04T09:01:24.0463592Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-12-04T09:01:24.0463709Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-12-04T09:01:24.0463713Z 2025-12-04T09:01:24.0463819Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0464015Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0464084Z res = mod(**inputs) 2025-12-04T09:01:24.0464359Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0464443Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0464698Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0464771Z outputs = layer_module( 2025-12-04T09:01:24.0465025Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0465102Z outputs = self.rel_attn( 2025-12-04T09:01:24.0465354Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:01:24.0465425Z attn_vec = self.rel_attn_core( 2025-12-04T09:01:24.0465705Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-12-04T09:01:24.0465852Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-12-04T09:01:24.0465856Z 2025-12-04T09:01:24.0465964Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0466162Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0466226Z res = mod(**inputs) 2025-12-04T09:01:24.0466492Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0466575Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0466830Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0466903Z outputs = layer_module( 2025-12-04T09:01:24.0467158Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0467235Z outputs = self.rel_attn( 2025-12-04T09:01:24.0467488Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-12-04T09:01:24.0467619Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-12-04T09:01:24.0467623Z 2025-12-04T09:01:24.0467731Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0467925Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0467999Z res = mod(**inputs) 2025-12-04T09:01:24.0468255Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0468339Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0468598Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0468668Z outputs = layer_module( 2025-12-04T09:01:24.0468922Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0468999Z outputs = self.rel_attn( 2025-12-04T09:01:24.0469651Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:01:24.0469736Z attn_vec = self.rel_attn_core( 2025-12-04T09:01:24.0470011Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-12-04T09:01:24.0470163Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-12-04T09:01:24.0470167Z 2025-12-04T09:01:24.0470279Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0470480Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0470563Z res = mod(**inputs) 2025-12-04T09:01:24.0470833Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0470915Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0471178Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0471244Z outputs = layer_module( 2025-12-04T09:01:24.0471495Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0471572Z outputs = self.rel_attn( 2025-12-04T09:01:24.0471826Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-12-04T09:01:24.0471930Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-12-04T09:01:24.0471934Z 2025-12-04T09:01:24.0472034Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0472244Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0472319Z res = mod(**inputs) 2025-12-04T09:01:24.0472575Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0472662Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0472913Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0472982Z outputs = layer_module( 2025-12-04T09:01:24.0473242Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0473310Z outputs = self.rel_attn( 2025-12-04T09:01:24.0473561Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:01:24.0473646Z attn_vec = self.rel_attn_core( 2025-12-04T09:01:24.0473925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-12-04T09:01:24.0474058Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-12-04T09:01:24.0474062Z 2025-12-04T09:01:24.0474166Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0474366Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0474443Z res = mod(**inputs) 2025-12-04T09:01:24.0474699Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0474782Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0475047Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0475117Z outputs = layer_module( 2025-12-04T09:01:24.0475395Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0475468Z outputs = self.rel_attn( 2025-12-04T09:01:24.0475761Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:01:24.0475867Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:01:24.0476164Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:01:24.0476318Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:01:24.0476322Z 2025-12-04T09:01:24.0476430Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0476643Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0476720Z res = mod(**inputs) 2025-12-04T09:01:24.0477012Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0477102Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0477387Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0477461Z outputs = layer_module( 2025-12-04T09:01:24.0477745Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0477821Z outputs = self.rel_attn( 2025-12-04T09:01:24.0478098Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:01:24.0478287Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:01:24.0478606Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:01:24.0478760Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:01:24.0478766Z 2025-12-04T09:01:24.0478881Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0479104Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0479187Z res = mod(**inputs) 2025-12-04T09:01:24.0479477Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0479580Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0479863Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0479935Z outputs = layer_module( 2025-12-04T09:01:24.0480220Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:01:24.0480447Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:01:24.0480735Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:01:24.0480829Z return forward_fn(*input_tensors) 2025-12-04T09:01:24.0481110Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:01:24.0481198Z output_x = self.ff(output_x) 2025-12-04T09:01:24.0481474Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-12-04T09:01:24.0481552Z output = self.layer_1(output) 2025-12-04T09:01:24.0481555Z 2025-12-04T09:01:24.0481675Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0481891Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0481962Z res = mod(**inputs) 2025-12-04T09:01:24.0482252Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0482356Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0482646Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0482720Z outputs = layer_module( 2025-12-04T09:01:24.0483014Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:01:24.0483246Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:01:24.0483529Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:01:24.0483621Z return forward_fn(*input_tensors) 2025-12-04T09:01:24.0483917Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:01:24.0483996Z output_x = self.ff(output_x) 2025-12-04T09:01:24.0484279Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-12-04T09:01:24.0484373Z output = self.activation_function(output) 2025-12-04T09:01:24.0484604Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:01:24.0484689Z return self.act(input) 2025-12-04T09:01:24.0484692Z 2025-12-04T09:01:24.0484802Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0485022Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0485090Z res = mod(**inputs) 2025-12-04T09:01:24.0485364Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0485476Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0485752Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0485832Z outputs = layer_module( 2025-12-04T09:01:24.0486104Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:01:24.0486328Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:01:24.0486627Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:01:24.0486708Z return forward_fn(*input_tensors) 2025-12-04T09:01:24.0486984Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:01:24.0487068Z output_x = self.ff(output_x) 2025-12-04T09:01:24.0487342Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-12-04T09:01:24.0487429Z output = self.layer_2(output) 2025-12-04T09:01:24.0487433Z 2025-12-04T09:01:24.0487543Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0487755Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0487832Z res = mod(**inputs) 2025-12-04T09:01:24.0488109Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0488203Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0488478Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0488553Z outputs = layer_module( 2025-12-04T09:01:24.0488837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0488905Z outputs = self.rel_attn( 2025-12-04T09:01:24.0489175Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-12-04T09:01:24.0489284Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-12-04T09:01:24.0489287Z 2025-12-04T09:01:24.0489404Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0489608Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0489671Z res = mod(**inputs) 2025-12-04T09:01:24.0489924Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0490012Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0490283Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0490359Z outputs = layer_module( 2025-12-04T09:01:24.0490613Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0490683Z outputs = self.rel_attn( 2025-12-04T09:01:24.0490942Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-12-04T09:01:24.0491043Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-12-04T09:01:24.0491046Z 2025-12-04T09:01:24.0491147Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0491354Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0491417Z res = mod(**inputs) 2025-12-04T09:01:24.0491678Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0491775Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0492030Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0492105Z outputs = layer_module( 2025-12-04T09:01:24.0492358Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0492428Z outputs = self.rel_attn( 2025-12-04T09:01:24.0492685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:01:24.0492757Z attn_vec = self.rel_attn_core( 2025-12-04T09:01:24.0493033Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-12-04T09:01:24.0493165Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-12-04T09:01:24.0493168Z 2025-12-04T09:01:24.0493270Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0493474Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0493536Z res = mod(**inputs) 2025-12-04T09:01:24.0493796Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0493880Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0494131Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0494205Z outputs = layer_module( 2025-12-04T09:01:24.0494457Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0494524Z outputs = self.rel_attn( 2025-12-04T09:01:24.0494787Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-12-04T09:01:24.0494922Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-12-04T09:01:24.0494926Z 2025-12-04T09:01:24.0495085Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0495282Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0495345Z res = mod(**inputs) 2025-12-04T09:01:24.0495625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0495705Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0495966Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0496034Z outputs = layer_module( 2025-12-04T09:01:24.0496289Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0496379Z outputs = self.rel_attn( 2025-12-04T09:01:24.0496633Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:01:24.0496704Z attn_vec = self.rel_attn_core( 2025-12-04T09:01:24.0496982Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-12-04T09:01:24.0497112Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-12-04T09:01:24.0497116Z 2025-12-04T09:01:24.0497224Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0497419Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0497484Z res = mod(**inputs) 2025-12-04T09:01:24.0497748Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0497846Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0498117Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0498185Z outputs = layer_module( 2025-12-04T09:01:24.0498448Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0498528Z outputs = self.rel_attn( 2025-12-04T09:01:24.0498791Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-12-04T09:01:24.0498892Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-12-04T09:01:24.0498896Z 2025-12-04T09:01:24.0499005Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0499208Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0499282Z res = mod(**inputs) 2025-12-04T09:01:24.0499551Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0499633Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0499898Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0499964Z outputs = layer_module( 2025-12-04T09:01:24.0500228Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0500295Z outputs = self.rel_attn( 2025-12-04T09:01:24.0500549Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:01:24.0500628Z attn_vec = self.rel_attn_core( 2025-12-04T09:01:24.0500904Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-12-04T09:01:24.0501026Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-12-04T09:01:24.0501030Z 2025-12-04T09:01:24.0501162Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0501358Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0501429Z res = mod(**inputs) 2025-12-04T09:01:24.0501701Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0501781Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0502046Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0502112Z outputs = layer_module( 2025-12-04T09:01:24.0502375Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0502467Z outputs = self.rel_attn( 2025-12-04T09:01:24.0502719Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:01:24.0502815Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:01:24.0503085Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:01:24.0503195Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:01:24.0503199Z 2025-12-04T09:01:24.0503306Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0503500Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0503570Z res = mod(**inputs) 2025-12-04T09:01:24.0503821Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0503920Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0504198Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0504264Z outputs = layer_module( 2025-12-04T09:01:24.0504521Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0504598Z outputs = self.rel_attn( 2025-12-04T09:01:24.0504851Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:01:24.0504948Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:01:24.0505224Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:01:24.0505334Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:01:24.0505339Z 2025-12-04T09:01:24.0505450Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0505647Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0505720Z res = mod(**inputs) 2025-12-04T09:01:24.0505979Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0506062Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0506326Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0506392Z outputs = layer_module( 2025-12-04T09:01:24.0506649Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:01:24.0506863Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:01:24.0507129Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:01:24.0507213Z return forward_fn(*input_tensors) 2025-12-04T09:01:24.0507493Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:01:24.0507565Z output_x = self.ff(output_x) 2025-12-04T09:01:24.0507823Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-12-04T09:01:24.0507912Z output = self.layer_1(output) 2025-12-04T09:01:24.0507915Z 2025-12-04T09:01:24.0508021Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0508215Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0508279Z res = mod(**inputs) 2025-12-04T09:01:24.0508556Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0508638Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0508891Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0508964Z outputs = layer_module( 2025-12-04T09:01:24.0509214Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:01:24.0509424Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:01:24.0509683Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:01:24.0509758Z return forward_fn(*input_tensors) 2025-12-04T09:01:24.0510018Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:01:24.0510106Z output_x = self.ff(output_x) 2025-12-04T09:01:24.0510369Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-12-04T09:01:24.0510455Z output = self.activation_function(output) 2025-12-04T09:01:24.0510676Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:01:24.0510754Z return self.act(input) 2025-12-04T09:01:24.0510757Z 2025-12-04T09:01:24.0510859Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0511054Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0511123Z res = mod(**inputs) 2025-12-04T09:01:24.0511378Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0511467Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0511723Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0511793Z outputs = layer_module( 2025-12-04T09:01:24.0512061Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:01:24.0512273Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:01:24.0512551Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:01:24.0512629Z return forward_fn(*input_tensors) 2025-12-04T09:01:24.0512892Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:01:24.0512976Z output_x = self.ff(output_x) 2025-12-04T09:01:24.0513238Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-12-04T09:01:24.0513313Z output = self.layer_2(output) 2025-12-04T09:01:24.0513323Z 2025-12-04T09:01:24.0513442Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0513645Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0513716Z res = mod(**inputs) 2025-12-04T09:01:24.0513993Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0514076Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0514361Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0514433Z outputs = layer_module( 2025-12-04T09:01:24.0514725Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0514818Z outputs = self.rel_attn( 2025-12-04T09:01:24.0515096Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-12-04T09:01:24.0515211Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-12-04T09:01:24.0515214Z 2025-12-04T09:01:24.0515324Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0515541Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0515618Z res = mod(**inputs) 2025-12-04T09:01:24.0515894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0515990Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0516266Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0516358Z outputs = layer_module( 2025-12-04T09:01:24.0516646Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0516723Z outputs = self.rel_attn( 2025-12-04T09:01:24.0517007Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-12-04T09:01:24.0517123Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-12-04T09:01:24.0517128Z 2025-12-04T09:01:24.0517242Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0517466Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0517537Z res = mod(**inputs) 2025-12-04T09:01:24.0517824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0517925Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0518279Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0518375Z outputs = layer_module( 2025-12-04T09:01:24.0518665Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0518743Z outputs = self.rel_attn( 2025-12-04T09:01:24.0519035Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:01:24.0519117Z attn_vec = self.rel_attn_core( 2025-12-04T09:01:24.0519429Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-12-04T09:01:24.0519582Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-12-04T09:01:24.0519588Z 2025-12-04T09:01:24.0519699Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0519923Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0519993Z res = mod(**inputs) 2025-12-04T09:01:24.0520296Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0520394Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0520671Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0520994Z outputs = layer_module( 2025-12-04T09:01:24.0521282Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0521352Z outputs = self.rel_attn( 2025-12-04T09:01:24.0521618Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-12-04T09:01:24.0521796Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-12-04T09:01:24.0521800Z 2025-12-04T09:01:24.0521917Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0522123Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0522189Z res = mod(**inputs) 2025-12-04T09:01:24.0522462Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0522547Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0522807Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0522884Z outputs = layer_module( 2025-12-04T09:01:24.0523146Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0523257Z outputs = self.rel_attn( 2025-12-04T09:01:24.0523518Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:01:24.0523595Z attn_vec = self.rel_attn_core( 2025-12-04T09:01:24.0523885Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-12-04T09:01:24.0524017Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-12-04T09:01:24.0524022Z 2025-12-04T09:01:24.0524125Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0524333Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0524398Z res = mod(**inputs) 2025-12-04T09:01:24.0524675Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0524760Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0525026Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0525102Z outputs = layer_module( 2025-12-04T09:01:24.0525362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0525437Z outputs = self.rel_attn( 2025-12-04T09:01:24.0525696Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-12-04T09:01:24.0525797Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-12-04T09:01:24.0525800Z 2025-12-04T09:01:24.0525908Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0526107Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0526173Z res = mod(**inputs) 2025-12-04T09:01:24.0526445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0526528Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0526819Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0526887Z outputs = layer_module( 2025-12-04T09:01:24.0527149Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0527254Z outputs = self.rel_attn( 2025-12-04T09:01:24.0527516Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-12-04T09:01:24.0527596Z attn_vec = self.rel_attn_core( 2025-12-04T09:01:24.0527878Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-12-04T09:01:24.0528022Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-12-04T09:01:24.0528026Z 2025-12-04T09:01:24.0528138Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0528340Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0528406Z res = mod(**inputs) 2025-12-04T09:01:24.0528676Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0528762Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0529034Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0529102Z outputs = layer_module( 2025-12-04T09:01:24.0529364Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0529460Z outputs = self.rel_attn( 2025-12-04T09:01:24.0529722Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:01:24.0529816Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:01:24.0530111Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:01:24.0530223Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:01:24.0530228Z 2025-12-04T09:01:24.0530370Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0530570Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0530637Z res = mod(**inputs) 2025-12-04T09:01:24.0530907Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0530992Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0531264Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0531334Z outputs = layer_module( 2025-12-04T09:01:24.0531597Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-12-04T09:01:24.0531675Z outputs = self.rel_attn( 2025-12-04T09:01:24.0531934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-12-04T09:01:24.0532027Z output_h = self.post_attention(h, attn_vec) 2025-12-04T09:01:24.0532320Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-12-04T09:01:24.0532430Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-12-04T09:01:24.0532435Z 2025-12-04T09:01:24.0532545Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0532746Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0532813Z res = mod(**inputs) 2025-12-04T09:01:24.0533103Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0533189Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0533495Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0533579Z outputs = layer_module( 2025-12-04T09:01:24.0533842Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:01:24.0534058Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:01:24.0534345Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:01:24.0534434Z return forward_fn(*input_tensors) 2025-12-04T09:01:24.0534697Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:01:24.0534772Z output_x = self.ff(output_x) 2025-12-04T09:01:24.0535039Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-12-04T09:01:24.0535116Z output = self.layer_1(output) 2025-12-04T09:01:24.0535120Z 2025-12-04T09:01:24.0535221Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0535427Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0535492Z res = mod(**inputs) 2025-12-04T09:01:24.0535760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0535861Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0536124Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0536202Z outputs = layer_module( 2025-12-04T09:01:24.0536461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:01:24.0536676Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:01:24.0536952Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:01:24.0537029Z return forward_fn(*input_tensors) 2025-12-04T09:01:24.0537296Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:01:24.0537371Z output_x = self.ff(output_x) 2025-12-04T09:01:24.0537632Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-12-04T09:01:24.0537730Z output = self.activation_function(output) 2025-12-04T09:01:24.0537951Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:01:24.0538030Z return self.act(input) 2025-12-04T09:01:24.0538034Z 2025-12-04T09:01:24.0538138Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0538339Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0538410Z res = mod(**inputs) 2025-12-04T09:01:24.0538669Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1616, in forward 2025-12-04T09:01:24.0538751Z transformer_outputs = self.transformer( 2025-12-04T09:01:24.0539028Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-12-04T09:01:24.0539101Z outputs = layer_module( 2025-12-04T09:01:24.0539409Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-12-04T09:01:24.0539631Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-12-04T09:01:24.0539930Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:01:24.0540019Z return forward_fn(*input_tensors) 2025-12-04T09:01:24.0540293Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-12-04T09:01:24.0540378Z output_x = self.ff(output_x) 2025-12-04T09:01:24.0540648Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-12-04T09:01:24.0540741Z output = self.layer_2(output) 2025-12-04T09:01:24.0540745Z 2025-12-04T09:01:24.0540856Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0541058Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0541123Z res = mod(**inputs) 2025-12-04T09:01:24.0541392Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1633, in forward 2025-12-04T09:01:24.0541490Z logits = self.lm_loss(transformer_outputs[0]) 2025-12-04T09:01:24.0541494Z 2025-12-04T09:01:24.0541608Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:01:24.0541824Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:01:24.0541891Z res = mod(**inputs) 2025-12-04T09:01:24.0542201Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1639, in forward 2025-12-04T09:01:24.0542346Z loss = loss_fct(logits.view(-1, logits.size(-1)), labels.view(-1)) 2025-12-04T09:01:24.0542350Z 2025-12-04T09:01:39.4717636Z Compilation time (from dynamo_timed): 33.366333972 2025-12-04T09:01:39.4760447Z pass 2025-12-04T09:01:39.4760843Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:01:39.4764588Z TIMING: _recursive_pre_grad_passes:0.0134 _recursive_joint_graph_passes:1.36099 _recursive_post_grad_passes:0.57827 async_compile.wait:1.01806 code_gen:12.86408 inductor_compile:17.91431 backend_compile:27.12739 gc:0.00268 entire_frame_compile:33.36633 total_wall_time:33.36633 2025-12-04T09:01:39.4765618Z STATS: call_* op count: 818 | FakeTensorMode.__torch_dispatch__:32908 | FakeTensor.__torch_dispatch__:15989 | ProxyTorchDispatchMode.__torch_dispatch__:6774 2025-12-04T09:01:39.4766160Z Dynamo produced 1 graphs covering 818 ops with 0 graph breaks (0 unique) 2025-12-04T09:01:42.6829174Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/__init__.py:65: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. 2025-12-04T09:01:42.6830097Z import pynvml # type: ignore[import] 2025-12-04T09:01:46.1238379Z 2025-12-04T09:01:47.5195870Z loading model: 0it [00:00, ?it/s] 2025-12-04T09:01:47.5196218Z loading model: 0it [00:01, ?it/s] 2025-12-04T09:01:47.5216461Z cpu eval YituTechConvBert 2025-12-04T09:01:49.4409992Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:01:50.1453328Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:01:50.8377723Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:02:02.9669019Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:02.9669786Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:02.9670760Z res = mod(**inputs) 2025-12-04T09:02:02.9671505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:02.9672238Z generator_hidden_states = self.convbert( 2025-12-04T09:02:02.9673068Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:02.9673809Z hidden_states = self.encoder( 2025-12-04T09:02:02.9674571Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:02.9675296Z layer_outputs = layer_module( 2025-12-04T09:02:02.9676041Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:02.9676807Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:02.9677625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:02:02.9678541Z self_attention_outputs = self.attention( 2025-12-04T09:02:02.9679352Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:02:02.9680160Z self_outputs = self.self( 2025-12-04T09:02:02.9680890Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 349, in forward 2025-12-04T09:02:02.9681636Z mixed_query_layer = self.query(hidden_states) 2025-12-04T09:02:02.9681926Z 2025-12-04T09:02:02.9682104Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:02.9682830Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:02.9683437Z res = mod(**inputs) 2025-12-04T09:02:02.9684141Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:02.9684891Z generator_hidden_states = self.convbert( 2025-12-04T09:02:02.9685626Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:02.9686360Z hidden_states = self.encoder( 2025-12-04T09:02:02.9687022Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:02.9687801Z layer_outputs = layer_module( 2025-12-04T09:02:02.9688449Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:02.9689101Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:02.9689894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:02:02.9690679Z self_attention_outputs = self.attention( 2025-12-04T09:02:02.9691452Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:02:02.9692209Z self_outputs = self.self( 2025-12-04T09:02:02.9692920Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 343, in forward 2025-12-04T09:02:02.9693680Z mixed_key_layer = self.key(hidden_states) 2025-12-04T09:02:02.9693945Z 2025-12-04T09:02:02.9694128Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:02.9694822Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:02.9695447Z res = mod(**inputs) 2025-12-04T09:02:02.9696160Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:02.9696896Z generator_hidden_states = self.convbert( 2025-12-04T09:02:02.9697715Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:02.9698492Z hidden_states = self.encoder( 2025-12-04T09:02:02.9699261Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:02.9700071Z layer_outputs = layer_module( 2025-12-04T09:02:02.9700716Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:02.9701406Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:02.9702155Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:02:02.9702987Z self_attention_outputs = self.attention( 2025-12-04T09:02:02.9703811Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:02:02.9704595Z self_outputs = self.self( 2025-12-04T09:02:02.9705359Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 344, in forward 2025-12-04T09:02:02.9706167Z mixed_value_layer = self.value(hidden_states) 2025-12-04T09:02:02.9706407Z 2025-12-04T09:02:02.9706565Z cudagraph partition due to non gpu ops 2025-12-04T09:02:02.9706950Z cudagraph partition due to non gpu ops 2025-12-04T09:02:02.9707408Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:02.9708066Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:02.9708659Z res = mod(**inputs) 2025-12-04T09:02:02.9709418Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:02.9710193Z generator_hidden_states = self.convbert( 2025-12-04T09:02:02.9710996Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:02.9711738Z hidden_states = self.encoder( 2025-12-04T09:02:02.9712469Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:02.9713182Z layer_outputs = layer_module( 2025-12-04T09:02:02.9713849Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:02.9714543Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:02.9715278Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:02:02.9716063Z self_attention_outputs = self.attention( 2025-12-04T09:02:02.9716832Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:02:02.9717560Z self_outputs = self.self( 2025-12-04T09:02:02.9718402Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 365, in forward 2025-12-04T09:02:02.9719260Z conv_out_layer = self.conv_out_layer(hidden_states) 2025-12-04T09:02:02.9719533Z 2025-12-04T09:02:02.9719681Z cudagraph partition due to non gpu ops 2025-12-04T09:02:02.9720132Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:02.9720964Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:02.9721592Z res = mod(**inputs) 2025-12-04T09:02:02.9722263Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:02.9723082Z generator_hidden_states = self.convbert( 2025-12-04T09:02:02.9723982Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:02.9724729Z hidden_states = self.encoder( 2025-12-04T09:02:02.9725496Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:02.9726368Z layer_outputs = layer_module( 2025-12-04T09:02:02.9727010Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:02.9727645Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:02.9728396Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:02:02.9729151Z self_attention_outputs = self.attention( 2025-12-04T09:02:02.9729972Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:02:02.9730724Z self_outputs = self.self( 2025-12-04T09:02:02.9731460Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 346, in forward 2025-12-04T09:02:02.9732354Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-12-04T09:02:02.9733251Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 281, in forward 2025-12-04T09:02:02.9734030Z x = self.depthwise(hidden_states) 2025-12-04T09:02:02.9734237Z 2025-12-04T09:02:02.9734413Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:02.9735084Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:02.9735725Z res = mod(**inputs) 2025-12-04T09:02:02.9736405Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:02.9737198Z generator_hidden_states = self.convbert( 2025-12-04T09:02:02.9737964Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:02.9738733Z hidden_states = self.encoder( 2025-12-04T09:02:02.9739439Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:02.9740175Z layer_outputs = layer_module( 2025-12-04T09:02:02.9740834Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:02.9741527Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:02.9742287Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:02:02.9743044Z self_attention_outputs = self.attention( 2025-12-04T09:02:02.9743815Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:02:02.9744523Z self_outputs = self.self( 2025-12-04T09:02:02.9745252Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 346, in forward 2025-12-04T09:02:02.9746193Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-12-04T09:02:02.9747121Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 282, in forward 2025-12-04T09:02:02.9747821Z x = self.pointwise(x) 2025-12-04T09:02:02.9748005Z 2025-12-04T09:02:02.9748181Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:02.9748757Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:02.9749285Z res = mod(**inputs) 2025-12-04T09:02:02.9749970Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:02.9750661Z generator_hidden_states = self.convbert( 2025-12-04T09:02:02.9751358Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:02.9752094Z hidden_states = self.encoder( 2025-12-04T09:02:02.9752789Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:02.9753470Z layer_outputs = layer_module( 2025-12-04T09:02:02.9754048Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:02.9754663Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:02.9755415Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:02:02.9756172Z self_attention_outputs = self.attention( 2025-12-04T09:02:02.9756919Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:02:02.9757670Z self_outputs = self.self( 2025-12-04T09:02:02.9758455Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 359, in forward 2025-12-04T09:02:02.9759337Z conv_attn_layer = torch.multiply(mixed_key_conv_attn_layer, mixed_query_layer) 2025-12-04T09:02:02.9759771Z 2025-12-04T09:02:02.9759943Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:02.9760580Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:02.9761214Z res = mod(**inputs) 2025-12-04T09:02:02.9761889Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:02.9762611Z generator_hidden_states = self.convbert( 2025-12-04T09:02:02.9763329Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:02.9764038Z hidden_states = self.encoder( 2025-12-04T09:02:02.9764745Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:02.9765459Z layer_outputs = layer_module( 2025-12-04T09:02:02.9766090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:02.9766672Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:02.9767382Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:02:02.9768152Z self_attention_outputs = self.attention( 2025-12-04T09:02:02.9768901Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:02:02.9769647Z self_outputs = self.self( 2025-12-04T09:02:02.9770361Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 361, in forward 2025-12-04T09:02:02.9771164Z conv_kernel_layer = self.conv_kernel_layer(conv_attn_layer) 2025-12-04T09:02:02.9771481Z 2025-12-04T09:02:02.9771659Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:02.9772300Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:02.9772829Z res = mod(**inputs) 2025-12-04T09:02:02.9773452Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:02.9774190Z generator_hidden_states = self.convbert( 2025-12-04T09:02:02.9774975Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:02.9775705Z hidden_states = self.encoder( 2025-12-04T09:02:02.9776394Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:02.9777116Z layer_outputs = layer_module( 2025-12-04T09:02:02.9777679Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:02.9778263Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:02.9778946Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:02:02.9779695Z self_attention_outputs = self.attention( 2025-12-04T09:02:02.9780461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:02:02.9781125Z self_outputs = self.self( 2025-12-04T09:02:02.9781736Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 379, in forward 2025-12-04T09:02:02.9782451Z conv_out_layer = torch.matmul(conv_out_layer, conv_kernel_layer) 2025-12-04T09:02:02.9782722Z 2025-12-04T09:02:02.9782834Z cudagraph partition due to non gpu ops 2025-12-04T09:02:02.9783134Z cudagraph partition due to non gpu ops 2025-12-04T09:02:02.9783511Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:02.9784213Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:02.9784800Z res = mod(**inputs) 2025-12-04T09:02:02.9785548Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:02.9786418Z generator_hidden_states = self.convbert( 2025-12-04T09:02:02.9787221Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:02.9787960Z hidden_states = self.encoder( 2025-12-04T09:02:02.9788568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:02.9789264Z layer_outputs = layer_module( 2025-12-04T09:02:02.9789866Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:02.9790488Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:02.9791228Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:02:02.9791986Z self_attention_outputs = self.attention( 2025-12-04T09:02:02.9792740Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:02:02.9793506Z self_outputs = self.self( 2025-12-04T09:02:02.9794232Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 404, in forward 2025-12-04T09:02:02.9795059Z context_layer = torch.cat([context_layer, conv_out], 2) 2025-12-04T09:02:02.9795361Z 2025-12-04T09:02:02.9795547Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:02.9796204Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:02.9796758Z res = mod(**inputs) 2025-12-04T09:02:02.9797409Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:02.9798138Z generator_hidden_states = self.convbert( 2025-12-04T09:02:02.9799037Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:02.9799809Z hidden_states = self.encoder( 2025-12-04T09:02:02.9800633Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:02.9801383Z layer_outputs = layer_module( 2025-12-04T09:02:02.9802023Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:02.9802743Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:02.9803505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:02:02.9804289Z self_attention_outputs = self.attention( 2025-12-04T09:02:02.9805008Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 470, in forward 2025-12-04T09:02:02.9805873Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:02:02.9806695Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 424, in forward 2025-12-04T09:02:02.9807442Z hidden_states = self.dense(hidden_states) 2025-12-04T09:02:02.9807686Z 2025-12-04T09:02:02.9807858Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:02.9808498Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:02.9809086Z res = mod(**inputs) 2025-12-04T09:02:02.9809785Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:02.9810501Z generator_hidden_states = self.convbert( 2025-12-04T09:02:02.9811174Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:02.9811890Z hidden_states = self.encoder( 2025-12-04T09:02:02.9812577Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:02.9813291Z layer_outputs = layer_module( 2025-12-04T09:02:02.9813807Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:02.9814436Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:02.9815154Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 585, in forward 2025-12-04T09:02:02.9815902Z layer_output = apply_chunking_to_forward( 2025-12-04T09:02:02.9816604Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:02:02.9817288Z return forward_fn(*input_tensors) 2025-12-04T09:02:02.9818018Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 592, in feed_forward_chunk 2025-12-04T09:02:02.9818893Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:02:02.9819653Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 512, in forward 2025-12-04T09:02:02.9820372Z hidden_states = self.dense(hidden_states) 2025-12-04T09:02:02.9820608Z 2025-12-04T09:02:02.9820942Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:02.9821604Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:02.9822162Z res = mod(**inputs) 2025-12-04T09:02:02.9822811Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:02.9823588Z generator_hidden_states = self.convbert( 2025-12-04T09:02:02.9824365Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:02.9825111Z hidden_states = self.encoder( 2025-12-04T09:02:02.9825983Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:02.9826711Z layer_outputs = layer_module( 2025-12-04T09:02:02.9827349Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:02.9828086Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:02.9828863Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 585, in forward 2025-12-04T09:02:02.9829636Z layer_output = apply_chunking_to_forward( 2025-12-04T09:02:02.9830398Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:02:02.9831264Z return forward_fn(*input_tensors) 2025-12-04T09:02:02.9832142Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 592, in feed_forward_chunk 2025-12-04T09:02:02.9833091Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:02:02.9833974Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 513, in forward 2025-12-04T09:02:02.9834855Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:02:02.9835607Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:02:02.9836288Z return self.act(input) 2025-12-04T09:02:02.9836499Z 2025-12-04T09:02:02.9836693Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:02.9837404Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:02.9838093Z res = mod(**inputs) 2025-12-04T09:02:02.9838973Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:02.9839827Z generator_hidden_states = self.convbert( 2025-12-04T09:02:02.9840576Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:02.9841321Z hidden_states = self.encoder( 2025-12-04T09:02:02.9842036Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:02.9842760Z layer_outputs = layer_module( 2025-12-04T09:02:02.9843363Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:02.9844020Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:02.9844775Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 585, in forward 2025-12-04T09:02:02.9845526Z layer_output = apply_chunking_to_forward( 2025-12-04T09:02:02.9846239Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:02:02.9846956Z return forward_fn(*input_tensors) 2025-12-04T09:02:02.9847729Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-12-04T09:02:02.9848618Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:02:02.9849467Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 530, in forward 2025-12-04T09:02:02.9850214Z hidden_states = self.dense(hidden_states) 2025-12-04T09:02:02.9850461Z 2025-12-04T09:02:02.9850650Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:02.9851294Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:02.9851873Z res = mod(**inputs) 2025-12-04T09:02:02.9852639Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:02.9853413Z generator_hidden_states = self.convbert( 2025-12-04T09:02:02.9854134Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:02.9854886Z hidden_states = self.encoder( 2025-12-04T09:02:02.9855589Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:02.9856300Z layer_outputs = layer_module( 2025-12-04T09:02:02.9856885Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:02.9857566Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:02.9858298Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:02:02.9859021Z self_attention_outputs = self.attention( 2025-12-04T09:02:02.9859758Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:02:02.9860493Z self_outputs = self.self( 2025-12-04T09:02:02.9861191Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 349, in forward 2025-12-04T09:02:02.9861960Z mixed_query_layer = self.query(hidden_states) 2025-12-04T09:02:02.9862235Z 2025-12-04T09:02:02.9862400Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:02.9863063Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:02.9863695Z res = mod(**inputs) 2025-12-04T09:02:02.9864413Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:02.9865198Z generator_hidden_states = self.convbert( 2025-12-04T09:02:02.9865981Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:02.9866726Z hidden_states = self.encoder( 2025-12-04T09:02:02.9867463Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:02.9868207Z layer_outputs = layer_module( 2025-12-04T09:02:02.9868810Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:02.9869482Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:02.9870297Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:02:02.9871122Z self_attention_outputs = self.attention( 2025-12-04T09:02:02.9871881Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:02:02.9872635Z self_outputs = self.self( 2025-12-04T09:02:02.9873385Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 343, in forward 2025-12-04T09:02:02.9874182Z mixed_key_layer = self.key(hidden_states) 2025-12-04T09:02:02.9874403Z 2025-12-04T09:02:02.9874554Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:02.9875168Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:02.9875797Z res = mod(**inputs) 2025-12-04T09:02:02.9876496Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:02.9877300Z generator_hidden_states = self.convbert( 2025-12-04T09:02:02.9878181Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:02.9879060Z hidden_states = self.encoder( 2025-12-04T09:02:02.9879824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:02.9880707Z layer_outputs = layer_module( 2025-12-04T09:02:02.9881382Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:02.9882096Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:02.9882885Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:02:02.9883694Z self_attention_outputs = self.attention( 2025-12-04T09:02:02.9884550Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:02:02.9885341Z self_outputs = self.self( 2025-12-04T09:02:02.9886120Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 344, in forward 2025-12-04T09:02:02.9886929Z mixed_value_layer = self.value(hidden_states) 2025-12-04T09:02:02.9887210Z 2025-12-04T09:02:02.9887360Z cudagraph partition due to non gpu ops 2025-12-04T09:02:02.9887746Z cudagraph partition due to non gpu ops 2025-12-04T09:02:02.9888196Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:02.9888890Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:02.9889502Z res = mod(**inputs) 2025-12-04T09:02:02.9890263Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:02.9891141Z generator_hidden_states = self.convbert( 2025-12-04T09:02:02.9891976Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:02.9892787Z hidden_states = self.encoder( 2025-12-04T09:02:02.9893572Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:02.9894360Z layer_outputs = layer_module( 2025-12-04T09:02:02.9895048Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:02.9895749Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:02.9896579Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:02:02.9897410Z self_attention_outputs = self.attention( 2025-12-04T09:02:02.9898230Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:02:02.9899025Z self_outputs = self.self( 2025-12-04T09:02:02.9899790Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 365, in forward 2025-12-04T09:02:02.9900627Z conv_out_layer = self.conv_out_layer(hidden_states) 2025-12-04T09:02:02.9900938Z 2025-12-04T09:02:02.9901082Z cudagraph partition due to non gpu ops 2025-12-04T09:02:02.9901512Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:02.9902191Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:02.9902791Z res = mod(**inputs) 2025-12-04T09:02:02.9903483Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:02.9904250Z generator_hidden_states = self.convbert( 2025-12-04T09:02:02.9905024Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:02.9905830Z hidden_states = self.encoder( 2025-12-04T09:02:02.9906575Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:02.9907334Z layer_outputs = layer_module( 2025-12-04T09:02:02.9908019Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:02.9908674Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:02.9909441Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:02:02.9910223Z self_attention_outputs = self.attention( 2025-12-04T09:02:02.9911037Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:02:02.9911780Z self_outputs = self.self( 2025-12-04T09:02:02.9912497Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 346, in forward 2025-12-04T09:02:02.9913381Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-12-04T09:02:02.9914269Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 281, in forward 2025-12-04T09:02:02.9915022Z x = self.depthwise(hidden_states) 2025-12-04T09:02:02.9915255Z 2025-12-04T09:02:02.9915451Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:02.9916111Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:02.9916696Z res = mod(**inputs) 2025-12-04T09:02:02.9917459Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:02.9918345Z generator_hidden_states = self.convbert( 2025-12-04T09:02:02.9919187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:02.9920022Z hidden_states = self.encoder( 2025-12-04T09:02:02.9920905Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:02.9921679Z layer_outputs = layer_module( 2025-12-04T09:02:02.9922309Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:02.9922994Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:02.9923758Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:02:02.9924530Z self_attention_outputs = self.attention( 2025-12-04T09:02:02.9925316Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:02:02.9926080Z self_outputs = self.self( 2025-12-04T09:02:02.9926820Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 346, in forward 2025-12-04T09:02:02.9927739Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-12-04T09:02:02.9928670Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 282, in forward 2025-12-04T09:02:02.9929417Z x = self.pointwise(x) 2025-12-04T09:02:02.9929608Z 2025-12-04T09:02:02.9930333Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:02.9930998Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:02.9931599Z res = mod(**inputs) 2025-12-04T09:02:02.9932317Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:02.9933243Z generator_hidden_states = self.convbert( 2025-12-04T09:02:02.9934028Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:02.9934765Z hidden_states = self.encoder( 2025-12-04T09:02:02.9935538Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:02.9936253Z layer_outputs = layer_module( 2025-12-04T09:02:02.9936890Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:02.9937538Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:02.9938299Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:02:02.9939040Z self_attention_outputs = self.attention( 2025-12-04T09:02:02.9939773Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:02:02.9940479Z self_outputs = self.self( 2025-12-04T09:02:02.9941170Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 359, in forward 2025-12-04T09:02:02.9942036Z conv_attn_layer = torch.multiply(mixed_key_conv_attn_layer, mixed_query_layer) 2025-12-04T09:02:02.9942421Z 2025-12-04T09:02:02.9942601Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:02.9943246Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:02.9943806Z res = mod(**inputs) 2025-12-04T09:02:02.9944545Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:02.9945309Z generator_hidden_states = self.convbert( 2025-12-04T09:02:02.9946040Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:02.9946751Z hidden_states = self.encoder( 2025-12-04T09:02:02.9947462Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:02.9948177Z layer_outputs = layer_module( 2025-12-04T09:02:02.9948789Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:02.9949430Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:02.9950100Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:02:02.9950796Z self_attention_outputs = self.attention( 2025-12-04T09:02:02.9951516Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:02:02.9952233Z self_outputs = self.self( 2025-12-04T09:02:02.9952900Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 361, in forward 2025-12-04T09:02:02.9953637Z conv_kernel_layer = self.conv_kernel_layer(conv_attn_layer) 2025-12-04T09:02:02.9953934Z 2025-12-04T09:02:02.9954099Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:02.9954728Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:02.9955274Z res = mod(**inputs) 2025-12-04T09:02:02.9955908Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:02.9956635Z generator_hidden_states = self.convbert( 2025-12-04T09:02:02.9957390Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:02.9958188Z hidden_states = self.encoder( 2025-12-04T09:02:02.9959045Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:02.9959857Z layer_outputs = layer_module( 2025-12-04T09:02:02.9960593Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:02.9961238Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:02.9961956Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:02:02.9962717Z self_attention_outputs = self.attention( 2025-12-04T09:02:02.9963518Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:02:02.9964241Z self_outputs = self.self( 2025-12-04T09:02:02.9964880Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 379, in forward 2025-12-04T09:02:02.9965670Z conv_out_layer = torch.matmul(conv_out_layer, conv_kernel_layer) 2025-12-04T09:02:02.9965984Z 2025-12-04T09:02:02.9966129Z cudagraph partition due to non gpu ops 2025-12-04T09:02:02.9966480Z cudagraph partition due to non gpu ops 2025-12-04T09:02:02.9966881Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:02.9967512Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:02.9968056Z res = mod(**inputs) 2025-12-04T09:02:02.9968731Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:02.9969520Z generator_hidden_states = self.convbert( 2025-12-04T09:02:02.9970263Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:02.9970998Z hidden_states = self.encoder( 2025-12-04T09:02:02.9971706Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:02.9972425Z layer_outputs = layer_module( 2025-12-04T09:02:02.9973039Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:02.9973670Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:02.9974404Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:02:02.9975162Z self_attention_outputs = self.attention( 2025-12-04T09:02:02.9975897Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:02:02.9976617Z self_outputs = self.self( 2025-12-04T09:02:02.9977302Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 404, in forward 2025-12-04T09:02:02.9978073Z context_layer = torch.cat([context_layer, conv_out], 2) 2025-12-04T09:02:02.9978351Z 2025-12-04T09:02:02.9978526Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:02.9979152Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:02.9979704Z res = mod(**inputs) 2025-12-04T09:02:02.9980354Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:02.9981084Z generator_hidden_states = self.convbert( 2025-12-04T09:02:02.9981811Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:02.9982520Z hidden_states = self.encoder( 2025-12-04T09:02:02.9983270Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:02.9983965Z layer_outputs = layer_module( 2025-12-04T09:02:02.9984560Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:02.9985223Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:02.9985947Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:02:02.9986675Z self_attention_outputs = self.attention( 2025-12-04T09:02:02.9987422Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 470, in forward 2025-12-04T09:02:02.9988256Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:02:02.9989096Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 424, in forward 2025-12-04T09:02:02.9989838Z hidden_states = self.dense(hidden_states) 2025-12-04T09:02:02.9990076Z 2025-12-04T09:02:02.9990265Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:02.9990901Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:02.9991505Z res = mod(**inputs) 2025-12-04T09:02:02.9992215Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:02.9992996Z generator_hidden_states = self.convbert( 2025-12-04T09:02:02.9993737Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:02.9994513Z hidden_states = self.encoder( 2025-12-04T09:02:02.9995257Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:02.9995982Z layer_outputs = layer_module( 2025-12-04T09:02:02.9996640Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:02.9997318Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:02.9998082Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 585, in forward 2025-12-04T09:02:02.9999029Z layer_output = apply_chunking_to_forward( 2025-12-04T09:02:02.9999850Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:02:03.0000621Z return forward_fn(*input_tensors) 2025-12-04T09:02:03.0001423Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 592, in feed_forward_chunk 2025-12-04T09:02:03.0002304Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:02:03.0003143Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 512, in forward 2025-12-04T09:02:03.0003902Z hidden_states = self.dense(hidden_states) 2025-12-04T09:02:03.0004147Z 2025-12-04T09:02:03.0004326Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:03.0004983Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:03.0005564Z res = mod(**inputs) 2025-12-04T09:02:03.0006251Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:03.0006993Z generator_hidden_states = self.convbert( 2025-12-04T09:02:03.0007760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:03.0008514Z hidden_states = self.encoder( 2025-12-04T09:02:03.0009325Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:03.0010074Z layer_outputs = layer_module( 2025-12-04T09:02:03.0010717Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:03.0011430Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:03.0012163Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 585, in forward 2025-12-04T09:02:03.0012939Z layer_output = apply_chunking_to_forward( 2025-12-04T09:02:03.0013704Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:02:03.0014500Z return forward_fn(*input_tensors) 2025-12-04T09:02:03.0015314Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 592, in feed_forward_chunk 2025-12-04T09:02:03.0016220Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:02:03.0017069Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 513, in forward 2025-12-04T09:02:03.0017891Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:02:03.0018578Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:02:03.0019219Z return self.act(input) 2025-12-04T09:02:03.0019419Z 2025-12-04T09:02:03.0019612Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:03.0020277Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:03.0021214Z res = mod(**inputs) 2025-12-04T09:02:03.0021934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:03.0022768Z generator_hidden_states = self.convbert( 2025-12-04T09:02:03.0023568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:03.0024368Z hidden_states = self.encoder( 2025-12-04T09:02:03.0025165Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:03.0025950Z layer_outputs = layer_module( 2025-12-04T09:02:03.0026628Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:03.0027327Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:03.0028141Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 585, in forward 2025-12-04T09:02:03.0028998Z layer_output = apply_chunking_to_forward( 2025-12-04T09:02:03.0029815Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:02:03.0030586Z return forward_fn(*input_tensors) 2025-12-04T09:02:03.0031432Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-12-04T09:02:03.0032405Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:02:03.0033348Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 530, in forward 2025-12-04T09:02:03.0034191Z hidden_states = self.dense(hidden_states) 2025-12-04T09:02:03.0034463Z 2025-12-04T09:02:03.0034672Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:03.0035382Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:03.0036028Z res = mod(**inputs) 2025-12-04T09:02:03.0036954Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:03.0037807Z generator_hidden_states = self.convbert( 2025-12-04T09:02:03.0038770Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:03.0039685Z hidden_states = self.encoder( 2025-12-04T09:02:03.0040500Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:03.0041286Z layer_outputs = layer_module( 2025-12-04T09:02:03.0041973Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:03.0042685Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:03.0043542Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:02:03.0044320Z self_attention_outputs = self.attention( 2025-12-04T09:02:03.0045128Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:02:03.0045933Z self_outputs = self.self( 2025-12-04T09:02:03.0046696Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 349, in forward 2025-12-04T09:02:03.0047534Z mixed_query_layer = self.query(hidden_states) 2025-12-04T09:02:03.0047829Z 2025-12-04T09:02:03.0048023Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:03.0048698Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:03.0049345Z res = mod(**inputs) 2025-12-04T09:02:03.0050094Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:03.0050910Z generator_hidden_states = self.convbert( 2025-12-04T09:02:03.0051731Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:03.0052535Z hidden_states = self.encoder( 2025-12-04T09:02:03.0053326Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:03.0054142Z layer_outputs = layer_module( 2025-12-04T09:02:03.0054839Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:03.0055548Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:03.0056325Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:02:03.0057145Z self_attention_outputs = self.attention( 2025-12-04T09:02:03.0057980Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:02:03.0058772Z self_outputs = self.self( 2025-12-04T09:02:03.0059554Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 343, in forward 2025-12-04T09:02:03.0060366Z mixed_key_layer = self.key(hidden_states) 2025-12-04T09:02:03.0060634Z 2025-12-04T09:02:03.0060830Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:03.0061544Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:03.0062176Z res = mod(**inputs) 2025-12-04T09:02:03.0062898Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:03.0063721Z generator_hidden_states = self.convbert( 2025-12-04T09:02:03.0064588Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:03.0065370Z hidden_states = self.encoder( 2025-12-04T09:02:03.0066156Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:03.0067034Z layer_outputs = layer_module( 2025-12-04T09:02:03.0067718Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:03.0068428Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:03.0069253Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:02:03.0070060Z self_attention_outputs = self.attention( 2025-12-04T09:02:03.0070863Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:02:03.0071594Z self_outputs = self.self( 2025-12-04T09:02:03.0072334Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 344, in forward 2025-12-04T09:02:03.0073131Z mixed_value_layer = self.value(hidden_states) 2025-12-04T09:02:03.0073404Z 2025-12-04T09:02:03.0073549Z cudagraph partition due to non gpu ops 2025-12-04T09:02:03.0073924Z cudagraph partition due to non gpu ops 2025-12-04T09:02:03.0074338Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:03.0075010Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:03.0075603Z res = mod(**inputs) 2025-12-04T09:02:03.0076297Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:03.0077165Z generator_hidden_states = self.convbert( 2025-12-04T09:02:03.0077964Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:03.0078866Z hidden_states = self.encoder( 2025-12-04T09:02:03.0079694Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:03.0080485Z layer_outputs = layer_module( 2025-12-04T09:02:03.0081118Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:03.0081833Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:03.0082642Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:02:03.0083460Z self_attention_outputs = self.attention( 2025-12-04T09:02:03.0084280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:02:03.0085083Z self_outputs = self.self( 2025-12-04T09:02:03.0085848Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 365, in forward 2025-12-04T09:02:03.0086698Z conv_out_layer = self.conv_out_layer(hidden_states) 2025-12-04T09:02:03.0087014Z 2025-12-04T09:02:03.0087156Z cudagraph partition due to non gpu ops 2025-12-04T09:02:03.0087609Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:03.0088325Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:03.0088970Z res = mod(**inputs) 2025-12-04T09:02:03.0089716Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:03.0090549Z generator_hidden_states = self.convbert( 2025-12-04T09:02:03.0091354Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:03.0092165Z hidden_states = self.encoder( 2025-12-04T09:02:03.0093048Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:03.0093849Z layer_outputs = layer_module( 2025-12-04T09:02:03.0094528Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:03.0095286Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:03.0096112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:02:03.0096951Z self_attention_outputs = self.attention( 2025-12-04T09:02:03.0097763Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:02:03.0098602Z self_outputs = self.self( 2025-12-04T09:02:03.0099382Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 346, in forward 2025-12-04T09:02:03.0100347Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-12-04T09:02:03.0101335Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 281, in forward 2025-12-04T09:02:03.0102143Z x = self.depthwise(hidden_states) 2025-12-04T09:02:03.0102395Z 2025-12-04T09:02:03.0102601Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:03.0103302Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:03.0103927Z res = mod(**inputs) 2025-12-04T09:02:03.0104650Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:03.0105509Z generator_hidden_states = self.convbert( 2025-12-04T09:02:03.0106331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:03.0107126Z hidden_states = self.encoder( 2025-12-04T09:02:03.0107900Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:03.0108682Z layer_outputs = layer_module( 2025-12-04T09:02:03.0109363Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:03.0110123Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:03.0110930Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:02:03.0111749Z self_attention_outputs = self.attention( 2025-12-04T09:02:03.0112563Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:02:03.0113345Z self_outputs = self.self( 2025-12-04T09:02:03.0114131Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 346, in forward 2025-12-04T09:02:03.0115136Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-12-04T09:02:03.0116146Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 282, in forward 2025-12-04T09:02:03.0116943Z x = self.pointwise(x) 2025-12-04T09:02:03.0117150Z 2025-12-04T09:02:03.0117341Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:03.0118078Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:03.0118837Z res = mod(**inputs) 2025-12-04T09:02:03.0119629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:03.0120448Z generator_hidden_states = self.convbert( 2025-12-04T09:02:03.0121581Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:03.0122427Z hidden_states = self.encoder( 2025-12-04T09:02:03.0123219Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:03.0124127Z layer_outputs = layer_module( 2025-12-04T09:02:03.0124837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:03.0125571Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:03.0126391Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:02:03.0127296Z self_attention_outputs = self.attention( 2025-12-04T09:02:03.0128149Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:02:03.0128970Z self_outputs = self.self( 2025-12-04T09:02:03.0129747Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 359, in forward 2025-12-04T09:02:03.0130923Z conv_attn_layer = torch.multiply(mixed_key_conv_attn_layer, mixed_query_layer) 2025-12-04T09:02:03.0131347Z 2025-12-04T09:02:03.0131558Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:03.0132281Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:03.0132934Z res = mod(**inputs) 2025-12-04T09:02:03.0133695Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:03.0134635Z generator_hidden_states = self.convbert( 2025-12-04T09:02:03.0135464Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:03.0136295Z hidden_states = self.encoder( 2025-12-04T09:02:03.0137110Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:03.0137945Z layer_outputs = layer_module( 2025-12-04T09:02:03.0138652Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:03.0139371Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:03.0140187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:02:03.0141011Z self_attention_outputs = self.attention( 2025-12-04T09:02:03.0141818Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:02:03.0142586Z self_outputs = self.self( 2025-12-04T09:02:03.0143358Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 361, in forward 2025-12-04T09:02:03.0144261Z conv_kernel_layer = self.conv_kernel_layer(conv_attn_layer) 2025-12-04T09:02:03.0144607Z 2025-12-04T09:02:03.0144809Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:03.0145509Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:03.0146141Z res = mod(**inputs) 2025-12-04T09:02:03.0146887Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:03.0147708Z generator_hidden_states = self.convbert( 2025-12-04T09:02:03.0148535Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:03.0149357Z hidden_states = self.encoder( 2025-12-04T09:02:03.0150348Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:03.0151145Z layer_outputs = layer_module( 2025-12-04T09:02:03.0151783Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:03.0152518Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:03.0153317Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:02:03.0154123Z self_attention_outputs = self.attention( 2025-12-04T09:02:03.0154909Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:02:03.0155751Z self_outputs = self.self( 2025-12-04T09:02:03.0156535Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 379, in forward 2025-12-04T09:02:03.0157454Z conv_out_layer = torch.matmul(conv_out_layer, conv_kernel_layer) 2025-12-04T09:02:03.0157816Z 2025-12-04T09:02:03.0157966Z cudagraph partition due to non gpu ops 2025-12-04T09:02:03.0158473Z cudagraph partition due to non gpu ops 2025-12-04T09:02:03.0158934Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:03.0159668Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:03.0160297Z res = mod(**inputs) 2025-12-04T09:02:03.0161058Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:03.0161942Z generator_hidden_states = self.convbert( 2025-12-04T09:02:03.0162777Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:03.0163600Z hidden_states = self.encoder( 2025-12-04T09:02:03.0164425Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:03.0165234Z layer_outputs = layer_module( 2025-12-04T09:02:03.0165935Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:03.0166694Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:03.0167503Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:02:03.0168333Z self_attention_outputs = self.attention( 2025-12-04T09:02:03.0169147Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:02:03.0169956Z self_outputs = self.self( 2025-12-04T09:02:03.0170742Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 404, in forward 2025-12-04T09:02:03.0171550Z context_layer = torch.cat([context_layer, conv_out], 2) 2025-12-04T09:02:03.0171861Z 2025-12-04T09:02:03.0172039Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:03.0172720Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:03.0173309Z res = mod(**inputs) 2025-12-04T09:02:03.0173991Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:03.0174755Z generator_hidden_states = self.convbert( 2025-12-04T09:02:03.0175528Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:03.0176285Z hidden_states = self.encoder( 2025-12-04T09:02:03.0177061Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:03.0177937Z layer_outputs = layer_module( 2025-12-04T09:02:03.0178624Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:03.0179341Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:03.0180199Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:02:03.0181032Z self_attention_outputs = self.attention( 2025-12-04T09:02:03.0181873Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 470, in forward 2025-12-04T09:02:03.0182800Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:02:03.0183796Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 424, in forward 2025-12-04T09:02:03.0184847Z hidden_states = self.dense(hidden_states) 2025-12-04T09:02:03.0185130Z 2025-12-04T09:02:03.0185324Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:03.0186033Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:03.0186665Z res = mod(**inputs) 2025-12-04T09:02:03.0187399Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:03.0188210Z generator_hidden_states = self.convbert( 2025-12-04T09:02:03.0189024Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:03.0189802Z hidden_states = self.encoder( 2025-12-04T09:02:03.0190639Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:03.0191428Z layer_outputs = layer_module( 2025-12-04T09:02:03.0192124Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:03.0192846Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:03.0193646Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 585, in forward 2025-12-04T09:02:03.0194475Z layer_output = apply_chunking_to_forward( 2025-12-04T09:02:03.0195267Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:02:03.0196050Z return forward_fn(*input_tensors) 2025-12-04T09:02:03.0196915Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 592, in feed_forward_chunk 2025-12-04T09:02:03.0197903Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:02:03.0198980Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 512, in forward 2025-12-04T09:02:03.0199815Z hidden_states = self.dense(hidden_states) 2025-12-04T09:02:03.0200097Z 2025-12-04T09:02:03.0200294Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:03.0201002Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:03.0201639Z res = mod(**inputs) 2025-12-04T09:02:03.0202383Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:03.0203209Z generator_hidden_states = self.convbert( 2025-12-04T09:02:03.0204027Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:03.0204838Z hidden_states = self.encoder( 2025-12-04T09:02:03.0205637Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:03.0206401Z layer_outputs = layer_module( 2025-12-04T09:02:03.0207058Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:03.0207726Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:03.0208545Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 585, in forward 2025-12-04T09:02:03.0209371Z layer_output = apply_chunking_to_forward( 2025-12-04T09:02:03.0210173Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:02:03.0210960Z return forward_fn(*input_tensors) 2025-12-04T09:02:03.0211849Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 592, in feed_forward_chunk 2025-12-04T09:02:03.0212818Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:02:03.0213734Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 513, in forward 2025-12-04T09:02:03.0214559Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:02:03.0215280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:02:03.0215920Z return self.act(input) 2025-12-04T09:02:03.0216118Z 2025-12-04T09:02:03.0216304Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:03.0216974Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:03.0217575Z res = mod(**inputs) 2025-12-04T09:02:03.0218328Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:03.0219115Z generator_hidden_states = self.convbert( 2025-12-04T09:02:03.0219913Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:03.0220861Z hidden_states = self.encoder( 2025-12-04T09:02:03.0221623Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:03.0222397Z layer_outputs = layer_module( 2025-12-04T09:02:03.0223073Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:03.0224566Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:03.0225087Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 585, in forward 2025-12-04T09:02:03.0225604Z layer_output = apply_chunking_to_forward( 2025-12-04T09:02:03.0226074Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:02:03.0226524Z return forward_fn(*input_tensors) 2025-12-04T09:02:03.0226999Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-12-04T09:02:03.0227549Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:02:03.0228070Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 530, in forward 2025-12-04T09:02:03.0228639Z hidden_states = self.dense(hidden_states) 2025-12-04T09:02:03.0228798Z 2025-12-04T09:02:03.0228921Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:03.0229327Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:03.0229696Z res = mod(**inputs) 2025-12-04T09:02:03.0230327Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:03.0230834Z generator_hidden_states = self.convbert( 2025-12-04T09:02:03.0231336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:03.0231845Z hidden_states = self.encoder( 2025-12-04T09:02:03.0232272Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:03.0232710Z layer_outputs = layer_module( 2025-12-04T09:02:03.0233096Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:03.0233517Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:03.0234025Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:02:03.0234504Z self_attention_outputs = self.attention( 2025-12-04T09:02:03.0234970Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:02:03.0235414Z self_outputs = self.self( 2025-12-04T09:02:03.0235856Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 349, in forward 2025-12-04T09:02:03.0236334Z mixed_query_layer = self.query(hidden_states) 2025-12-04T09:02:03.0236499Z 2025-12-04T09:02:03.0236624Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:03.0237039Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:03.0237413Z res = mod(**inputs) 2025-12-04T09:02:03.0237882Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:03.0238513Z generator_hidden_states = self.convbert( 2025-12-04T09:02:03.0238989Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:03.0239447Z hidden_states = self.encoder( 2025-12-04T09:02:03.0239943Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:03.0240392Z layer_outputs = layer_module( 2025-12-04T09:02:03.0240791Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:03.0241209Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:03.0241679Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:02:03.0242134Z self_attention_outputs = self.attention( 2025-12-04T09:02:03.0242601Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:02:03.0243058Z self_outputs = self.self( 2025-12-04T09:02:03.0243487Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 343, in forward 2025-12-04T09:02:03.0243953Z mixed_key_layer = self.key(hidden_states) 2025-12-04T09:02:03.0244116Z 2025-12-04T09:02:03.0244234Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:03.0244643Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:03.0245000Z res = mod(**inputs) 2025-12-04T09:02:03.0245425Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:03.0245914Z generator_hidden_states = self.convbert( 2025-12-04T09:02:03.0246375Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:03.0246919Z hidden_states = self.encoder( 2025-12-04T09:02:03.0247363Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:03.0247812Z layer_outputs = layer_module( 2025-12-04T09:02:03.0248224Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:03.0248643Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:03.0249095Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:02:03.0249561Z self_attention_outputs = self.attention( 2025-12-04T09:02:03.0250009Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:02:03.0250499Z self_outputs = self.self( 2025-12-04T09:02:03.0250933Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 344, in forward 2025-12-04T09:02:03.0251396Z mixed_value_layer = self.value(hidden_states) 2025-12-04T09:02:03.0251571Z 2025-12-04T09:02:03.0251666Z cudagraph partition due to non gpu ops 2025-12-04T09:02:03.0251916Z cudagraph partition due to non gpu ops 2025-12-04T09:02:03.0252176Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:03.0252562Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:03.0252920Z res = mod(**inputs) 2025-12-04T09:02:03.0253341Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:03.0253802Z generator_hidden_states = self.convbert( 2025-12-04T09:02:03.0254253Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:03.0254695Z hidden_states = self.encoder( 2025-12-04T09:02:03.0255131Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:03.0255561Z layer_outputs = layer_module( 2025-12-04T09:02:03.0255945Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:03.0256346Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:03.0256791Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:02:03.0257229Z self_attention_outputs = self.attention( 2025-12-04T09:02:03.0257672Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:02:03.0258111Z self_outputs = self.self( 2025-12-04T09:02:03.0258525Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 365, in forward 2025-12-04T09:02:03.0258998Z conv_out_layer = self.conv_out_layer(hidden_states) 2025-12-04T09:02:03.0259174Z 2025-12-04T09:02:03.0259264Z cudagraph partition due to non gpu ops 2025-12-04T09:02:03.0259529Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:03.0259919Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:03.0260271Z res = mod(**inputs) 2025-12-04T09:02:03.0260684Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:03.0261126Z generator_hidden_states = self.convbert( 2025-12-04T09:02:03.0261576Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:03.0262014Z hidden_states = self.encoder( 2025-12-04T09:02:03.0262471Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:03.0262906Z layer_outputs = layer_module( 2025-12-04T09:02:03.0263300Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:03.0263719Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:03.0264161Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:02:03.0264620Z self_attention_outputs = self.attention( 2025-12-04T09:02:03.0265062Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:02:03.0265506Z self_outputs = self.self( 2025-12-04T09:02:03.0265948Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 346, in forward 2025-12-04T09:02:03.0266495Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-12-04T09:02:03.0267027Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 281, in forward 2025-12-04T09:02:03.0267478Z x = self.depthwise(hidden_states) 2025-12-04T09:02:03.0267616Z 2025-12-04T09:02:03.0267727Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:03.0268103Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:03.0268457Z res = mod(**inputs) 2025-12-04T09:02:03.0268888Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:03.0269363Z generator_hidden_states = self.convbert( 2025-12-04T09:02:03.0269821Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:03.0270265Z hidden_states = self.encoder( 2025-12-04T09:02:03.0270692Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:03.0271217Z layer_outputs = layer_module( 2025-12-04T09:02:03.0271611Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:03.0272013Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:03.0272470Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:02:03.0272927Z self_attention_outputs = self.attention( 2025-12-04T09:02:03.0273380Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:02:03.0273819Z self_outputs = self.self( 2025-12-04T09:02:03.0274247Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 346, in forward 2025-12-04T09:02:03.0274789Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-12-04T09:02:03.0275323Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 282, in forward 2025-12-04T09:02:03.0275767Z x = self.pointwise(x) 2025-12-04T09:02:03.0275899Z 2025-12-04T09:02:03.0276020Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:03.0276415Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:03.0276776Z res = mod(**inputs) 2025-12-04T09:02:03.0277224Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:03.0277692Z generator_hidden_states = self.convbert( 2025-12-04T09:02:03.0278187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:03.0278713Z hidden_states = self.encoder( 2025-12-04T09:02:03.0279176Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:03.0279668Z layer_outputs = layer_module( 2025-12-04T09:02:03.0280136Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:03.0280514Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:03.0280942Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:02:03.0281421Z self_attention_outputs = self.attention( 2025-12-04T09:02:03.0281920Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:02:03.0282369Z self_outputs = self.self( 2025-12-04T09:02:03.0282853Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 359, in forward 2025-12-04T09:02:03.0283404Z conv_attn_layer = torch.multiply(mixed_key_conv_attn_layer, mixed_query_layer) 2025-12-04T09:02:03.0283646Z 2025-12-04T09:02:03.0283770Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:03.0284171Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:03.0284530Z res = mod(**inputs) 2025-12-04T09:02:03.0284956Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:03.0285457Z generator_hidden_states = self.convbert( 2025-12-04T09:02:03.0285918Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:03.0286372Z hidden_states = self.encoder( 2025-12-04T09:02:03.0286813Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:03.0287253Z layer_outputs = layer_module( 2025-12-04T09:02:03.0287642Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:03.0288049Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:03.0288493Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:02:03.0288959Z self_attention_outputs = self.attention( 2025-12-04T09:02:03.0289407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:02:03.0289839Z self_outputs = self.self( 2025-12-04T09:02:03.0290255Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 361, in forward 2025-12-04T09:02:03.0290765Z conv_kernel_layer = self.conv_kernel_layer(conv_attn_layer) 2025-12-04T09:02:03.0290963Z 2025-12-04T09:02:03.0291078Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:03.0291473Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:03.0291821Z res = mod(**inputs) 2025-12-04T09:02:03.0292232Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:03.0292681Z generator_hidden_states = self.convbert( 2025-12-04T09:02:03.0293119Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:03.0293561Z hidden_states = self.encoder( 2025-12-04T09:02:03.0294015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:03.0294464Z layer_outputs = layer_module( 2025-12-04T09:02:03.0294844Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:03.0295269Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:03.0295720Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:02:03.0296170Z self_attention_outputs = self.attention( 2025-12-04T09:02:03.0296610Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:02:03.0297052Z self_outputs = self.self( 2025-12-04T09:02:03.0297501Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 379, in forward 2025-12-04T09:02:03.0297996Z conv_out_layer = torch.matmul(conv_out_layer, conv_kernel_layer) 2025-12-04T09:02:03.0298201Z 2025-12-04T09:02:03.0298290Z cudagraph partition due to non gpu ops 2025-12-04T09:02:03.0298523Z cudagraph partition due to non gpu ops 2025-12-04T09:02:03.0298782Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:03.0299171Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:03.0299525Z res = mod(**inputs) 2025-12-04T09:02:03.0299936Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:03.0300380Z generator_hidden_states = self.convbert( 2025-12-04T09:02:03.0300948Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:03.0301387Z hidden_states = self.encoder( 2025-12-04T09:02:03.0301821Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:03.0302255Z layer_outputs = layer_module( 2025-12-04T09:02:03.0302642Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:03.0303049Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:03.0303494Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:02:03.0303941Z self_attention_outputs = self.attention( 2025-12-04T09:02:03.0304393Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:02:03.0304834Z self_outputs = self.self( 2025-12-04T09:02:03.0305249Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 404, in forward 2025-12-04T09:02:03.0305719Z context_layer = torch.cat([context_layer, conv_out], 2) 2025-12-04T09:02:03.0305897Z 2025-12-04T09:02:03.0306005Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:03.0306375Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:03.0306701Z res = mod(**inputs) 2025-12-04T09:02:03.0307088Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:03.0307509Z generator_hidden_states = self.convbert( 2025-12-04T09:02:03.0307918Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:03.0308330Z hidden_states = self.encoder( 2025-12-04T09:02:03.0308737Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:03.0309213Z layer_outputs = layer_module( 2025-12-04T09:02:03.0309589Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:03.0309988Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:03.0310490Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:02:03.0310939Z self_attention_outputs = self.attention( 2025-12-04T09:02:03.0311378Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 470, in forward 2025-12-04T09:02:03.0311879Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:02:03.0312400Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 424, in forward 2025-12-04T09:02:03.0312845Z hidden_states = self.dense(hidden_states) 2025-12-04T09:02:03.0313006Z 2025-12-04T09:02:03.0313122Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:03.0313577Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:03.0313932Z res = mod(**inputs) 2025-12-04T09:02:03.0314344Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:03.0314798Z generator_hidden_states = self.convbert( 2025-12-04T09:02:03.0315251Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:03.0315693Z hidden_states = self.encoder( 2025-12-04T09:02:03.0316122Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:03.0316583Z layer_outputs = layer_module( 2025-12-04T09:02:03.0316963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:03.0317357Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:03.0317808Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 585, in forward 2025-12-04T09:02:03.0318351Z layer_output = apply_chunking_to_forward( 2025-12-04T09:02:03.0318810Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:02:03.0319235Z return forward_fn(*input_tensors) 2025-12-04T09:02:03.0319709Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 592, in feed_forward_chunk 2025-12-04T09:02:03.0320251Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:02:03.0320977Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 512, in forward 2025-12-04T09:02:03.0321445Z hidden_states = self.dense(hidden_states) 2025-12-04T09:02:03.0321613Z 2025-12-04T09:02:03.0321728Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:03.0322153Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:03.0322522Z res = mod(**inputs) 2025-12-04T09:02:03.0322954Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:03.0323430Z generator_hidden_states = self.convbert( 2025-12-04T09:02:03.0323903Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:03.0324346Z hidden_states = self.encoder( 2025-12-04T09:02:03.0324791Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:03.0325310Z layer_outputs = layer_module( 2025-12-04T09:02:03.0325692Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:03.0326084Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:03.0326556Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 585, in forward 2025-12-04T09:02:03.0327009Z layer_output = apply_chunking_to_forward( 2025-12-04T09:02:03.0327442Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:02:03.0327878Z return forward_fn(*input_tensors) 2025-12-04T09:02:03.0328357Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 592, in feed_forward_chunk 2025-12-04T09:02:03.0328917Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:02:03.0329409Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 513, in forward 2025-12-04T09:02:03.0329899Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:02:03.0330321Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:02:03.0330704Z return self.act(input) 2025-12-04T09:02:03.0330826Z 2025-12-04T09:02:03.0330940Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:03.0331334Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:03.0331688Z res = mod(**inputs) 2025-12-04T09:02:03.0332099Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:03.0332593Z generator_hidden_states = self.convbert( 2025-12-04T09:02:03.0333047Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:03.0333492Z hidden_states = self.encoder( 2025-12-04T09:02:03.0333920Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:03.0334369Z layer_outputs = layer_module( 2025-12-04T09:02:03.0334758Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:03.0335156Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:03.0335608Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 585, in forward 2025-12-04T09:02:03.0336072Z layer_output = apply_chunking_to_forward( 2025-12-04T09:02:03.0336518Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:02:03.0336925Z return forward_fn(*input_tensors) 2025-12-04T09:02:03.0337376Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-12-04T09:02:03.0337881Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:02:03.0338370Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 530, in forward 2025-12-04T09:02:03.0338801Z hidden_states = self.dense(hidden_states) 2025-12-04T09:02:03.0338958Z 2025-12-04T09:02:03.0339073Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:03.0339472Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:03.0339835Z res = mod(**inputs) 2025-12-04T09:02:03.0340249Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:03.0340711Z generator_hidden_states = self.convbert( 2025-12-04T09:02:03.0340987Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:03.0341073Z hidden_states = self.encoder( 2025-12-04T09:02:03.0341371Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:03.0341445Z layer_outputs = layer_module( 2025-12-04T09:02:03.0341694Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:03.0341780Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:03.0342066Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:02:03.0342168Z self_attention_outputs = self.attention( 2025-12-04T09:02:03.0342449Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:02:03.0342533Z self_outputs = self.self( 2025-12-04T09:02:03.0342808Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 349, in forward 2025-12-04T09:02:03.0342913Z mixed_query_layer = self.query(hidden_states) 2025-12-04T09:02:03.0342917Z 2025-12-04T09:02:03.0343024Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:03.0343233Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:03.0343308Z res = mod(**inputs) 2025-12-04T09:02:03.0343586Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:03.0343690Z generator_hidden_states = self.convbert( 2025-12-04T09:02:03.0343973Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:03.0344049Z hidden_states = self.encoder( 2025-12-04T09:02:03.0344329Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:03.0344404Z layer_outputs = layer_module( 2025-12-04T09:02:03.0344631Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:03.0344719Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:03.0344990Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:02:03.0345083Z self_attention_outputs = self.attention( 2025-12-04T09:02:03.0345359Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:02:03.0345433Z self_outputs = self.self( 2025-12-04T09:02:03.0345715Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 343, in forward 2025-12-04T09:02:03.0345800Z mixed_key_layer = self.key(hidden_states) 2025-12-04T09:02:03.0345803Z 2025-12-04T09:02:03.0345911Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:03.0346121Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:03.0346187Z res = mod(**inputs) 2025-12-04T09:02:03.0346468Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:03.0346548Z generator_hidden_states = self.convbert( 2025-12-04T09:02:03.0346823Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:03.0346903Z hidden_states = self.encoder( 2025-12-04T09:02:03.0347196Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:03.0347276Z layer_outputs = layer_module( 2025-12-04T09:02:03.0347504Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:03.0347607Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:03.0347889Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:02:03.0347973Z self_attention_outputs = self.attention( 2025-12-04T09:02:03.0348252Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:02:03.0348334Z self_outputs = self.self( 2025-12-04T09:02:03.0348624Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 344, in forward 2025-12-04T09:02:03.0348730Z mixed_value_layer = self.value(hidden_states) 2025-12-04T09:02:03.0348734Z 2025-12-04T09:02:03.0348818Z cudagraph partition due to non gpu ops 2025-12-04T09:02:03.0348902Z cudagraph partition due to non gpu ops 2025-12-04T09:02:03.0349019Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:03.0349238Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:03.0349308Z res = mod(**inputs) 2025-12-04T09:02:03.0349611Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:03.0349708Z generator_hidden_states = self.convbert( 2025-12-04T09:02:03.0350015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:03.0350091Z hidden_states = self.encoder( 2025-12-04T09:02:03.0350370Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:03.0350450Z layer_outputs = layer_module( 2025-12-04T09:02:03.0350678Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:03.0350769Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:03.0351061Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:02:03.0351147Z self_attention_outputs = self.attention( 2025-12-04T09:02:03.0351448Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:02:03.0351527Z self_outputs = self.self( 2025-12-04T09:02:03.0351820Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 365, in forward 2025-12-04T09:02:03.0351942Z conv_out_layer = self.conv_out_layer(hidden_states) 2025-12-04T09:02:03.0351946Z 2025-12-04T09:02:03.0352031Z cudagraph partition due to non gpu ops 2025-12-04T09:02:03.0352156Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:03.0352378Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:03.0352450Z res = mod(**inputs) 2025-12-04T09:02:03.0352749Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:03.0352837Z generator_hidden_states = self.convbert( 2025-12-04T09:02:03.0353136Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:03.0353218Z hidden_states = self.encoder( 2025-12-04T09:02:03.0353546Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:03.0353634Z layer_outputs = layer_module( 2025-12-04T09:02:03.0353998Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:03.0354119Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:03.0354422Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:02:03.0354525Z self_attention_outputs = self.attention( 2025-12-04T09:02:03.0354824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:02:03.0354907Z self_outputs = self.self( 2025-12-04T09:02:03.0355219Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 346, in forward 2025-12-04T09:02:03.0355408Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-12-04T09:02:03.0355704Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 281, in forward 2025-12-04T09:02:03.0355798Z x = self.depthwise(hidden_states) 2025-12-04T09:02:03.0355805Z 2025-12-04T09:02:03.0355917Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:03.0356134Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:03.0356211Z res = mod(**inputs) 2025-12-04T09:02:03.0356504Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:03.0356640Z generator_hidden_states = self.convbert( 2025-12-04T09:02:03.0356940Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:03.0357016Z hidden_states = self.encoder( 2025-12-04T09:02:03.0357315Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:03.0357390Z layer_outputs = layer_module( 2025-12-04T09:02:03.0357633Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:03.0357725Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:03.0358017Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:02:03.0358105Z self_attention_outputs = self.attention( 2025-12-04T09:02:03.0358472Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:02:03.0358564Z self_outputs = self.self( 2025-12-04T09:02:03.0358879Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 346, in forward 2025-12-04T09:02:03.0359056Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-12-04T09:02:03.0359356Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 282, in forward 2025-12-04T09:02:03.0359447Z x = self.pointwise(x) 2025-12-04T09:02:03.0359452Z 2025-12-04T09:02:03.0359569Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:03.0359809Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:03.0359882Z res = mod(**inputs) 2025-12-04T09:02:03.0360191Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:03.0360291Z generator_hidden_states = self.convbert( 2025-12-04T09:02:03.0360615Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:03.0360701Z hidden_states = self.encoder( 2025-12-04T09:02:03.0360998Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:03.0361100Z layer_outputs = layer_module( 2025-12-04T09:02:03.0361357Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:03.0361441Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:03.0361739Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:02:03.0361837Z self_attention_outputs = self.attention( 2025-12-04T09:02:03.0362155Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:02:03.0362239Z self_outputs = self.self( 2025-12-04T09:02:03.0362530Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 359, in forward 2025-12-04T09:02:03.0362699Z conv_attn_layer = torch.multiply(mixed_key_conv_attn_layer, mixed_query_layer) 2025-12-04T09:02:03.0362705Z 2025-12-04T09:02:03.0362823Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:03.0363038Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:03.0363115Z res = mod(**inputs) 2025-12-04T09:02:03.0363404Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:03.0363513Z generator_hidden_states = self.convbert( 2025-12-04T09:02:03.0363812Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:03.0363887Z hidden_states = self.encoder( 2025-12-04T09:02:03.0364179Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:03.0364264Z layer_outputs = layer_module( 2025-12-04T09:02:03.0364502Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:03.0364594Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:03.0364887Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:02:03.0364974Z self_attention_outputs = self.attention( 2025-12-04T09:02:03.0365272Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:02:03.0365350Z self_outputs = self.self( 2025-12-04T09:02:03.0365651Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 361, in forward 2025-12-04T09:02:03.0365782Z conv_kernel_layer = self.conv_kernel_layer(conv_attn_layer) 2025-12-04T09:02:03.0365786Z 2025-12-04T09:02:03.0365897Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:03.0366122Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:03.0366195Z res = mod(**inputs) 2025-12-04T09:02:03.0366490Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:03.0366584Z generator_hidden_states = self.convbert( 2025-12-04T09:02:03.0366878Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:03.0366967Z hidden_states = self.encoder( 2025-12-04T09:02:03.0367281Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:03.0367358Z layer_outputs = layer_module( 2025-12-04T09:02:03.0367605Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:03.0367690Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:03.0368009Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:02:03.0368103Z self_attention_outputs = self.attention( 2025-12-04T09:02:03.0368399Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:02:03.0368485Z self_outputs = self.self( 2025-12-04T09:02:03.0368795Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 379, in forward 2025-12-04T09:02:03.0368939Z conv_out_layer = torch.matmul(conv_out_layer, conv_kernel_layer) 2025-12-04T09:02:03.0368945Z 2025-12-04T09:02:03.0369044Z cudagraph partition due to non gpu ops 2025-12-04T09:02:03.0369132Z cudagraph partition due to non gpu ops 2025-12-04T09:02:03.0369252Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:03.0369468Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:03.0369539Z res = mod(**inputs) 2025-12-04T09:02:03.0369839Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:03.0369926Z generator_hidden_states = self.convbert( 2025-12-04T09:02:03.0370221Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:03.0370342Z hidden_states = self.encoder( 2025-12-04T09:02:03.0370646Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:03.0370731Z layer_outputs = layer_module( 2025-12-04T09:02:03.0370982Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:03.0371067Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:03.0371374Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:02:03.0371459Z self_attention_outputs = self.attention( 2025-12-04T09:02:03.0371765Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:02:03.0371843Z self_outputs = self.self( 2025-12-04T09:02:03.0372148Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 404, in forward 2025-12-04T09:02:03.0372276Z context_layer = torch.cat([context_layer, conv_out], 2) 2025-12-04T09:02:03.0372281Z 2025-12-04T09:02:03.0372391Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:03.0372608Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:03.0372687Z res = mod(**inputs) 2025-12-04T09:02:03.0372988Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:03.0373082Z generator_hidden_states = self.convbert( 2025-12-04T09:02:03.0373393Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:03.0373472Z hidden_states = self.encoder( 2025-12-04T09:02:03.0373801Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:03.0373880Z layer_outputs = layer_module( 2025-12-04T09:02:03.0374165Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:03.0374250Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:03.0374542Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:02:03.0374657Z self_attention_outputs = self.attention( 2025-12-04T09:02:03.0374950Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 470, in forward 2025-12-04T09:02:03.0375090Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:02:03.0375392Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 424, in forward 2025-12-04T09:02:03.0375506Z hidden_states = self.dense(hidden_states) 2025-12-04T09:02:03.0375510Z 2025-12-04T09:02:03.0375631Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:03.0375848Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:03.0375919Z res = mod(**inputs) 2025-12-04T09:02:03.0376214Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:03.0376300Z generator_hidden_states = self.convbert( 2025-12-04T09:02:03.0376599Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:03.0376678Z hidden_states = self.encoder( 2025-12-04T09:02:03.0376966Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:03.0377088Z layer_outputs = layer_module( 2025-12-04T09:02:03.0377332Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:03.0377419Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:03.0377718Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 585, in forward 2025-12-04T09:02:03.0377810Z layer_output = apply_chunking_to_forward( 2025-12-04T09:02:03.0378102Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:02:03.0378185Z return forward_fn(*input_tensors) 2025-12-04T09:02:03.0378510Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 592, in feed_forward_chunk 2025-12-04T09:02:03.0378647Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:02:03.0378939Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 512, in forward 2025-12-04T09:02:03.0379036Z hidden_states = self.dense(hidden_states) 2025-12-04T09:02:03.0379042Z 2025-12-04T09:02:03.0379153Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:03.0379367Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:03.0379445Z res = mod(**inputs) 2025-12-04T09:02:03.0379739Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:03.0379825Z generator_hidden_states = self.convbert( 2025-12-04T09:02:03.0380135Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:03.0380212Z hidden_states = self.encoder( 2025-12-04T09:02:03.0380511Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:03.0380587Z layer_outputs = layer_module( 2025-12-04T09:02:03.0380894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:03.0380987Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:03.0381281Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 585, in forward 2025-12-04T09:02:03.0381399Z layer_output = apply_chunking_to_forward( 2025-12-04T09:02:03.0381683Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:02:03.0381766Z return forward_fn(*input_tensors) 2025-12-04T09:02:03.0382103Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 592, in feed_forward_chunk 2025-12-04T09:02:03.0382258Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:02:03.0382553Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 513, in forward 2025-12-04T09:02:03.0382682Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:02:03.0382917Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:02:03.0383004Z return self.act(input) 2025-12-04T09:02:03.0383008Z 2025-12-04T09:02:03.0383122Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:03.0383341Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:03.0383422Z res = mod(**inputs) 2025-12-04T09:02:03.0383713Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:03.0383832Z generator_hidden_states = self.convbert( 2025-12-04T09:02:03.0384127Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:03.0384207Z hidden_states = self.encoder( 2025-12-04T09:02:03.0384513Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:03.0384592Z layer_outputs = layer_module( 2025-12-04T09:02:03.0384846Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:03.0384938Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:03.0385241Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 585, in forward 2025-12-04T09:02:03.0385334Z layer_output = apply_chunking_to_forward( 2025-12-04T09:02:03.0385631Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:02:03.0385716Z return forward_fn(*input_tensors) 2025-12-04T09:02:03.0386066Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-12-04T09:02:03.0386215Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:02:03.0386530Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 530, in forward 2025-12-04T09:02:03.0386622Z hidden_states = self.dense(hidden_states) 2025-12-04T09:02:03.0386626Z 2025-12-04T09:02:03.0386735Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:03.0386969Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:03.0387044Z res = mod(**inputs) 2025-12-04T09:02:03.0387336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:03.0387432Z generator_hidden_states = self.convbert( 2025-12-04T09:02:03.0387753Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:03.0387841Z hidden_states = self.encoder( 2025-12-04T09:02:03.0388135Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:03.0388240Z layer_outputs = layer_module( 2025-12-04T09:02:03.0388494Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:03.0388577Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:03.0388878Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:02:03.0388971Z self_attention_outputs = self.attention( 2025-12-04T09:02:03.0389288Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:02:03.0389377Z self_outputs = self.self( 2025-12-04T09:02:03.0389670Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 349, in forward 2025-12-04T09:02:03.0389770Z mixed_query_layer = self.query(hidden_states) 2025-12-04T09:02:03.0389782Z 2025-12-04T09:02:03.0389895Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:03.0390109Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:03.0390185Z res = mod(**inputs) 2025-12-04T09:02:03.0390477Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:03.0390589Z generator_hidden_states = self.convbert( 2025-12-04T09:02:03.0390904Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:03.0390992Z hidden_states = self.encoder( 2025-12-04T09:02:03.0391304Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:03.0391387Z layer_outputs = layer_module( 2025-12-04T09:02:03.0391644Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:03.0391743Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:03.0392056Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:02:03.0392145Z self_attention_outputs = self.attention( 2025-12-04T09:02:03.0392455Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:02:03.0392535Z self_outputs = self.self( 2025-12-04T09:02:03.0392843Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 343, in forward 2025-12-04T09:02:03.0392936Z mixed_key_layer = self.key(hidden_states) 2025-12-04T09:02:03.0392939Z 2025-12-04T09:02:03.0393052Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:03.0393281Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:03.0393355Z res = mod(**inputs) 2025-12-04T09:02:03.0393661Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:03.0393749Z generator_hidden_states = self.convbert( 2025-12-04T09:02:03.0394053Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:03.0394146Z hidden_states = self.encoder( 2025-12-04T09:02:03.0394475Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:03.0394554Z layer_outputs = layer_module( 2025-12-04T09:02:03.0394804Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:03.0394907Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:03.0395207Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:02:03.0395294Z self_attention_outputs = self.attention( 2025-12-04T09:02:03.0395582Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:02:03.0395667Z self_outputs = self.self( 2025-12-04T09:02:03.0395985Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 344, in forward 2025-12-04T09:02:03.0396096Z mixed_value_layer = self.value(hidden_states) 2025-12-04T09:02:03.0396100Z 2025-12-04T09:02:03.0396190Z cudagraph partition due to non gpu ops 2025-12-04T09:02:03.0396280Z cudagraph partition due to non gpu ops 2025-12-04T09:02:03.0396400Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:03.0396625Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:03.0396698Z res = mod(**inputs) 2025-12-04T09:02:03.0397009Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:03.0397099Z generator_hidden_states = self.convbert( 2025-12-04T09:02:03.0397407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:03.0397511Z hidden_states = self.encoder( 2025-12-04T09:02:03.0397818Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:03.0397905Z layer_outputs = layer_module( 2025-12-04T09:02:03.0398154Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:03.0398333Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:03.0398655Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:02:03.0398747Z self_attention_outputs = self.attention( 2025-12-04T09:02:03.0399056Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:02:03.0399140Z self_outputs = self.self( 2025-12-04T09:02:03.0399440Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 365, in forward 2025-12-04T09:02:03.0399564Z conv_out_layer = self.conv_out_layer(hidden_states) 2025-12-04T09:02:03.0399570Z 2025-12-04T09:02:03.0399659Z cudagraph partition due to non gpu ops 2025-12-04T09:02:03.0399784Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:03.0400014Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:03.0400088Z res = mod(**inputs) 2025-12-04T09:02:03.0400387Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:03.0400474Z generator_hidden_states = self.convbert( 2025-12-04T09:02:03.0400768Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:03.0400857Z hidden_states = self.encoder( 2025-12-04T09:02:03.0401150Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:03.0401265Z layer_outputs = layer_module( 2025-12-04T09:02:03.0401513Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:03.0401602Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:03.0401925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:02:03.0402013Z self_attention_outputs = self.attention( 2025-12-04T09:02:03.0402313Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:02:03.0402391Z self_outputs = self.self( 2025-12-04T09:02:03.0402686Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 346, in forward 2025-12-04T09:02:03.0402891Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-12-04T09:02:03.0403192Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 281, in forward 2025-12-04T09:02:03.0403278Z x = self.depthwise(hidden_states) 2025-12-04T09:02:03.0403282Z 2025-12-04T09:02:03.0403404Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:03.0403621Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:03.0403697Z res = mod(**inputs) 2025-12-04T09:02:03.0403997Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:03.0404084Z generator_hidden_states = self.convbert( 2025-12-04T09:02:03.0404413Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:03.0404490Z hidden_states = self.encoder( 2025-12-04T09:02:03.0404791Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:03.0404868Z layer_outputs = layer_module( 2025-12-04T09:02:03.0405109Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:03.0405202Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:03.0405492Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:02:03.0405579Z self_attention_outputs = self.attention( 2025-12-04T09:02:03.0405875Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:02:03.0405954Z self_outputs = self.self( 2025-12-04T09:02:03.0406253Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 346, in forward 2025-12-04T09:02:03.0406422Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-12-04T09:02:03.0406713Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 282, in forward 2025-12-04T09:02:03.0406800Z x = self.pointwise(x) 2025-12-04T09:02:03.0406804Z 2025-12-04T09:02:03.0406916Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:03.0407133Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:03.0407204Z res = mod(**inputs) 2025-12-04T09:02:03.0407497Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:03.0407593Z generator_hidden_states = self.convbert( 2025-12-04T09:02:03.0407885Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:03.0407985Z hidden_states = self.encoder( 2025-12-04T09:02:03.0408288Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:03.0408364Z layer_outputs = layer_module( 2025-12-04T09:02:03.0408635Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:03.0408718Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:03.0409014Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:02:03.0409108Z self_attention_outputs = self.attention( 2025-12-04T09:02:03.0409423Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:02:03.0409509Z self_outputs = self.self( 2025-12-04T09:02:03.0409802Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 359, in forward 2025-12-04T09:02:03.0409969Z conv_attn_layer = torch.multiply(mixed_key_conv_attn_layer, mixed_query_layer) 2025-12-04T09:02:03.0409973Z 2025-12-04T09:02:03.0410093Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:03.0410310Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:03.0410380Z res = mod(**inputs) 2025-12-04T09:02:03.0410691Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:03.0410781Z generator_hidden_states = self.convbert( 2025-12-04T09:02:03.0411126Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:03.0411204Z hidden_states = self.encoder( 2025-12-04T09:02:03.0411500Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:03.0411583Z layer_outputs = layer_module( 2025-12-04T09:02:03.0411826Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:03.0411918Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:03.0412214Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:02:03.0412300Z self_attention_outputs = self.attention( 2025-12-04T09:02:03.0412600Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:02:03.0412679Z self_outputs = self.self( 2025-12-04T09:02:03.0412971Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 361, in forward 2025-12-04T09:02:03.0413109Z conv_kernel_layer = self.conv_kernel_layer(conv_attn_layer) 2025-12-04T09:02:03.0413113Z 2025-12-04T09:02:03.0413222Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:03.0413444Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:03.0413516Z res = mod(**inputs) 2025-12-04T09:02:03.0413802Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:03.0413896Z generator_hidden_states = self.convbert( 2025-12-04T09:02:03.0414185Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:03.0414272Z hidden_states = self.encoder( 2025-12-04T09:02:03.0414565Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:03.0414669Z layer_outputs = layer_module( 2025-12-04T09:02:03.0414916Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:03.0414999Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:03.0415307Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:02:03.0415402Z self_attention_outputs = self.attention( 2025-12-04T09:02:03.0415691Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:02:03.0415772Z self_outputs = self.self( 2025-12-04T09:02:03.0416065Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 379, in forward 2025-12-04T09:02:03.0416226Z conv_out_layer = torch.matmul(conv_out_layer, conv_kernel_layer) 2025-12-04T09:02:03.0416230Z 2025-12-04T09:02:03.0416325Z cudagraph partition due to non gpu ops 2025-12-04T09:02:03.0416410Z cudagraph partition due to non gpu ops 2025-12-04T09:02:03.0416521Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:03.0416744Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:03.0416816Z res = mod(**inputs) 2025-12-04T09:02:03.0417114Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:03.0417202Z generator_hidden_states = self.convbert( 2025-12-04T09:02:03.0417501Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:03.0417614Z hidden_states = self.encoder( 2025-12-04T09:02:03.0417908Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:03.0417994Z layer_outputs = layer_module( 2025-12-04T09:02:03.0418235Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:03.0418318Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:03.0418618Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:02:03.0418705Z self_attention_outputs = self.attention( 2025-12-04T09:02:03.0418996Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:02:03.0419081Z self_outputs = self.self( 2025-12-04T09:02:03.0419374Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 404, in forward 2025-12-04T09:02:03.0419503Z context_layer = torch.cat([context_layer, conv_out], 2) 2025-12-04T09:02:03.0419507Z 2025-12-04T09:02:03.0419618Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:03.0419832Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:03.0419912Z res = mod(**inputs) 2025-12-04T09:02:03.0420201Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:03.0420297Z generator_hidden_states = self.convbert( 2025-12-04T09:02:03.0420587Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:03.0420665Z hidden_states = self.encoder( 2025-12-04T09:02:03.0421222Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:03.0421310Z layer_outputs = layer_module( 2025-12-04T09:02:03.0421620Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:03.0421717Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:03.0422008Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:02:03.0422137Z self_attention_outputs = self.attention( 2025-12-04T09:02:03.0422430Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 470, in forward 2025-12-04T09:02:03.0422569Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:02:03.0422870Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 424, in forward 2025-12-04T09:02:03.0422963Z hidden_states = self.dense(hidden_states) 2025-12-04T09:02:03.0422967Z 2025-12-04T09:02:03.0423126Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:03.0423346Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:03.0423419Z res = mod(**inputs) 2025-12-04T09:02:03.0423721Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:03.0423809Z generator_hidden_states = self.convbert( 2025-12-04T09:02:03.0424105Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:03.0424191Z hidden_states = self.encoder( 2025-12-04T09:02:03.0424485Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:03.0424611Z layer_outputs = layer_module( 2025-12-04T09:02:03.0424855Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:03.0424938Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:03.0425252Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 585, in forward 2025-12-04T09:02:03.0425342Z layer_output = apply_chunking_to_forward( 2025-12-04T09:02:03.0425635Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:02:03.0425719Z return forward_fn(*input_tensors) 2025-12-04T09:02:03.0426050Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 592, in feed_forward_chunk 2025-12-04T09:02:03.0426187Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:02:03.0426487Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 512, in forward 2025-12-04T09:02:03.0426575Z hidden_states = self.dense(hidden_states) 2025-12-04T09:02:03.0426587Z 2025-12-04T09:02:03.0426699Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:03.0426912Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:03.0426991Z res = mod(**inputs) 2025-12-04T09:02:03.0427284Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:03.0427370Z generator_hidden_states = self.convbert( 2025-12-04T09:02:03.0427675Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:03.0427753Z hidden_states = self.encoder( 2025-12-04T09:02:03.0428066Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:03.0428146Z layer_outputs = layer_module( 2025-12-04T09:02:03.0428416Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:03.0428511Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:03.0428804Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 585, in forward 2025-12-04T09:02:03.0428926Z layer_output = apply_chunking_to_forward( 2025-12-04T09:02:03.0429220Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:02:03.0429302Z return forward_fn(*input_tensors) 2025-12-04T09:02:03.0429649Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 592, in feed_forward_chunk 2025-12-04T09:02:03.0429778Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:02:03.0430087Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 513, in forward 2025-12-04T09:02:03.0430221Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:02:03.0430456Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:02:03.0430540Z return self.act(input) 2025-12-04T09:02:03.0430545Z 2025-12-04T09:02:03.0430657Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:03.0430870Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:03.0430946Z res = mod(**inputs) 2025-12-04T09:02:03.0431237Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:03.0431347Z generator_hidden_states = self.convbert( 2025-12-04T09:02:03.0431647Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:03.0431722Z hidden_states = self.encoder( 2025-12-04T09:02:03.0432021Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:03.0432097Z layer_outputs = layer_module( 2025-12-04T09:02:03.0432339Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:03.0432430Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:03.0432724Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 585, in forward 2025-12-04T09:02:03.0432821Z layer_output = apply_chunking_to_forward( 2025-12-04T09:02:03.0433105Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:02:03.0433190Z return forward_fn(*input_tensors) 2025-12-04T09:02:03.0433526Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-12-04T09:02:03.0433670Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:02:03.0433983Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 530, in forward 2025-12-04T09:02:03.0434081Z hidden_states = self.dense(hidden_states) 2025-12-04T09:02:03.0434084Z 2025-12-04T09:02:03.0434194Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:03.0434416Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:03.0434487Z res = mod(**inputs) 2025-12-04T09:02:03.0434776Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:03.0434875Z generator_hidden_states = self.convbert( 2025-12-04T09:02:03.0435189Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:03.0435277Z hidden_states = self.encoder( 2025-12-04T09:02:03.0435570Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:03.0435665Z layer_outputs = layer_module( 2025-12-04T09:02:03.0435914Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:03.0435998Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:03.0436290Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:02:03.0436388Z self_attention_outputs = self.attention( 2025-12-04T09:02:03.0436750Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:02:03.0436839Z self_outputs = self.self( 2025-12-04T09:02:03.0437131Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 349, in forward 2025-12-04T09:02:03.0437230Z mixed_query_layer = self.query(hidden_states) 2025-12-04T09:02:03.0437235Z 2025-12-04T09:02:03.0437355Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:03.0437569Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:03.0437645Z res = mod(**inputs) 2025-12-04T09:02:03.0437934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:03.0438042Z generator_hidden_states = self.convbert( 2025-12-04T09:02:03.0438410Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:03.0438495Z hidden_states = self.encoder( 2025-12-04T09:02:03.0438796Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:03.0438883Z layer_outputs = layer_module( 2025-12-04T09:02:03.0439132Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:03.0439226Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:03.0439528Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:02:03.0439615Z self_attention_outputs = self.attention( 2025-12-04T09:02:03.0439912Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:02:03.0439993Z self_outputs = self.self( 2025-12-04T09:02:03.0440294Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 343, in forward 2025-12-04T09:02:03.0440384Z mixed_key_layer = self.key(hidden_states) 2025-12-04T09:02:03.0440388Z 2025-12-04T09:02:03.0440499Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:03.0440723Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:03.0440796Z res = mod(**inputs) 2025-12-04T09:02:03.0441088Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:03.0441185Z generator_hidden_states = self.convbert( 2025-12-04T09:02:03.0441491Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:03.0441582Z hidden_states = self.encoder( 2025-12-04T09:02:03.0441895Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:03.0442005Z layer_outputs = layer_module( 2025-12-04T09:02:03.0442263Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:03.0442351Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:03.0442682Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:02:03.0442772Z self_attention_outputs = self.attention( 2025-12-04T09:02:03.0443082Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:02:03.0443167Z self_outputs = self.self( 2025-12-04T09:02:03.0443498Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 344, in forward 2025-12-04T09:02:03.0443600Z mixed_value_layer = self.value(hidden_states) 2025-12-04T09:02:03.0443611Z 2025-12-04T09:02:03.0443702Z cudagraph partition due to non gpu ops 2025-12-04T09:02:03.0443789Z cudagraph partition due to non gpu ops 2025-12-04T09:02:03.0443909Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:03.0444128Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:03.0444202Z res = mod(**inputs) 2025-12-04T09:02:03.0444513Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:03.0444603Z generator_hidden_states = self.convbert( 2025-12-04T09:02:03.0444904Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:03.0445011Z hidden_states = self.encoder( 2025-12-04T09:02:03.0445314Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:03.0445402Z layer_outputs = layer_module( 2025-12-04T09:02:03.0445654Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:03.0445740Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:03.0446051Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:02:03.0446141Z self_attention_outputs = self.attention( 2025-12-04T09:02:03.0446458Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:02:03.0446537Z self_outputs = self.self( 2025-12-04T09:02:03.0446850Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 365, in forward 2025-12-04T09:02:03.0446969Z conv_out_layer = self.conv_out_layer(hidden_states) 2025-12-04T09:02:03.0446973Z 2025-12-04T09:02:03.0447062Z cudagraph partition due to non gpu ops 2025-12-04T09:02:03.0447176Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:03.0447405Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:03.0447480Z res = mod(**inputs) 2025-12-04T09:02:03.0447800Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:03.0447890Z generator_hidden_states = self.convbert( 2025-12-04T09:02:03.0448194Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:03.0448283Z hidden_states = self.encoder( 2025-12-04T09:02:03.0448583Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:03.0448668Z layer_outputs = layer_module( 2025-12-04T09:02:03.0448942Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:03.0449031Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:03.0449335Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:02:03.0449444Z self_attention_outputs = self.attention( 2025-12-04T09:02:03.0449744Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:02:03.0449827Z self_outputs = self.self( 2025-12-04T09:02:03.0450130Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 346, in forward 2025-12-04T09:02:03.0450333Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-12-04T09:02:03.0450641Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 281, in forward 2025-12-04T09:02:03.0450737Z x = self.depthwise(hidden_states) 2025-12-04T09:02:03.0450741Z 2025-12-04T09:02:03.0450860Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:03.0451078Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:03.0451153Z res = mod(**inputs) 2025-12-04T09:02:03.0451453Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:03.0451536Z generator_hidden_states = self.convbert( 2025-12-04T09:02:03.0451821Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:03.0451919Z hidden_states = self.encoder( 2025-12-04T09:02:03.0452200Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:03.0452280Z layer_outputs = layer_module( 2025-12-04T09:02:03.0452510Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:03.0452597Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:03.0452874Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:02:03.0452955Z self_attention_outputs = self.attention( 2025-12-04T09:02:03.0453237Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:02:03.0453311Z self_outputs = self.self( 2025-12-04T09:02:03.0453597Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 346, in forward 2025-12-04T09:02:03.0453759Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-12-04T09:02:03.0454041Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 282, in forward 2025-12-04T09:02:03.0454124Z x = self.pointwise(x) 2025-12-04T09:02:03.0454129Z 2025-12-04T09:02:03.0454233Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:03.0454437Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:03.0454513Z res = mod(**inputs) 2025-12-04T09:02:03.0454789Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:03.0454881Z generator_hidden_states = self.convbert( 2025-12-04T09:02:03.0455159Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:03.0455232Z hidden_states = self.encoder( 2025-12-04T09:02:03.0455551Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:03.0455624Z layer_outputs = layer_module( 2025-12-04T09:02:03.0455861Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:03.0455966Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:03.0456238Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:02:03.0456328Z self_attention_outputs = self.attention( 2025-12-04T09:02:03.0456617Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:02:03.0456714Z self_outputs = self.self( 2025-12-04T09:02:03.0457015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 359, in forward 2025-12-04T09:02:03.0457185Z conv_attn_layer = torch.multiply(mixed_key_conv_attn_layer, mixed_query_layer) 2025-12-04T09:02:03.0457189Z 2025-12-04T09:02:03.0457300Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:03.0457510Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:03.0457576Z res = mod(**inputs) 2025-12-04T09:02:03.0457855Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:03.0457937Z generator_hidden_states = self.convbert( 2025-12-04T09:02:03.0458215Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:03.0458404Z hidden_states = self.encoder( 2025-12-04T09:02:03.0458719Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:03.0458805Z layer_outputs = layer_module( 2025-12-04T09:02:03.0459032Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:03.0459112Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:03.0459394Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:02:03.0459480Z self_attention_outputs = self.attention( 2025-12-04T09:02:03.0459763Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:02:03.0459840Z self_outputs = self.self( 2025-12-04T09:02:03.0460120Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 361, in forward 2025-12-04T09:02:03.0460255Z conv_kernel_layer = self.conv_kernel_layer(conv_attn_layer) 2025-12-04T09:02:03.0460260Z 2025-12-04T09:02:03.0460367Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:03.0460577Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:03.0460649Z res = mod(**inputs) 2025-12-04T09:02:03.0460927Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:03.0461021Z generator_hidden_states = self.convbert( 2025-12-04T09:02:03.0461298Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:03.0461374Z hidden_states = self.encoder( 2025-12-04T09:02:03.0461664Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:03.0461739Z layer_outputs = layer_module( 2025-12-04T09:02:03.0462003Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:03.0462083Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:03.0462356Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:02:03.0462466Z self_attention_outputs = self.attention( 2025-12-04T09:02:03.0462739Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:02:03.0462818Z self_outputs = self.self( 2025-12-04T09:02:03.0463093Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 379, in forward 2025-12-04T09:02:03.0463245Z conv_out_layer = torch.matmul(conv_out_layer, conv_kernel_layer) 2025-12-04T09:02:03.0463249Z 2025-12-04T09:02:03.0463340Z cudagraph partition due to non gpu ops 2025-12-04T09:02:03.0463420Z cudagraph partition due to non gpu ops 2025-12-04T09:02:03.0463526Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:03.0463739Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:03.0463807Z res = mod(**inputs) 2025-12-04T09:02:03.0464091Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:03.0464172Z generator_hidden_states = self.convbert( 2025-12-04T09:02:03.0464447Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:03.0464547Z hidden_states = self.encoder( 2025-12-04T09:02:03.0464825Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:03.0464896Z layer_outputs = layer_module( 2025-12-04T09:02:03.0465133Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:03.0465211Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:03.0465508Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:02:03.0465590Z self_attention_outputs = self.attention( 2025-12-04T09:02:03.0465861Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:02:03.0465935Z self_outputs = self.self( 2025-12-04T09:02:03.0466208Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 404, in forward 2025-12-04T09:02:03.0466329Z context_layer = torch.cat([context_layer, conv_out], 2) 2025-12-04T09:02:03.0466335Z 2025-12-04T09:02:03.0466435Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:03.0466638Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:03.0466711Z res = mod(**inputs) 2025-12-04T09:02:03.0466983Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:03.0467064Z generator_hidden_states = self.convbert( 2025-12-04T09:02:03.0467343Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:03.0467414Z hidden_states = self.encoder( 2025-12-04T09:02:03.0467698Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:03.0467771Z layer_outputs = layer_module( 2025-12-04T09:02:03.0468002Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:03.0468107Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:03.0468388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:02:03.0468480Z self_attention_outputs = self.attention( 2025-12-04T09:02:03.0468769Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 470, in forward 2025-12-04T09:02:03.0468899Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:02:03.0469181Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 424, in forward 2025-12-04T09:02:03.0469268Z hidden_states = self.dense(hidden_states) 2025-12-04T09:02:03.0469272Z 2025-12-04T09:02:03.0469407Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:03.0469613Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:03.0469682Z res = mod(**inputs) 2025-12-04T09:02:03.0469969Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:03.0470051Z generator_hidden_states = self.convbert( 2025-12-04T09:02:03.0470331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:03.0470413Z hidden_states = self.encoder( 2025-12-04T09:02:03.0470692Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:03.0470772Z layer_outputs = layer_module( 2025-12-04T09:02:03.0471035Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:03.0471117Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:03.0471424Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 585, in forward 2025-12-04T09:02:03.0471513Z layer_output = apply_chunking_to_forward( 2025-12-04T09:02:03.0471801Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:02:03.0471894Z return forward_fn(*input_tensors) 2025-12-04T09:02:03.0472227Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 592, in feed_forward_chunk 2025-12-04T09:02:03.0472367Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:02:03.0472665Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 512, in forward 2025-12-04T09:02:03.0472756Z hidden_states = self.dense(hidden_states) 2025-12-04T09:02:03.0472760Z 2025-12-04T09:02:03.0472883Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:03.0473102Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:03.0473180Z res = mod(**inputs) 2025-12-04T09:02:03.0473477Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:03.0473567Z generator_hidden_states = self.convbert( 2025-12-04T09:02:03.0473872Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:03.0473949Z hidden_states = self.encoder( 2025-12-04T09:02:03.0474244Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:03.0474330Z layer_outputs = layer_module( 2025-12-04T09:02:03.0474574Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:03.0474686Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:03.0474975Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 585, in forward 2025-12-04T09:02:03.0475067Z layer_output = apply_chunking_to_forward( 2025-12-04T09:02:03.0475379Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:02:03.0475462Z return forward_fn(*input_tensors) 2025-12-04T09:02:03.0475810Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 592, in feed_forward_chunk 2025-12-04T09:02:03.0475940Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:02:03.0476269Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 513, in forward 2025-12-04T09:02:03.0476399Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:02:03.0476635Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:02:03.0476713Z return self.act(input) 2025-12-04T09:02:03.0476724Z 2025-12-04T09:02:03.0476836Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:03.0477053Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:03.0477129Z res = mod(**inputs) 2025-12-04T09:02:03.0477417Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:03.0477504Z generator_hidden_states = self.convbert( 2025-12-04T09:02:03.0477852Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:03.0477932Z hidden_states = self.encoder( 2025-12-04T09:02:03.0478314Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:03.0478401Z layer_outputs = layer_module( 2025-12-04T09:02:03.0478652Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:03.0478751Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:03.0479053Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 585, in forward 2025-12-04T09:02:03.0479144Z layer_output = apply_chunking_to_forward( 2025-12-04T09:02:03.0479445Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:02:03.0479533Z return forward_fn(*input_tensors) 2025-12-04T09:02:03.0479880Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-12-04T09:02:03.0480025Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:02:03.0480322Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 530, in forward 2025-12-04T09:02:03.0480423Z hidden_states = self.dense(hidden_states) 2025-12-04T09:02:03.0480427Z 2025-12-04T09:02:03.0480540Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:03.0480765Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:03.0480835Z res = mod(**inputs) 2025-12-04T09:02:03.0481130Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:03.0481228Z generator_hidden_states = self.convbert( 2025-12-04T09:02:03.0481563Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:03.0481672Z hidden_states = self.encoder( 2025-12-04T09:02:03.0481976Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:03.0482074Z layer_outputs = layer_module( 2025-12-04T09:02:03.0482320Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:03.0482404Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:03.0482691Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:02:03.0482785Z self_attention_outputs = self.attention( 2025-12-04T09:02:03.0483096Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:02:03.0483181Z self_outputs = self.self( 2025-12-04T09:02:03.0483510Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 349, in forward 2025-12-04T09:02:03.0483608Z mixed_query_layer = self.query(hidden_states) 2025-12-04T09:02:03.0483612Z 2025-12-04T09:02:03.0483730Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:03.0483947Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:03.0484017Z res = mod(**inputs) 2025-12-04T09:02:03.0484321Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:03.0484407Z generator_hidden_states = self.convbert( 2025-12-04T09:02:03.0484726Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:03.0484803Z hidden_states = self.encoder( 2025-12-04T09:02:03.0485099Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:03.0485183Z layer_outputs = layer_module( 2025-12-04T09:02:03.0485422Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:03.0485515Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:03.0485809Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:02:03.0485897Z self_attention_outputs = self.attention( 2025-12-04T09:02:03.0486197Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:02:03.0486274Z self_outputs = self.self( 2025-12-04T09:02:03.0486568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 343, in forward 2025-12-04T09:02:03.0486668Z mixed_key_layer = self.key(hidden_states) 2025-12-04T09:02:03.0486672Z 2025-12-04T09:02:03.0486783Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:03.0487006Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:03.0487077Z res = mod(**inputs) 2025-12-04T09:02:03.0487369Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:03.0487464Z generator_hidden_states = self.convbert( 2025-12-04T09:02:03.0487756Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:03.0487843Z hidden_states = self.encoder( 2025-12-04T09:02:03.0488135Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:03.0488211Z layer_outputs = layer_module( 2025-12-04T09:02:03.0488484Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:03.0488570Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:03.0488863Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:02:03.0488977Z self_attention_outputs = self.attention( 2025-12-04T09:02:03.0489268Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:02:03.0489352Z self_outputs = self.self( 2025-12-04T09:02:03.0489647Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 344, in forward 2025-12-04T09:02:03.0489767Z mixed_value_layer = self.value(hidden_states) 2025-12-04T09:02:03.0489771Z 2025-12-04T09:02:03.0489869Z cudagraph partition due to non gpu ops 2025-12-04T09:02:03.0489957Z cudagraph partition due to non gpu ops 2025-12-04T09:02:03.0490077Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:03.0490295Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:03.0490377Z res = mod(**inputs) 2025-12-04T09:02:03.0490679Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:03.0490765Z generator_hidden_states = self.convbert( 2025-12-04T09:02:03.0491057Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:03.0491167Z hidden_states = self.encoder( 2025-12-04T09:02:03.0491462Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:03.0491545Z layer_outputs = layer_module( 2025-12-04T09:02:03.0491790Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:03.0491873Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:03.0492187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:02:03.0492275Z self_attention_outputs = self.attention( 2025-12-04T09:02:03.0492569Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:02:03.0492653Z self_outputs = self.self( 2025-12-04T09:02:03.0492946Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 365, in forward 2025-12-04T09:02:03.0493068Z conv_out_layer = self.conv_out_layer(hidden_states) 2025-12-04T09:02:03.0493071Z 2025-12-04T09:02:03.0493156Z cudagraph partition due to non gpu ops 2025-12-04T09:02:03.0493268Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:03.0493492Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:03.0493563Z res = mod(**inputs) 2025-12-04T09:02:03.0493866Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:03.0493955Z generator_hidden_states = self.convbert( 2025-12-04T09:02:03.0494249Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:03.0494335Z hidden_states = self.encoder( 2025-12-04T09:02:03.0494628Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:03.0494706Z layer_outputs = layer_module( 2025-12-04T09:02:03.0494980Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:03.0495066Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:03.0495363Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:02:03.0495473Z self_attention_outputs = self.attention( 2025-12-04T09:02:03.0495765Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:02:03.0495848Z self_outputs = self.self( 2025-12-04T09:02:03.0496141Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 346, in forward 2025-12-04T09:02:03.0496322Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-12-04T09:02:03.0496631Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 281, in forward 2025-12-04T09:02:03.0496717Z x = self.depthwise(hidden_states) 2025-12-04T09:02:03.0496721Z 2025-12-04T09:02:03.0496837Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:03.0497055Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:03.0497122Z res = mod(**inputs) 2025-12-04T09:02:03.0497402Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:03.0497484Z generator_hidden_states = self.convbert( 2025-12-04T09:02:03.0497765Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:03.0497859Z hidden_states = self.encoder( 2025-12-04T09:02:03.0498154Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:03.0498240Z layer_outputs = layer_module( 2025-12-04T09:02:03.0498492Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:03.0498577Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:03.0498858Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:02:03.0498941Z self_attention_outputs = self.attention( 2025-12-04T09:02:03.0499228Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:02:03.0499299Z self_outputs = self.self( 2025-12-04T09:02:03.0499581Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 346, in forward 2025-12-04T09:02:03.0499749Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-12-04T09:02:03.0500031Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 282, in forward 2025-12-04T09:02:03.0500110Z x = self.pointwise(x) 2025-12-04T09:02:03.0500114Z 2025-12-04T09:02:03.0500219Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:03.0500426Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:03.0500498Z res = mod(**inputs) 2025-12-04T09:02:03.0500783Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:03.0500874Z generator_hidden_states = self.convbert( 2025-12-04T09:02:03.0501157Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:03.0501232Z hidden_states = self.encoder( 2025-12-04T09:02:03.0501552Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:03.0501626Z layer_outputs = layer_module( 2025-12-04T09:02:03.0501851Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:03.0501961Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:03.0502234Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:02:03.0502326Z self_attention_outputs = self.attention( 2025-12-04T09:02:03.0502603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:02:03.0502677Z self_outputs = self.self( 2025-12-04T09:02:03.0502988Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 359, in forward 2025-12-04T09:02:03.0503148Z conv_attn_layer = torch.multiply(mixed_key_conv_attn_layer, mixed_query_layer) 2025-12-04T09:02:03.0503152Z 2025-12-04T09:02:03.0503264Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:03.0503466Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:03.0503535Z res = mod(**inputs) 2025-12-04T09:02:03.0503824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:03.0503906Z generator_hidden_states = self.convbert( 2025-12-04T09:02:03.0504177Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:03.0504284Z hidden_states = self.encoder( 2025-12-04T09:02:03.0504561Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:03.0504641Z layer_outputs = layer_module( 2025-12-04T09:02:03.0504869Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:03.0504948Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:03.0505231Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:02:03.0505314Z self_attention_outputs = self.attention( 2025-12-04T09:02:03.0505595Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:02:03.0505665Z self_outputs = self.self( 2025-12-04T09:02:03.0505941Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 361, in forward 2025-12-04T09:02:03.0506069Z conv_kernel_layer = self.conv_kernel_layer(conv_attn_layer) 2025-12-04T09:02:03.0506073Z 2025-12-04T09:02:03.0506181Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:03.0506383Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:03.0506458Z res = mod(**inputs) 2025-12-04T09:02:03.0506734Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:03.0506824Z generator_hidden_states = self.convbert( 2025-12-04T09:02:03.0507098Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:03.0507169Z hidden_states = self.encoder( 2025-12-04T09:02:03.0507450Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:03.0507527Z layer_outputs = layer_module( 2025-12-04T09:02:03.0507783Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:03.0507864Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:03.0508140Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:02:03.0508252Z self_attention_outputs = self.attention( 2025-12-04T09:02:03.0508529Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:02:03.0508603Z self_outputs = self.self( 2025-12-04T09:02:03.0508904Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 379, in forward 2025-12-04T09:02:03.0509042Z conv_out_layer = torch.matmul(conv_out_layer, conv_kernel_layer) 2025-12-04T09:02:03.0509046Z 2025-12-04T09:02:03.0509159Z cudagraph partition due to non gpu ops 2025-12-04T09:02:03.0509244Z cudagraph partition due to non gpu ops 2025-12-04T09:02:03.0509356Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:03.0509579Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:03.0509649Z res = mod(**inputs) 2025-12-04T09:02:03.0509941Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:03.0510037Z generator_hidden_states = self.convbert( 2025-12-04T09:02:03.0510324Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:03.0510415Z hidden_states = self.encoder( 2025-12-04T09:02:03.0510706Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:03.0510779Z layer_outputs = layer_module( 2025-12-04T09:02:03.0511018Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:03.0511101Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:03.0511396Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:02:03.0511486Z self_attention_outputs = self.attention( 2025-12-04T09:02:03.0511776Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:02:03.0511861Z self_outputs = self.self( 2025-12-04T09:02:03.0512150Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 404, in forward 2025-12-04T09:02:03.0512271Z context_layer = torch.cat([context_layer, conv_out], 2) 2025-12-04T09:02:03.0512283Z 2025-12-04T09:02:03.0512394Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:03.0512607Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:03.0512685Z res = mod(**inputs) 2025-12-04T09:02:03.0512971Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:03.0513058Z generator_hidden_states = self.convbert( 2025-12-04T09:02:03.0513354Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:03.0513431Z hidden_states = self.encoder( 2025-12-04T09:02:03.0513729Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:03.0513804Z layer_outputs = layer_module( 2025-12-04T09:02:03.0514044Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:03.0514134Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:03.0514456Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:02:03.0514547Z self_attention_outputs = self.attention( 2025-12-04T09:02:03.0514862Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 470, in forward 2025-12-04T09:02:03.0515028Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:02:03.0515338Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 424, in forward 2025-12-04T09:02:03.0515428Z hidden_states = self.dense(hidden_states) 2025-12-04T09:02:03.0515434Z 2025-12-04T09:02:03.0515547Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:03.0515799Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:03.0515874Z res = mod(**inputs) 2025-12-04T09:02:03.0516182Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:03.0516273Z generator_hidden_states = self.convbert( 2025-12-04T09:02:03.0516570Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:03.0516659Z hidden_states = self.encoder( 2025-12-04T09:02:03.0516956Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:03.0517034Z layer_outputs = layer_module( 2025-12-04T09:02:03.0517288Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:03.0517398Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:03.0517709Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 585, in forward 2025-12-04T09:02:03.0517801Z layer_output = apply_chunking_to_forward( 2025-12-04T09:02:03.0518099Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:02:03.0518276Z return forward_fn(*input_tensors) 2025-12-04T09:02:03.0518622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 592, in feed_forward_chunk 2025-12-04T09:02:03.0518769Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:02:03.0519069Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 512, in forward 2025-12-04T09:02:03.0519166Z hidden_states = self.dense(hidden_states) 2025-12-04T09:02:03.0519170Z 2025-12-04T09:02:03.0519294Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:03.0519520Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:03.0519594Z res = mod(**inputs) 2025-12-04T09:02:03.0519905Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:03.0519998Z generator_hidden_states = self.convbert( 2025-12-04T09:02:03.0520322Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:03.0520401Z hidden_states = self.encoder( 2025-12-04T09:02:03.0520819Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:03.0520920Z layer_outputs = layer_module( 2025-12-04T09:02:03.0521168Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:03.0521261Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:03.0521612Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 585, in forward 2025-12-04T09:02:03.0521706Z layer_output = apply_chunking_to_forward( 2025-12-04T09:02:03.0522034Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:02:03.0522117Z return forward_fn(*input_tensors) 2025-12-04T09:02:03.0522448Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 592, in feed_forward_chunk 2025-12-04T09:02:03.0522589Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:02:03.0522914Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 513, in forward 2025-12-04T09:02:03.0523046Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:02:03.0523284Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:02:03.0523363Z return self.act(input) 2025-12-04T09:02:03.0523367Z 2025-12-04T09:02:03.0523488Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:03.0523709Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:03.0523790Z res = mod(**inputs) 2025-12-04T09:02:03.0524089Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:03.0524177Z generator_hidden_states = self.convbert( 2025-12-04T09:02:03.0524481Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:03.0524594Z hidden_states = self.encoder( 2025-12-04T09:02:03.0524890Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:03.0524974Z layer_outputs = layer_module( 2025-12-04T09:02:03.0525219Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:03.0525313Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:03.0525607Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 585, in forward 2025-12-04T09:02:03.0525696Z layer_output = apply_chunking_to_forward( 2025-12-04T09:02:03.0525987Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:02:03.0526071Z return forward_fn(*input_tensors) 2025-12-04T09:02:03.0526430Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-12-04T09:02:03.0526567Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:02:03.0526846Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 530, in forward 2025-12-04T09:02:03.0526940Z hidden_states = self.dense(hidden_states) 2025-12-04T09:02:03.0526945Z 2025-12-04T09:02:03.0527050Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:03.0527262Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:03.0527328Z res = mod(**inputs) 2025-12-04T09:02:03.0527603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:03.0527696Z generator_hidden_states = self.convbert( 2025-12-04T09:02:03.0527977Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:03.0528050Z hidden_states = self.encoder( 2025-12-04T09:02:03.0528359Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:03.0528433Z layer_outputs = layer_module( 2025-12-04T09:02:03.0528700Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:03.0528782Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:03.0529071Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:02:03.0529168Z self_attention_outputs = self.attention( 2025-12-04T09:02:03.0529460Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:02:03.0529558Z self_outputs = self.self( 2025-12-04T09:02:03.0529860Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 349, in forward 2025-12-04T09:02:03.0529958Z mixed_query_layer = self.query(hidden_states) 2025-12-04T09:02:03.0529962Z 2025-12-04T09:02:03.0530080Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:03.0530299Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:03.0530368Z res = mod(**inputs) 2025-12-04T09:02:03.0530670Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:03.0530758Z generator_hidden_states = self.convbert( 2025-12-04T09:02:03.0531054Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:03.0531153Z hidden_states = self.encoder( 2025-12-04T09:02:03.0531448Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:03.0531533Z layer_outputs = layer_module( 2025-12-04T09:02:03.0531776Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:03.0531861Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:03.0532163Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:02:03.0532251Z self_attention_outputs = self.attention( 2025-12-04T09:02:03.0532555Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:02:03.0532633Z self_outputs = self.self( 2025-12-04T09:02:03.0532925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 343, in forward 2025-12-04T09:02:03.0533022Z mixed_key_layer = self.key(hidden_states) 2025-12-04T09:02:03.0533025Z 2025-12-04T09:02:03.0533135Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:03.0533358Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:03.0533428Z res = mod(**inputs) 2025-12-04T09:02:03.0533723Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:03.0533818Z generator_hidden_states = self.convbert( 2025-12-04T09:02:03.0534110Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:03.0534186Z hidden_states = self.encoder( 2025-12-04T09:02:03.0534490Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:03.0534566Z layer_outputs = layer_module( 2025-12-04T09:02:03.0534836Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:03.0534920Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:03.0535210Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:02:03.0535332Z self_attention_outputs = self.attention( 2025-12-04T09:02:03.0535625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:02:03.0535706Z self_outputs = self.self( 2025-12-04T09:02:03.0535997Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 344, in forward 2025-12-04T09:02:03.0536096Z mixed_value_layer = self.value(hidden_states) 2025-12-04T09:02:03.0536119Z 2025-12-04T09:02:03.0536215Z cudagraph partition due to non gpu ops 2025-12-04T09:02:03.0536302Z cudagraph partition due to non gpu ops 2025-12-04T09:02:03.0536415Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:03.0536639Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:03.0536709Z res = mod(**inputs) 2025-12-04T09:02:03.0537015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:03.0537104Z generator_hidden_states = self.convbert( 2025-12-04T09:02:03.0537395Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:03.0537480Z hidden_states = self.encoder( 2025-12-04T09:02:03.0537791Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:03.0537868Z layer_outputs = layer_module( 2025-12-04T09:02:03.0538116Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:03.0538200Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:03.0538496Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:02:03.0538587Z self_attention_outputs = self.attention( 2025-12-04T09:02:03.0538878Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:02:03.0538963Z self_outputs = self.self( 2025-12-04T09:02:03.0539252Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 365, in forward 2025-12-04T09:02:03.0539371Z conv_out_layer = self.conv_out_layer(hidden_states) 2025-12-04T09:02:03.0539376Z 2025-12-04T09:02:03.0539469Z cudagraph partition due to non gpu ops 2025-12-04T09:02:03.0539574Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:03.0539786Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:03.0539851Z res = mod(**inputs) 2025-12-04T09:02:03.0540125Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:03.0540215Z generator_hidden_states = self.convbert( 2025-12-04T09:02:03.0540489Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:03.0540567Z hidden_states = self.encoder( 2025-12-04T09:02:03.0540842Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:03.0540916Z layer_outputs = layer_module( 2025-12-04T09:02:03.0541151Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:03.0541246Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:03.0541532Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:02:03.0541634Z self_attention_outputs = self.attention( 2025-12-04T09:02:03.0541911Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:02:03.0541991Z self_outputs = self.self( 2025-12-04T09:02:03.0542265Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 346, in forward 2025-12-04T09:02:03.0542426Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-12-04T09:02:03.0542730Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 281, in forward 2025-12-04T09:02:03.0542812Z x = self.depthwise(hidden_states) 2025-12-04T09:02:03.0542816Z 2025-12-04T09:02:03.0542927Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:03.0543131Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:03.0543198Z res = mod(**inputs) 2025-12-04T09:02:03.0543484Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:03.0543566Z generator_hidden_states = self.convbert( 2025-12-04T09:02:03.0543848Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:03.0543940Z hidden_states = self.encoder( 2025-12-04T09:02:03.0544227Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:03.0544308Z layer_outputs = layer_module( 2025-12-04T09:02:03.0544538Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:03.0544617Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:03.0544899Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:02:03.0544984Z self_attention_outputs = self.attention( 2025-12-04T09:02:03.0545269Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:02:03.0545341Z self_outputs = self.self( 2025-12-04T09:02:03.0545619Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 346, in forward 2025-12-04T09:02:03.0545789Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-12-04T09:02:03.0546070Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 282, in forward 2025-12-04T09:02:03.0546153Z x = self.pointwise(x) 2025-12-04T09:02:03.0546157Z 2025-12-04T09:02:03.0546261Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:03.0546464Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:03.0546537Z res = mod(**inputs) 2025-12-04T09:02:03.0546822Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:03.0546906Z generator_hidden_states = self.convbert( 2025-12-04T09:02:03.0547187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:03.0547264Z hidden_states = self.encoder( 2025-12-04T09:02:03.0547562Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:03.0547636Z layer_outputs = layer_module( 2025-12-04T09:02:03.0547863Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:03.0547969Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:03.0548242Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:02:03.0548328Z self_attention_outputs = self.attention( 2025-12-04T09:02:03.0548598Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:02:03.0548671Z self_outputs = self.self( 2025-12-04T09:02:03.0548970Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 359, in forward 2025-12-04T09:02:03.0549132Z conv_attn_layer = torch.multiply(mixed_key_conv_attn_layer, mixed_query_layer) 2025-12-04T09:02:03.0549136Z 2025-12-04T09:02:03.0549239Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:03.0549449Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:03.0549515Z res = mod(**inputs) 2025-12-04T09:02:03.0549797Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:03.0549879Z generator_hidden_states = self.convbert( 2025-12-04T09:02:03.0550158Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:03.0550257Z hidden_states = self.encoder( 2025-12-04T09:02:03.0550531Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:03.0550611Z layer_outputs = layer_module( 2025-12-04T09:02:03.0550839Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:03.0550917Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:03.0551196Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:02:03.0551279Z self_attention_outputs = self.attention( 2025-12-04T09:02:03.0551552Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:02:03.0551628Z self_outputs = self.self( 2025-12-04T09:02:03.0551915Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 361, in forward 2025-12-04T09:02:03.0552049Z conv_kernel_layer = self.conv_kernel_layer(conv_attn_layer) 2025-12-04T09:02:03.0552052Z 2025-12-04T09:02:03.0552158Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:03.0552360Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:03.0552433Z res = mod(**inputs) 2025-12-04T09:02:03.0552709Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:03.0552800Z generator_hidden_states = self.convbert( 2025-12-04T09:02:03.0553075Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:03.0553148Z hidden_states = self.encoder( 2025-12-04T09:02:03.0553430Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:03.0553503Z layer_outputs = layer_module( 2025-12-04T09:02:03.0553733Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:03.0553846Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:03.0554125Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:02:03.0554214Z self_attention_outputs = self.attention( 2025-12-04T09:02:03.0554522Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:02:03.0554599Z self_outputs = self.self( 2025-12-04T09:02:03.0554897Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 379, in forward 2025-12-04T09:02:03.0555035Z conv_out_layer = torch.matmul(conv_out_layer, conv_kernel_layer) 2025-12-04T09:02:03.0555040Z 2025-12-04T09:02:03.0555133Z cudagraph partition due to non gpu ops 2025-12-04T09:02:03.0555238Z cudagraph partition due to non gpu ops 2025-12-04T09:02:03.0555352Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:03.0555578Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:03.0555648Z res = mod(**inputs) 2025-12-04T09:02:03.0555944Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:03.0556042Z generator_hidden_states = self.convbert( 2025-12-04T09:02:03.0556333Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:03.0556426Z hidden_states = self.encoder( 2025-12-04T09:02:03.0556722Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:03.0556817Z layer_outputs = layer_module( 2025-12-04T09:02:03.0557072Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:03.0557157Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:03.0557453Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:02:03.0557548Z self_attention_outputs = self.attention( 2025-12-04T09:02:03.0557845Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:02:03.0557927Z self_outputs = self.self( 2025-12-04T09:02:03.0558295Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 404, in forward 2025-12-04T09:02:03.0558426Z context_layer = torch.cat([context_layer, conv_out], 2) 2025-12-04T09:02:03.0558434Z 2025-12-04T09:02:03.0558558Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:03.0558771Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:03.0558853Z res = mod(**inputs) 2025-12-04T09:02:03.0559153Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:03.0559243Z generator_hidden_states = self.convbert( 2025-12-04T09:02:03.0559557Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:03.0559636Z hidden_states = self.encoder( 2025-12-04T09:02:03.0559929Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:03.0560015Z layer_outputs = layer_module( 2025-12-04T09:02:03.0560260Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:03.0560356Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:03.0560674Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:02:03.0560763Z self_attention_outputs = self.attention( 2025-12-04T09:02:03.0561062Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 470, in forward 2025-12-04T09:02:03.0561227Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:02:03.0561527Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 424, in forward 2025-12-04T09:02:03.0561617Z hidden_states = self.dense(hidden_states) 2025-12-04T09:02:03.0561621Z 2025-12-04T09:02:03.0561731Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:03.0561977Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:03.0562048Z res = mod(**inputs) 2025-12-04T09:02:03.0562341Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:03.0562436Z generator_hidden_states = self.convbert( 2025-12-04T09:02:03.0562725Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:03.0562811Z hidden_states = self.encoder( 2025-12-04T09:02:03.0563100Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:03.0563175Z layer_outputs = layer_module( 2025-12-04T09:02:03.0563422Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:03.0563527Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:03.0563825Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 585, in forward 2025-12-04T09:02:03.0563917Z layer_output = apply_chunking_to_forward( 2025-12-04T09:02:03.0564202Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:02:03.0564291Z return forward_fn(*input_tensors) 2025-12-04T09:02:03.0564621Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 592, in feed_forward_chunk 2025-12-04T09:02:03.0564750Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:02:03.0565046Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 512, in forward 2025-12-04T09:02:03.0565137Z hidden_states = self.dense(hidden_states) 2025-12-04T09:02:03.0565140Z 2025-12-04T09:02:03.0565257Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:03.0565475Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:03.0565546Z res = mod(**inputs) 2025-12-04T09:02:03.0565843Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:03.0565929Z generator_hidden_states = self.convbert( 2025-12-04T09:02:03.0566228Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:03.0566305Z hidden_states = self.encoder( 2025-12-04T09:02:03.0566595Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:03.0566678Z layer_outputs = layer_module( 2025-12-04T09:02:03.0566923Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:03.0567005Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:03.0567324Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 585, in forward 2025-12-04T09:02:03.0567416Z layer_output = apply_chunking_to_forward( 2025-12-04T09:02:03.0567708Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:02:03.0567813Z return forward_fn(*input_tensors) 2025-12-04T09:02:03.0568142Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 592, in feed_forward_chunk 2025-12-04T09:02:03.0568271Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:02:03.0568543Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 513, in forward 2025-12-04T09:02:03.0568683Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:02:03.0568905Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:02:03.0568976Z return self.act(input) 2025-12-04T09:02:03.0568980Z 2025-12-04T09:02:03.0569091Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:03.0569293Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:03.0569367Z res = mod(**inputs) 2025-12-04T09:02:03.0569634Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:03.0569716Z generator_hidden_states = self.convbert( 2025-12-04T09:02:03.0569993Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:03.0570087Z hidden_states = self.encoder( 2025-12-04T09:02:03.0570356Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:03.0570439Z layer_outputs = layer_module( 2025-12-04T09:02:03.0570662Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:03.0570746Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:03.0571015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 585, in forward 2025-12-04T09:02:03.0571097Z layer_output = apply_chunking_to_forward( 2025-12-04T09:02:03.0571367Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:02:03.0571443Z return forward_fn(*input_tensors) 2025-12-04T09:02:03.0571746Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-12-04T09:02:03.0571886Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:02:03.0572158Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 530, in forward 2025-12-04T09:02:03.0572248Z hidden_states = self.dense(hidden_states) 2025-12-04T09:02:03.0572251Z 2025-12-04T09:02:03.0572355Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:03.0572555Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:03.0572626Z res = mod(**inputs) 2025-12-04T09:02:03.0572895Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:03.0572982Z generator_hidden_states = self.convbert( 2025-12-04T09:02:03.0573255Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:03.0573326Z hidden_states = self.encoder( 2025-12-04T09:02:03.0573618Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:03.0573689Z layer_outputs = layer_module( 2025-12-04T09:02:03.0573912Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:03.0574014Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:03.0574282Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:02:03.0574372Z self_attention_outputs = self.attention( 2025-12-04T09:02:03.0574651Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:02:03.0574725Z self_outputs = self.self( 2025-12-04T09:02:03.0575019Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 349, in forward 2025-12-04T09:02:03.0575115Z mixed_query_layer = self.query(hidden_states) 2025-12-04T09:02:03.0575119Z 2025-12-04T09:02:03.0575226Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:03.0575424Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:03.0575489Z res = mod(**inputs) 2025-12-04T09:02:03.0575774Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:03.0575855Z generator_hidden_states = self.convbert( 2025-12-04T09:02:03.0576122Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:03.0576229Z hidden_states = self.encoder( 2025-12-04T09:02:03.0576502Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:03.0576583Z layer_outputs = layer_module( 2025-12-04T09:02:03.0576806Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:03.0576883Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:03.0577159Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:02:03.0577240Z self_attention_outputs = self.attention( 2025-12-04T09:02:03.0577512Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:02:03.0577584Z self_outputs = self.self( 2025-12-04T09:02:03.0577852Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 343, in forward 2025-12-04T09:02:03.0577943Z mixed_key_layer = self.key(hidden_states) 2025-12-04T09:02:03.0577947Z 2025-12-04T09:02:03.0578048Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:03.0578252Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:03.0578316Z res = mod(**inputs) 2025-12-04T09:02:03.0578585Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:03.0578673Z generator_hidden_states = self.convbert( 2025-12-04T09:02:03.0578948Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:03.0579018Z hidden_states = self.encoder( 2025-12-04T09:02:03.0579292Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:03.0579364Z layer_outputs = layer_module( 2025-12-04T09:02:03.0579620Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:03.0579696Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:03.0579955Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:02:03.0580059Z self_attention_outputs = self.attention( 2025-12-04T09:02:03.0580322Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:02:03.0580391Z self_outputs = self.self( 2025-12-04T09:02:03.0580668Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 344, in forward 2025-12-04T09:02:03.0580755Z mixed_value_layer = self.value(hidden_states) 2025-12-04T09:02:03.0580758Z 2025-12-04T09:02:03.0580859Z cudagraph partition due to non gpu ops 2025-12-04T09:02:03.0580938Z cudagraph partition due to non gpu ops 2025-12-04T09:02:03.0581039Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:03.0581240Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:03.0581302Z res = mod(**inputs) 2025-12-04T09:02:03.0581570Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:03.0581650Z generator_hidden_states = self.convbert( 2025-12-04T09:02:03.0581909Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:03.0581984Z hidden_states = self.encoder( 2025-12-04T09:02:03.0582253Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:03.0582342Z layer_outputs = layer_module( 2025-12-04T09:02:03.0582566Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:03.0582643Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:03.0582910Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:02:03.0582990Z self_attention_outputs = self.attention( 2025-12-04T09:02:03.0583248Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:02:03.0583324Z self_outputs = self.self( 2025-12-04T09:02:03.0583586Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 365, in forward 2025-12-04T09:02:03.0583687Z conv_out_layer = self.conv_out_layer(hidden_states) 2025-12-04T09:02:03.0583698Z 2025-12-04T09:02:03.0583776Z cudagraph partition due to non gpu ops 2025-12-04T09:02:03.0583873Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:03.0584084Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:03.0584147Z res = mod(**inputs) 2025-12-04T09:02:03.0584406Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:03.0584492Z generator_hidden_states = self.convbert( 2025-12-04T09:02:03.0584755Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:03.0584832Z hidden_states = self.encoder( 2025-12-04T09:02:03.0585103Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:03.0585173Z layer_outputs = layer_module( 2025-12-04T09:02:03.0585397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:03.0585472Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:03.0585759Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:02:03.0585847Z self_attention_outputs = self.attention( 2025-12-04T09:02:03.0586128Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:02:03.0586206Z self_outputs = self.self( 2025-12-04T09:02:03.0586473Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 346, in forward 2025-12-04T09:02:03.0586631Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-12-04T09:02:03.0586927Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 281, in forward 2025-12-04T09:02:03.0587008Z x = self.depthwise(hidden_states) 2025-12-04T09:02:03.0587012Z 2025-12-04T09:02:03.0587122Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:03.0587319Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:03.0587383Z res = mod(**inputs) 2025-12-04T09:02:03.0587660Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:03.0587741Z generator_hidden_states = self.convbert( 2025-12-04T09:02:03.0588015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:03.0588093Z hidden_states = self.encoder( 2025-12-04T09:02:03.0588382Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:03.0588464Z layer_outputs = layer_module( 2025-12-04T09:02:03.0588686Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:03.0588762Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:03.0589035Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:02:03.0589116Z self_attention_outputs = self.attention( 2025-12-04T09:02:03.0589389Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:02:03.0589460Z self_outputs = self.self( 2025-12-04T09:02:03.0589725Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 346, in forward 2025-12-04T09:02:03.0589889Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-12-04T09:02:03.0590155Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 282, in forward 2025-12-04T09:02:03.0590228Z x = self.pointwise(x) 2025-12-04T09:02:03.0590240Z 2025-12-04T09:02:03.0590344Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:03.0590547Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:03.0590622Z res = mod(**inputs) 2025-12-04T09:02:03.0590898Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:03.0590979Z generator_hidden_states = self.convbert( 2025-12-04T09:02:03.0591261Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:03.0591333Z hidden_states = self.encoder( 2025-12-04T09:02:03.0591615Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:03.0591711Z layer_outputs = layer_module( 2025-12-04T09:02:03.0591938Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:03.0592026Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:03.0592329Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:02:03.0592410Z self_attention_outputs = self.attention( 2025-12-04T09:02:03.0592685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:02:03.0592755Z self_outputs = self.self( 2025-12-04T09:02:03.0593032Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 359, in forward 2025-12-04T09:02:03.0593199Z conv_attn_layer = torch.multiply(mixed_key_conv_attn_layer, mixed_query_layer) 2025-12-04T09:02:03.0593203Z 2025-12-04T09:02:03.0593309Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:03.0593512Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:03.0593576Z res = mod(**inputs) 2025-12-04T09:02:03.0593859Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:03.0593939Z generator_hidden_states = self.convbert( 2025-12-04T09:02:03.0594205Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:03.0594284Z hidden_states = self.encoder( 2025-12-04T09:02:03.0594580Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:03.0594653Z layer_outputs = layer_module( 2025-12-04T09:02:03.0594884Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:03.0594958Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:03.0595232Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:02:03.0595314Z self_attention_outputs = self.attention( 2025-12-04T09:02:03.0595578Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:02:03.0595653Z self_outputs = self.self( 2025-12-04T09:02:03.0595925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 361, in forward 2025-12-04T09:02:03.0596049Z conv_kernel_layer = self.conv_kernel_layer(conv_attn_layer) 2025-12-04T09:02:03.0596052Z 2025-12-04T09:02:03.0596154Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:03.0596350Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:03.0596419Z res = mod(**inputs) 2025-12-04T09:02:03.0596683Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:03.0596764Z generator_hidden_states = self.convbert( 2025-12-04T09:02:03.0597037Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:03.0597105Z hidden_states = self.encoder( 2025-12-04T09:02:03.0597377Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:03.0597449Z layer_outputs = layer_module( 2025-12-04T09:02:03.0597671Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:03.0597754Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:03.0598038Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:02:03.0598127Z self_attention_outputs = self.attention( 2025-12-04T09:02:03.0599056Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:02:03.0599134Z self_outputs = self.self( 2025-12-04T09:02:03.0599440Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 379, in forward 2025-12-04T09:02:03.0599582Z conv_out_layer = torch.matmul(conv_out_layer, conv_kernel_layer) 2025-12-04T09:02:03.0599590Z 2025-12-04T09:02:03.0599680Z cudagraph partition due to non gpu ops 2025-12-04T09:02:03.0599817Z cudagraph partition due to non gpu ops 2025-12-04T09:02:03.0599933Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:03.0600162Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:03.0600245Z res = mod(**inputs) 2025-12-04T09:02:03.0600517Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:03.0600608Z generator_hidden_states = self.convbert( 2025-12-04T09:02:03.0600881Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:03.0600960Z hidden_states = self.encoder( 2025-12-04T09:02:03.0601229Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:03.0601319Z layer_outputs = layer_module( 2025-12-04T09:02:03.0601553Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:03.0601630Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:03.0601902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:02:03.0601991Z self_attention_outputs = self.attention( 2025-12-04T09:02:03.0602262Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:02:03.0602339Z self_outputs = self.self( 2025-12-04T09:02:03.0602608Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 404, in forward 2025-12-04T09:02:03.0602718Z context_layer = torch.cat([context_layer, conv_out], 2) 2025-12-04T09:02:03.0602723Z 2025-12-04T09:02:03.0602832Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:03.0603029Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:03.0603100Z res = mod(**inputs) 2025-12-04T09:02:03.0603373Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:03.0603454Z generator_hidden_states = self.convbert( 2025-12-04T09:02:03.0603732Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:03.0603804Z hidden_states = self.encoder( 2025-12-04T09:02:03.0604076Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:03.0604153Z layer_outputs = layer_module( 2025-12-04T09:02:03.0604377Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:03.0604463Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:03.0604752Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:02:03.0604835Z self_attention_outputs = self.attention( 2025-12-04T09:02:03.0605110Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 470, in forward 2025-12-04T09:02:03.0605257Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:02:03.0605534Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 424, in forward 2025-12-04T09:02:03.0605619Z hidden_states = self.dense(hidden_states) 2025-12-04T09:02:03.0605622Z 2025-12-04T09:02:03.0605724Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:03.0605931Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:03.0606012Z res = mod(**inputs) 2025-12-04T09:02:03.0606282Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:03.0606368Z generator_hidden_states = self.convbert( 2025-12-04T09:02:03.0606637Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:03.0606716Z hidden_states = self.encoder( 2025-12-04T09:02:03.0606987Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:03.0607058Z layer_outputs = layer_module( 2025-12-04T09:02:03.0607287Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:03.0607385Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:03.0607653Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 585, in forward 2025-12-04T09:02:03.0607745Z layer_output = apply_chunking_to_forward( 2025-12-04T09:02:03.0608005Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:02:03.0608089Z return forward_fn(*input_tensors) 2025-12-04T09:02:03.0608389Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 592, in feed_forward_chunk 2025-12-04T09:02:03.0608511Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:02:03.0608785Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 512, in forward 2025-12-04T09:02:03.0608867Z hidden_states = self.dense(hidden_states) 2025-12-04T09:02:03.0608872Z 2025-12-04T09:02:03.0608980Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:03.0609179Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:03.0609245Z res = mod(**inputs) 2025-12-04T09:02:03.0609522Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:03.0609603Z generator_hidden_states = self.convbert( 2025-12-04T09:02:03.0609876Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:03.0609949Z hidden_states = self.encoder( 2025-12-04T09:02:03.0610212Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:03.0610290Z layer_outputs = layer_module( 2025-12-04T09:02:03.0610510Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:03.0610590Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:03.0610880Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 585, in forward 2025-12-04T09:02:03.0610964Z layer_output = apply_chunking_to_forward( 2025-12-04T09:02:03.0611230Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:02:03.0611327Z return forward_fn(*input_tensors) 2025-12-04T09:02:03.0611630Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 592, in feed_forward_chunk 2025-12-04T09:02:03.0611756Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:02:03.0612026Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 513, in forward 2025-12-04T09:02:03.0612146Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:02:03.0612379Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:02:03.0612453Z return self.act(input) 2025-12-04T09:02:03.0612457Z 2025-12-04T09:02:03.0612564Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:03.0612770Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:03.0612837Z res = mod(**inputs) 2025-12-04T09:02:03.0613113Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:03.0613192Z generator_hidden_states = self.convbert( 2025-12-04T09:02:03.0613466Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:03.0613555Z hidden_states = self.encoder( 2025-12-04T09:02:03.0613827Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:03.0613904Z layer_outputs = layer_module( 2025-12-04T09:02:03.0614128Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:03.0614204Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:03.0614478Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 585, in forward 2025-12-04T09:02:03.0614563Z layer_output = apply_chunking_to_forward( 2025-12-04T09:02:03.0614829Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:02:03.0614904Z return forward_fn(*input_tensors) 2025-12-04T09:02:03.0615202Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-12-04T09:02:03.0615345Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:02:03.0615619Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 530, in forward 2025-12-04T09:02:03.0615708Z hidden_states = self.dense(hidden_states) 2025-12-04T09:02:03.0615711Z 2025-12-04T09:02:03.0615813Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:03.0616015Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:03.0616089Z res = mod(**inputs) 2025-12-04T09:02:03.0616360Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:03.0616447Z generator_hidden_states = self.convbert( 2025-12-04T09:02:03.0616725Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:03.0616798Z hidden_states = self.encoder( 2025-12-04T09:02:03.0617088Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:03.0617159Z layer_outputs = layer_module( 2025-12-04T09:02:03.0617377Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:03.0617479Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:03.0617738Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:02:03.0617825Z self_attention_outputs = self.attention( 2025-12-04T09:02:03.0618083Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:02:03.0618154Z self_outputs = self.self( 2025-12-04T09:02:03.0618439Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 349, in forward 2025-12-04T09:02:03.0618530Z mixed_query_layer = self.query(hidden_states) 2025-12-04T09:02:03.0618534Z 2025-12-04T09:02:03.0618640Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:03.0618832Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:03.0618898Z res = mod(**inputs) 2025-12-04T09:02:03.0619166Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:03.0619245Z generator_hidden_states = self.convbert( 2025-12-04T09:02:03.0619503Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:03.0619599Z hidden_states = self.encoder( 2025-12-04T09:02:03.0619860Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:03.0619936Z layer_outputs = layer_module( 2025-12-04T09:02:03.0620152Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:03.0620240Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:03.0620509Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:02:03.0620588Z self_attention_outputs = self.attention( 2025-12-04T09:02:03.0620977Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:02:03.0621061Z self_outputs = self.self( 2025-12-04T09:02:03.0621322Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 343, in forward 2025-12-04T09:02:03.0621418Z mixed_key_layer = self.key(hidden_states) 2025-12-04T09:02:03.0621422Z 2025-12-04T09:02:03.0621521Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:03.0621714Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:03.0621787Z res = mod(**inputs) 2025-12-04T09:02:03.0622049Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:03.0622138Z generator_hidden_states = self.convbert( 2025-12-04T09:02:03.0622414Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:03.0622484Z hidden_states = self.encoder( 2025-12-04T09:02:03.0622758Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:03.0622829Z layer_outputs = layer_module( 2025-12-04T09:02:03.0623050Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:03.0623184Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:03.0623450Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:02:03.0623536Z self_attention_outputs = self.attention( 2025-12-04T09:02:03.0623846Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:02:03.0623919Z self_outputs = self.self( 2025-12-04T09:02:03.0624206Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 344, in forward 2025-12-04T09:02:03.0624299Z mixed_value_layer = self.value(hidden_states) 2025-12-04T09:02:03.0624304Z 2025-12-04T09:02:03.0624395Z cudagraph partition due to non gpu ops 2025-12-04T09:02:03.0624508Z cudagraph partition due to non gpu ops 2025-12-04T09:02:03.0624623Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:03.0624849Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:03.0624920Z res = mod(**inputs) 2025-12-04T09:02:03.0625214Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:03.0625312Z generator_hidden_states = self.convbert( 2025-12-04T09:02:03.0625604Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:03.0625686Z hidden_states = self.encoder( 2025-12-04T09:02:03.0625980Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:03.0626087Z layer_outputs = layer_module( 2025-12-04T09:02:03.0626335Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:03.0626420Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:03.0626718Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:02:03.0626804Z self_attention_outputs = self.attention( 2025-12-04T09:02:03.0627096Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:02:03.0627178Z self_outputs = self.self( 2025-12-04T09:02:03.0627469Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 365, in forward 2025-12-04T09:02:03.0627579Z conv_out_layer = self.conv_out_layer(hidden_states) 2025-12-04T09:02:03.0627584Z 2025-12-04T09:02:03.0627683Z cudagraph partition due to non gpu ops 2025-12-04T09:02:03.0627787Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:03.0627998Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:03.0628063Z res = mod(**inputs) 2025-12-04T09:02:03.0647418Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:03.0647710Z generator_hidden_states = self.convbert( 2025-12-04T09:02:03.0648054Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:03.0648140Z hidden_states = self.encoder( 2025-12-04T09:02:03.0648440Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:03.0648526Z layer_outputs = layer_module( 2025-12-04T09:02:03.0648772Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:03.0648858Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:03.0649215Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:02:03.0649314Z self_attention_outputs = self.attention( 2025-12-04T09:02:03.0649582Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:02:03.0649697Z self_outputs = self.self( 2025-12-04T09:02:03.0649961Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 346, in forward 2025-12-04T09:02:03.0650120Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-12-04T09:02:03.0650395Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 281, in forward 2025-12-04T09:02:03.0650506Z x = self.depthwise(hidden_states) 2025-12-04T09:02:03.0650514Z 2025-12-04T09:02:03.0650634Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:03.0650840Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:03.0650909Z res = mod(**inputs) 2025-12-04T09:02:03.0651180Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:03.0651265Z generator_hidden_states = self.convbert( 2025-12-04T09:02:03.0651526Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:03.0651609Z hidden_states = self.encoder( 2025-12-04T09:02:03.0651869Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:03.0651973Z layer_outputs = layer_module( 2025-12-04T09:02:03.0652196Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:03.0652276Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:03.0652545Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:02:03.0652626Z self_attention_outputs = self.attention( 2025-12-04T09:02:03.0652887Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:02:03.0652967Z self_outputs = self.self( 2025-12-04T09:02:03.0653229Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 346, in forward 2025-12-04T09:02:03.0653394Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-12-04T09:02:03.0653657Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 282, in forward 2025-12-04T09:02:03.0653728Z x = self.pointwise(x) 2025-12-04T09:02:03.0653734Z 2025-12-04T09:02:03.0653847Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:03.0654041Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:03.0654116Z res = mod(**inputs) 2025-12-04T09:02:03.0654377Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:03.0654460Z generator_hidden_states = self.convbert( 2025-12-04T09:02:03.0654728Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:03.0654799Z hidden_states = self.encoder( 2025-12-04T09:02:03.0655070Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:03.0655139Z layer_outputs = layer_module( 2025-12-04T09:02:03.0655380Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:03.0655466Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:03.0655726Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:02:03.0655826Z self_attention_outputs = self.attention( 2025-12-04T09:02:03.0656095Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:02:03.0656163Z self_outputs = self.self( 2025-12-04T09:02:03.0656432Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 359, in forward 2025-12-04T09:02:03.0656620Z conv_attn_layer = torch.multiply(mixed_key_conv_attn_layer, mixed_query_layer) 2025-12-04T09:02:03.0656624Z 2025-12-04T09:02:03.0656730Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:03.0656931Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:03.0656994Z res = mod(**inputs) 2025-12-04T09:02:03.0657261Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:03.0657339Z generator_hidden_states = self.convbert( 2025-12-04T09:02:03.0657596Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:03.0657671Z hidden_states = self.encoder( 2025-12-04T09:02:03.0657927Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:03.0658016Z layer_outputs = layer_module( 2025-12-04T09:02:03.0658240Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:03.0658315Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:03.0658581Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:02:03.0658661Z self_attention_outputs = self.attention( 2025-12-04T09:02:03.0658921Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:02:03.0658997Z self_outputs = self.self( 2025-12-04T09:02:03.0659258Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 361, in forward 2025-12-04T09:02:03.0659374Z conv_kernel_layer = self.conv_kernel_layer(conv_attn_layer) 2025-12-04T09:02:03.0659385Z 2025-12-04T09:02:03.0659487Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:03.0659678Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:03.0659750Z res = mod(**inputs) 2025-12-04T09:02:03.0660008Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:03.0660087Z generator_hidden_states = self.convbert( 2025-12-04T09:02:03.0660357Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:03.0660426Z hidden_states = self.encoder( 2025-12-04T09:02:03.0660694Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:03.0660764Z layer_outputs = layer_module( 2025-12-04T09:02:03.0660979Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:03.0661062Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:03.0661340Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:02:03.0661421Z self_attention_outputs = self.attention( 2025-12-04T09:02:03.0661690Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:02:03.0661776Z self_outputs = self.self( 2025-12-04T09:02:03.0662043Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 379, in forward 2025-12-04T09:02:03.0662169Z conv_out_layer = torch.matmul(conv_out_layer, conv_kernel_layer) 2025-12-04T09:02:03.0662173Z 2025-12-04T09:02:03.0662255Z cudagraph partition due to non gpu ops 2025-12-04T09:02:03.0662345Z cudagraph partition due to non gpu ops 2025-12-04T09:02:03.0662463Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:03.0662670Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:03.0662736Z res = mod(**inputs) 2025-12-04T09:02:03.0663000Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:03.0663086Z generator_hidden_states = self.convbert( 2025-12-04T09:02:03.0663359Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:03.0663429Z hidden_states = self.encoder( 2025-12-04T09:02:03.0663701Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:03.0663771Z layer_outputs = layer_module( 2025-12-04T09:02:03.0664016Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:03.0664092Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:03.0664355Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:02:03.0664441Z self_attention_outputs = self.attention( 2025-12-04T09:02:03.0664701Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:02:03.0664776Z self_outputs = self.self( 2025-12-04T09:02:03.0665033Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 404, in forward 2025-12-04T09:02:03.0665160Z context_layer = torch.cat([context_layer, conv_out], 2) 2025-12-04T09:02:03.0665164Z 2025-12-04T09:02:03.0665266Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:03.0665465Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:03.0665537Z res = mod(**inputs) 2025-12-04T09:02:03.0665804Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:03.0665890Z generator_hidden_states = self.convbert( 2025-12-04T09:02:03.0666155Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:03.0666226Z hidden_states = self.encoder( 2025-12-04T09:02:03.0666498Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:03.0666568Z layer_outputs = layer_module( 2025-12-04T09:02:03.0666787Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:03.0666871Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:03.0667149Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:02:03.0667255Z self_attention_outputs = self.attention( 2025-12-04T09:02:03.0667532Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 470, in forward 2025-12-04T09:02:03.0667660Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:02:03.0667943Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 424, in forward 2025-12-04T09:02:03.0668027Z hidden_states = self.dense(hidden_states) 2025-12-04T09:02:03.0668030Z 2025-12-04T09:02:03.0668138Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:03.0668332Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:03.0668399Z res = mod(**inputs) 2025-12-04T09:02:03.0668695Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:03.0668778Z generator_hidden_states = self.convbert( 2025-12-04T09:02:03.0669043Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:03.0669122Z hidden_states = self.encoder( 2025-12-04T09:02:03.0669389Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:03.0669467Z layer_outputs = layer_module( 2025-12-04T09:02:03.0669688Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:03.0669764Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:03.0670062Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 585, in forward 2025-12-04T09:02:03.0670151Z layer_output = apply_chunking_to_forward( 2025-12-04T09:02:03.0670424Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:02:03.0670505Z return forward_fn(*input_tensors) 2025-12-04T09:02:03.0670812Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 592, in feed_forward_chunk 2025-12-04T09:02:03.0670947Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:02:03.0671217Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 512, in forward 2025-12-04T09:02:03.0671300Z hidden_states = self.dense(hidden_states) 2025-12-04T09:02:03.0671311Z 2025-12-04T09:02:03.0671414Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:03.0671617Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:03.0671691Z res = mod(**inputs) 2025-12-04T09:02:03.0671963Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:03.0672045Z generator_hidden_states = self.convbert( 2025-12-04T09:02:03.0672322Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:03.0672395Z hidden_states = self.encoder( 2025-12-04T09:02:03.0672668Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:03.0672739Z layer_outputs = layer_module( 2025-12-04T09:02:03.0672959Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:03.0673046Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:03.0673314Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 585, in forward 2025-12-04T09:02:03.0673414Z layer_output = apply_chunking_to_forward( 2025-12-04T09:02:03.0673685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:02:03.0673764Z return forward_fn(*input_tensors) 2025-12-04T09:02:03.0674098Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 592, in feed_forward_chunk 2025-12-04T09:02:03.0674221Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:02:03.0674490Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 513, in forward 2025-12-04T09:02:03.0674612Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:02:03.0674846Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:02:03.0674927Z return self.act(input) 2025-12-04T09:02:03.0674931Z 2025-12-04T09:02:03.0675034Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:03.0675233Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:03.0675308Z res = mod(**inputs) 2025-12-04T09:02:03.0675586Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:03.0675669Z generator_hidden_states = self.convbert( 2025-12-04T09:02:03.0675950Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:03.0676023Z hidden_states = self.encoder( 2025-12-04T09:02:03.0676328Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:03.0676401Z layer_outputs = layer_module( 2025-12-04T09:02:03.0676629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:03.0676716Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:03.0676995Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 585, in forward 2025-12-04T09:02:03.0677094Z layer_output = apply_chunking_to_forward( 2025-12-04T09:02:03.0677374Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:02:03.0677455Z return forward_fn(*input_tensors) 2025-12-04T09:02:03.0677791Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-12-04T09:02:03.0677940Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:02:03.0678321Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 530, in forward 2025-12-04T09:02:03.0678429Z hidden_states = self.dense(hidden_states) 2025-12-04T09:02:03.0678433Z 2025-12-04T09:02:03.0678545Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:03.0678772Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:03.0678846Z res = mod(**inputs) 2025-12-04T09:02:03.0679143Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:03.0679243Z generator_hidden_states = self.convbert( 2025-12-04T09:02:03.0679555Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:03.0679638Z hidden_states = self.encoder( 2025-12-04T09:02:03.0679917Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:03.0680015Z layer_outputs = layer_module( 2025-12-04T09:02:03.0680253Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:03.0680332Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:03.0680640Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:02:03.0680731Z self_attention_outputs = self.attention( 2025-12-04T09:02:03.0681003Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:02:03.0681081Z self_outputs = self.self( 2025-12-04T09:02:03.0681374Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 349, in forward 2025-12-04T09:02:03.0681471Z mixed_query_layer = self.query(hidden_states) 2025-12-04T09:02:03.0681475Z 2025-12-04T09:02:03.0681588Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:03.0681794Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:03.0681867Z res = mod(**inputs) 2025-12-04T09:02:03.0682141Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:03.0682222Z generator_hidden_states = self.convbert( 2025-12-04T09:02:03.0682501Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:03.0682575Z hidden_states = self.encoder( 2025-12-04T09:02:03.0682868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:03.0682947Z layer_outputs = layer_module( 2025-12-04T09:02:03.0683176Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:03.0683262Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:03.0683536Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:02:03.0683621Z self_attention_outputs = self.attention( 2025-12-04T09:02:03.0683902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:02:03.0683977Z self_outputs = self.self( 2025-12-04T09:02:03.0684276Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 343, in forward 2025-12-04T09:02:03.0684366Z mixed_key_layer = self.key(hidden_states) 2025-12-04T09:02:03.0684369Z 2025-12-04T09:02:03.0684480Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:03.0684707Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:03.0684777Z res = mod(**inputs) 2025-12-04T09:02:03.0685065Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:03.0685162Z generator_hidden_states = self.convbert( 2025-12-04T09:02:03.0685449Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:03.0685532Z hidden_states = self.encoder( 2025-12-04T09:02:03.0685818Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:03.0685897Z layer_outputs = layer_module( 2025-12-04T09:02:03.0686144Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:03.0686227Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:03.0686541Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:02:03.0686629Z self_attention_outputs = self.attention( 2025-12-04T09:02:03.0686920Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:02:03.0687024Z self_outputs = self.self( 2025-12-04T09:02:03.0687327Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 344, in forward 2025-12-04T09:02:03.0687426Z mixed_value_layer = self.value(hidden_states) 2025-12-04T09:02:03.0687430Z 2025-12-04T09:02:03.0687527Z cudagraph partition due to non gpu ops 2025-12-04T09:02:03.0687613Z cudagraph partition due to non gpu ops 2025-12-04T09:02:03.0687748Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:03.0687968Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:03.0688037Z res = mod(**inputs) 2025-12-04T09:02:03.0688337Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:03.0688426Z generator_hidden_states = self.convbert( 2025-12-04T09:02:03.0688718Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:03.0688802Z hidden_states = self.encoder( 2025-12-04T09:02:03.0689093Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:03.0689196Z layer_outputs = layer_module( 2025-12-04T09:02:03.0689446Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:03.0689529Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:03.0689834Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:02:03.0689922Z self_attention_outputs = self.attention( 2025-12-04T09:02:03.0690222Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:02:03.0690299Z self_outputs = self.self( 2025-12-04T09:02:03.0690594Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 365, in forward 2025-12-04T09:02:03.0690712Z conv_out_layer = self.conv_out_layer(hidden_states) 2025-12-04T09:02:03.0690716Z 2025-12-04T09:02:03.0690803Z cudagraph partition due to non gpu ops 2025-12-04T09:02:03.0690914Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:03.0691135Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:03.0691204Z res = mod(**inputs) 2025-12-04T09:02:03.0691507Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:03.0691593Z generator_hidden_states = self.convbert( 2025-12-04T09:02:03.0691889Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:03.0691973Z hidden_states = self.encoder( 2025-12-04T09:02:03.0692266Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:03.0692349Z layer_outputs = layer_module( 2025-12-04T09:02:03.0692590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:03.0692676Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:03.0692995Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:02:03.0693085Z self_attention_outputs = self.attention( 2025-12-04T09:02:03.0693378Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:02:03.0693481Z self_outputs = self.self( 2025-12-04T09:02:03.0693769Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 346, in forward 2025-12-04T09:02:03.0693947Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-12-04T09:02:03.0694237Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 281, in forward 2025-12-04T09:02:03.0694324Z x = self.depthwise(hidden_states) 2025-12-04T09:02:03.0694354Z 2025-12-04T09:02:03.0694475Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:03.0694692Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:03.0694769Z res = mod(**inputs) 2025-12-04T09:02:03.0695060Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:03.0695149Z generator_hidden_states = self.convbert( 2025-12-04T09:02:03.0695447Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:03.0695525Z hidden_states = self.encoder( 2025-12-04T09:02:03.0695814Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:03.0695921Z layer_outputs = layer_module( 2025-12-04T09:02:03.0696164Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:03.0696253Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:03.0696545Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:02:03.0696632Z self_attention_outputs = self.attention( 2025-12-04T09:02:03.0696928Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:02:03.0697014Z self_outputs = self.self( 2025-12-04T09:02:03.0697313Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 346, in forward 2025-12-04T09:02:03.0697474Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-12-04T09:02:03.0697757Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 282, in forward 2025-12-04T09:02:03.0697838Z x = self.pointwise(x) 2025-12-04T09:02:03.0697842Z 2025-12-04T09:02:03.0697948Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:03.0698154Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:03.0698219Z res = mod(**inputs) 2025-12-04T09:02:03.0698500Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:03.0698591Z generator_hidden_states = self.convbert( 2025-12-04T09:02:03.0698866Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:03.0698944Z hidden_states = self.encoder( 2025-12-04T09:02:03.0699225Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:03.0699297Z layer_outputs = layer_module( 2025-12-04T09:02:03.0699547Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:03.0699627Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:03.0699901Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:02:03.0700011Z self_attention_outputs = self.attention( 2025-12-04T09:02:03.0700286Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:02:03.0700363Z self_outputs = self.self( 2025-12-04T09:02:03.0700638Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 359, in forward 2025-12-04T09:02:03.0700796Z conv_attn_layer = torch.multiply(mixed_key_conv_attn_layer, mixed_query_layer) 2025-12-04T09:02:03.0700818Z 2025-12-04T09:02:03.0700942Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:03.0701137Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:03.0701207Z res = mod(**inputs) 2025-12-04T09:02:03.0701473Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:03.0701554Z generator_hidden_states = self.convbert( 2025-12-04T09:02:03.0701827Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:03.0701897Z hidden_states = self.encoder( 2025-12-04T09:02:03.0702168Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:03.0702269Z layer_outputs = layer_module( 2025-12-04T09:02:03.0702502Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:03.0702587Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:03.0702871Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:02:03.0702951Z self_attention_outputs = self.attention( 2025-12-04T09:02:03.0703242Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:02:03.0703319Z self_outputs = self.self( 2025-12-04T09:02:03.0703599Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 361, in forward 2025-12-04T09:02:03.0703717Z conv_kernel_layer = self.conv_kernel_layer(conv_attn_layer) 2025-12-04T09:02:03.0703722Z 2025-12-04T09:02:03.0703823Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:03.0704032Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:03.0704096Z res = mod(**inputs) 2025-12-04T09:02:03.0704377Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:03.0704462Z generator_hidden_states = self.convbert( 2025-12-04T09:02:03.0704732Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:03.0704810Z hidden_states = self.encoder( 2025-12-04T09:02:03.0705079Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:03.0705147Z layer_outputs = layer_module( 2025-12-04T09:02:03.0705378Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:03.0705457Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:03.0705764Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:02:03.0705846Z self_attention_outputs = self.attention( 2025-12-04T09:02:03.0706115Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:02:03.0706211Z self_outputs = self.self( 2025-12-04T09:02:03.0706481Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 379, in forward 2025-12-04T09:02:03.0706608Z conv_out_layer = torch.matmul(conv_out_layer, conv_kernel_layer) 2025-12-04T09:02:03.0706619Z 2025-12-04T09:02:03.0706697Z cudagraph partition due to non gpu ops 2025-12-04T09:02:03.0706776Z cudagraph partition due to non gpu ops 2025-12-04T09:02:03.0706888Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:03.0707104Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:03.0707170Z res = mod(**inputs) 2025-12-04T09:02:03.0707451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:03.0707531Z generator_hidden_states = self.convbert( 2025-12-04T09:02:03.0707810Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:03.0707881Z hidden_states = self.encoder( 2025-12-04T09:02:03.0708158Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:03.0708235Z layer_outputs = layer_module( 2025-12-04T09:02:03.0708461Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:03.0708560Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:03.0708838Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:02:03.0708918Z self_attention_outputs = self.attention( 2025-12-04T09:02:03.0709194Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 463, in forward 2025-12-04T09:02:03.0709267Z self_outputs = self.self( 2025-12-04T09:02:03.0709541Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 404, in forward 2025-12-04T09:02:03.0709663Z context_layer = torch.cat([context_layer, conv_out], 2) 2025-12-04T09:02:03.0709667Z 2025-12-04T09:02:03.0709770Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:03.0709980Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:03.0710048Z res = mod(**inputs) 2025-12-04T09:02:03.0710325Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:03.0710413Z generator_hidden_states = self.convbert( 2025-12-04T09:02:03.0710698Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:03.0710772Z hidden_states = self.encoder( 2025-12-04T09:02:03.0711054Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:03.0711125Z layer_outputs = layer_module( 2025-12-04T09:02:03.0711358Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:03.0711434Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:03.0711711Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 560, in forward 2025-12-04T09:02:03.0711810Z self_attention_outputs = self.attention( 2025-12-04T09:02:03.0712097Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 470, in forward 2025-12-04T09:02:03.0712234Z attention_output = self.output(self_outputs[0], hidden_states) 2025-12-04T09:02:03.0712521Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 424, in forward 2025-12-04T09:02:03.0712603Z hidden_states = self.dense(hidden_states) 2025-12-04T09:02:03.0712607Z 2025-12-04T09:02:03.0712715Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:03.0712914Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:03.0712980Z res = mod(**inputs) 2025-12-04T09:02:03.0713268Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:03.0713349Z generator_hidden_states = self.convbert( 2025-12-04T09:02:03.0713624Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:03.0713696Z hidden_states = self.encoder( 2025-12-04T09:02:03.0713973Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:03.0714053Z layer_outputs = layer_module( 2025-12-04T09:02:03.0714271Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:03.0714352Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:03.0714618Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 585, in forward 2025-12-04T09:02:03.0714725Z layer_output = apply_chunking_to_forward( 2025-12-04T09:02:03.0715000Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:02:03.0715078Z return forward_fn(*input_tensors) 2025-12-04T09:02:03.0715388Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 592, in feed_forward_chunk 2025-12-04T09:02:03.0715521Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:02:03.0715795Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 512, in forward 2025-12-04T09:02:03.0715883Z hidden_states = self.dense(hidden_states) 2025-12-04T09:02:03.0715886Z 2025-12-04T09:02:03.0715989Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:03.0716195Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:03.0716268Z res = mod(**inputs) 2025-12-04T09:02:03.0716549Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:03.0716640Z generator_hidden_states = self.convbert( 2025-12-04T09:02:03.0716920Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:03.0716994Z hidden_states = self.encoder( 2025-12-04T09:02:03.0717272Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:03.0717343Z layer_outputs = layer_module( 2025-12-04T09:02:03.0717566Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:03.0717652Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:03.0717928Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 585, in forward 2025-12-04T09:02:03.0718017Z layer_output = apply_chunking_to_forward( 2025-12-04T09:02:03.0718394Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:02:03.0718481Z return forward_fn(*input_tensors) 2025-12-04T09:02:03.0718818Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 592, in feed_forward_chunk 2025-12-04T09:02:03.0718972Z intermediate_output = self.intermediate(attention_output) 2025-12-04T09:02:03.0719291Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 513, in forward 2025-12-04T09:02:03.0719413Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-12-04T09:02:03.0719650Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/activations.py", line 85, in forward 2025-12-04T09:02:03.0719775Z return self.act(input) 2025-12-04T09:02:03.0719780Z 2025-12-04T09:02:03.0719895Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:03.0720120Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:03.0720192Z res = mod(**inputs) 2025-12-04T09:02:03.0720465Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 924, in forward 2025-12-04T09:02:03.0720553Z generator_hidden_states = self.convbert( 2025-12-04T09:02:03.0720967Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 852, in forward 2025-12-04T09:02:03.0721045Z hidden_states = self.encoder( 2025-12-04T09:02:03.0721325Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 624, in forward 2025-12-04T09:02:03.0721468Z layer_outputs = layer_module( 2025-12-04T09:02:03.0721710Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-12-04T09:02:03.0721791Z return super().__call__(*args, **kwargs) 2025-12-04T09:02:03.0722082Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 585, in forward 2025-12-04T09:02:03.0722175Z layer_output = apply_chunking_to_forward( 2025-12-04T09:02:03.0722447Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/pytorch_utils.py", line 257, in apply_chunking_to_forward 2025-12-04T09:02:03.0722524Z return forward_fn(*input_tensors) 2025-12-04T09:02:03.0722848Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-12-04T09:02:03.0722988Z layer_output = self.output(intermediate_output, attention_output) 2025-12-04T09:02:03.0723279Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 530, in forward 2025-12-04T09:02:03.0723363Z hidden_states = self.dense(hidden_states) 2025-12-04T09:02:03.0723367Z 2025-12-04T09:02:03.0723471Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:03.0723684Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:03.0723752Z res = mod(**inputs) 2025-12-04T09:02:03.0724041Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 937, in forward 2025-12-04T09:02:03.0724197Z prediction_scores = self.generator_predictions(generator_sequence_output) 2025-12-04T09:02:03.0724481Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 875, in forward 2025-12-04T09:02:03.0724595Z hidden_states = self.dense(generator_hidden_states) 2025-12-04T09:02:03.0724598Z 2025-12-04T09:02:03.0724704Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:03.0724939Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:03.0725015Z res = mod(**inputs) 2025-12-04T09:02:03.0725290Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 938, in forward 2025-12-04T09:02:03.0725459Z prediction_scores = self.generator_lm_head(prediction_scores) 2025-12-04T09:02:03.0725463Z 2025-12-04T09:02:03.0725566Z cudagraph partition due to non gpu ops. Found from : 2025-12-04T09:02:03.0725768Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 553, in forward_pass 2025-12-04T09:02:03.0725841Z res = mod(**inputs) 2025-12-04T09:02:03.0726119Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 944, in forward 2025-12-04T09:02:03.0726329Z loss = loss_fct(prediction_scores.view(-1, self.config.vocab_size), labels.view(-1)) 2025-12-04T09:02:03.0726333Z 2025-12-04T09:02:14.8376124Z Compilation time (from dynamo_timed): 22.923459457 2025-12-04T09:02:14.8429880Z pass 2025-12-04T09:02:14.8430252Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-12-04T09:02:14.8431080Z TIMING: _recursive_pre_grad_passes:0.01027 _recursive_joint_graph_passes:0.64342 _recursive_post_grad_passes:0.49691 async_compile.wait:0.71611 code_gen:10.69563 inductor_compile:13.20621 backend_compile:18.3352 gc:0.0002 entire_frame_compile:22.92346 total_wall_time:22.92346 2025-12-04T09:02:14.8432163Z STATS: call_* op count: 634 | FakeTensorMode.__torch_dispatch__:13628 | FakeTensor.__torch_dispatch__:7175 | ProxyTorchDispatchMode.__torch_dispatch__:3966 2025-12-04T09:02:14.8432768Z Dynamo produced 1 graphs covering 634 ops with 0 graph breaks (0 unique) 2025-12-04T09:02:16.6657035Z accuracy pass_rate=92.59% 2025-12-04T09:02:16.6658905Z calls_captured gmean=0.00x mean=579.519x 2025-12-04T09:02:16.6659188Z unique_graphs gmean=0.00x mean=1.111x 2025-12-04T09:02:16.6659456Z graph_breaks gmean=0.00x mean=0.222x 2025-12-04T09:02:16.6669319Z unique_graph_breaks gmean=0.00x mean=0.074x 2025-12-04T09:02:16.6669720Z autograd_captures gmean=0.00x mean=0.000x 2025-12-04T09:02:16.6673096Z autograd_compiles gmean=0.00x mean=0.000x 2025-12-04T09:02:16.6673577Z cudagraph_skips gmean=0.00x mean=1.111x 2025-12-04T09:02:16.6673949Z compilation_latency mean=22.300 seconds 2025-12-04T09:02:17.6600916Z + python benchmarks/dynamo/check_accuracy.py --actual /var/lib/jenkins/workspace/test/test-reports/inference_huggingface.csv --expected benchmarks/dynamo/ci_expected_accuracy/dynamic_cpu_inductor_huggingface_inference.csv 2025-12-04T09:02:17.9542451Z AlbertForMaskedLM PASS 2025-12-04T09:02:17.9543285Z AllenaiLongformerBase PASS 2025-12-04T09:02:17.9552174Z BartForCausalLM PASS 2025-12-04T09:02:17.9552547Z BertForMaskedLM PASS 2025-12-04T09:02:17.9555617Z BlenderbotForCausalLM XFAIL 2025-12-04T09:02:17.9558937Z DebertaV2ForMaskedLM XFAIL 2025-12-04T09:02:17.9559937Z DistilBertForMaskedLM PASS 2025-12-04T09:02:17.9565401Z DistillGPT2 PASS 2025-12-04T09:02:17.9568770Z ElectraForCausalLM PASS 2025-12-04T09:02:17.9572258Z GPT2ForSequenceClassification PASS 2025-12-04T09:02:17.9577670Z GoogleFnet PASS 2025-12-04T09:02:17.9581503Z LayoutLMForMaskedLM PASS 2025-12-04T09:02:17.9588177Z M2M100ForConditionalGeneration PASS 2025-12-04T09:02:17.9595593Z MBartForCausalLM PASS 2025-12-04T09:02:17.9598129Z MT5ForConditionalGeneration PASS 2025-12-04T09:02:17.9598551Z MegatronBertForCausalLM PASS 2025-12-04T09:02:17.9598823Z MobileBertForMaskedLM PASS 2025-12-04T09:02:17.9602419Z OPTForCausalLM PASS 2025-12-04T09:02:17.9607875Z PLBartForCausalLM PASS 2025-12-04T09:02:17.9609544Z PegasusForCausalLM PASS 2025-12-04T09:02:17.9619710Z RobertaForCausalLM PASS 2025-12-04T09:02:17.9620005Z T5ForConditionalGeneration PASS 2025-12-04T09:02:17.9624414Z T5Small PASS 2025-12-04T09:02:17.9625094Z TrOCRForCausalLM PASS 2025-12-04T09:02:17.9633421Z XGLMForCausalLM PASS 2025-12-04T09:02:17.9633854Z XLNetLMHeadModel PASS 2025-12-04T09:02:17.9638071Z YituTechConvBert PASS 2025-12-04T09:02:18.0125809Z + python benchmarks/dynamo/check_graph_breaks.py --actual /var/lib/jenkins/workspace/test/test-reports/inference_huggingface.csv --expected benchmarks/dynamo/ci_expected_accuracy/dynamic_cpu_inductor_huggingface_inference.csv 2025-12-04T09:02:18.2834625Z AlbertForMaskedLM PASS 2025-12-04T09:02:18.2835271Z AllenaiLongformerBase PASS 2025-12-04T09:02:18.2835526Z BartForCausalLM PASS 2025-12-04T09:02:18.2842322Z BertForMaskedLM PASS 2025-12-04T09:02:18.2857543Z BlenderbotForCausalLM PASS 2025-12-04T09:02:18.2857829Z DebertaV2ForMaskedLM PASS 2025-12-04T09:02:18.2858065Z DistilBertForMaskedLM PASS 2025-12-04T09:02:18.2858312Z DistillGPT2 PASS 2025-12-04T09:02:18.2867258Z ElectraForCausalLM PASS 2025-12-04T09:02:18.2867679Z GPT2ForSequenceClassification PASS 2025-12-04T09:02:18.2870983Z GoogleFnet PASS 2025-12-04T09:02:18.2871271Z LayoutLMForMaskedLM PASS 2025-12-04T09:02:18.2876003Z M2M100ForConditionalGeneration PASS 2025-12-04T09:02:18.2876292Z MBartForCausalLM PASS 2025-12-04T09:02:18.2878527Z MT5ForConditionalGeneration PASS 2025-12-04T09:02:18.2888670Z MegatronBertForCausalLM PASS 2025-12-04T09:02:18.2888976Z MobileBertForMaskedLM PASS 2025-12-04T09:02:18.2891729Z OPTForCausalLM PASS 2025-12-04T09:02:18.2897495Z PLBartForCausalLM PASS 2025-12-04T09:02:18.2897992Z PegasusForCausalLM PASS 2025-12-04T09:02:18.2904877Z RobertaForCausalLM PASS 2025-12-04T09:02:18.2905334Z T5ForConditionalGeneration PASS 2025-12-04T09:02:18.2910117Z T5Small PASS 2025-12-04T09:02:18.2910381Z TrOCRForCausalLM PASS 2025-12-04T09:02:18.2917691Z XGLMForCausalLM PASS_BUT_FLAKY 2025-12-04T09:02:18.2917991Z XLNetLMHeadModel PASS 2025-12-04T09:02:18.2925009Z YituTechConvBert PASS 2025-12-04T09:02:18.3449620Z + sccache_epilogue 2025-12-04T09:02:18.3452073Z + echo '::group::Sccache Compilation Log' 2025-12-04T09:02:18.3453632Z ##[group]Sccache Compilation Log 2025-12-04T09:02:18.3454019Z + echo '=================== sccache compilation log ===================' 2025-12-04T09:02:18.3454317Z =================== sccache compilation log =================== 2025-12-04T09:02:18.3454755Z + python /var/lib/jenkins/workspace/.ci/pytorch/print_sccache_log.py /var/lib/jenkins/sccache_error.log 2025-12-04T09:02:18.3685661Z + echo '=========== If your build fails, please take a look at the log above for possible reasons ===========' 2025-12-04T09:02:18.3686186Z =========== If your build fails, please take a look at the log above for possible reasons =========== 2025-12-04T09:02:18.3686508Z + sccache --show-stats 2025-12-04T09:02:18.3716060Z Compile requests 276 2025-12-04T09:02:18.3716368Z Compile requests executed 0 2025-12-04T09:02:18.3716600Z Cache hits 0 2025-12-04T09:02:18.3716816Z Cache misses 0 2025-12-04T09:02:18.3717044Z Cache hits rate - 2025-12-04T09:02:18.3717290Z Cache timeouts 0 2025-12-04T09:02:18.3717513Z Cache read errors 0 2025-12-04T09:02:18.3717732Z Forced recaches 0 2025-12-04T09:02:18.3718172Z Cache write errors 0 2025-12-04T09:02:18.3718461Z Cache errors 0 2025-12-04T09:02:18.3718691Z Compilations 0 2025-12-04T09:02:18.3718923Z Compilation failures 0 2025-12-04T09:02:18.3719177Z Non-cacheable compilations 0 2025-12-04T09:02:18.3719477Z Non-cacheable calls 25 2025-12-04T09:02:18.3719705Z Non-compilation calls 251 2025-12-04T09:02:18.3719952Z Unsupported compiler calls 0 2025-12-04T09:02:18.3720297Z Average cache write 0.000 s 2025-12-04T09:02:18.3720534Z Average compiler 0.000 s 2025-12-04T09:02:18.3720900Z Average cache read hit 0.000 s 2025-12-04T09:02:18.3721124Z Failed distributed compilations 0 2025-12-04T09:02:18.3721271Z 2025-12-04T09:02:18.3721348Z Non-cacheable reasons: 2025-12-04T09:02:18.3721594Z -E 25 2025-12-04T09:02:18.3721738Z 2025-12-04T09:02:18.3721929Z Cache location s3, name: ossci-compiler-cache-circleci-v2, prefix: / 2025-12-04T09:02:18.3722249Z Version (client) 0.10.0 2025-12-04T09:02:18.3722472Z + sccache --stop-server 2025-12-04T09:02:18.3738474Z Stopping sccache server... 2025-12-04T09:02:18.3741856Z Compile requests 276 2025-12-04T09:02:18.3742127Z Compile requests executed 0 2025-12-04T09:02:18.3742371Z Cache hits 0 2025-12-04T09:02:18.3742593Z Cache misses 0 2025-12-04T09:02:18.3742824Z Cache hits rate - 2025-12-04T09:02:18.3743049Z Cache timeouts 0 2025-12-04T09:02:18.3743272Z Cache read errors 0 2025-12-04T09:02:18.3743647Z Forced recaches 0 2025-12-04T09:02:18.3743885Z Cache write errors 0 2025-12-04T09:02:18.3744105Z Cache errors 0 2025-12-04T09:02:18.3744324Z Compilations 0 2025-12-04T09:02:18.3744552Z Compilation failures 0 2025-12-04T09:02:18.3744784Z Non-cacheable compilations 0 2025-12-04T09:02:18.3745005Z Non-cacheable calls 25 2025-12-04T09:02:18.3745235Z Non-compilation calls 251 2025-12-04T09:02:18.3745464Z Unsupported compiler calls 0 2025-12-04T09:02:18.3745701Z Average cache write 0.000 s 2025-12-04T09:02:18.3745932Z Average compiler 0.000 s 2025-12-04T09:02:18.3746162Z Average cache read hit 0.000 s 2025-12-04T09:02:18.3746394Z Failed distributed compilations 0 2025-12-04T09:02:18.3746541Z 2025-12-04T09:02:18.3746620Z Non-cacheable reasons: 2025-12-04T09:02:18.3746820Z -E 25 2025-12-04T09:02:18.3746960Z 2025-12-04T09:02:18.3747149Z Cache location s3, name: ossci-compiler-cache-circleci-v2, prefix: / 2025-12-04T09:02:18.3747467Z Version (client) 0.10.0 2025-12-04T09:02:18.3747739Z + echo ::endgroup:: 2025-12-04T09:02:18.3748179Z ##[endgroup] 2025-12-04T09:02:18.3748357Z + cleanup_workspace 2025-12-04T09:02:18.3748706Z + echo 'sudo may print the following warning message that can be ignored. The chown command will still run.' 2025-12-04T09:02:18.3749209Z sudo may print the following warning message that can be ignored. The chown command will still run. 2025-12-04T09:02:18.3749636Z + echo ' sudo: setrlimit(RLIMIT_STACK): Operation not permitted' 2025-12-04T09:02:18.3749972Z sudo: setrlimit(RLIMIT_STACK): Operation not permitted 2025-12-04T09:02:18.3750349Z + echo 'For more details refer to https://github.com/sudo-project/sudo/issues/42' 2025-12-04T09:02:18.3750752Z For more details refer to https://github.com/sudo-project/sudo/issues/42 2025-12-04T09:02:18.3751083Z + sudo chown -R 1000 /var/lib/jenkins/workspace 2025-12-04T09:02:26.1712443Z ##[group]Run pytorch/test-infra/.github/actions/upload-benchmark-results@main 2025-12-04T09:02:26.1712762Z with: 2025-12-04T09:02:26.1712952Z benchmark-results-dir: test/test-reports 2025-12-04T09:02:26.1713163Z dry-run: false 2025-12-04T09:02:26.1713335Z schema-version: v3 2025-12-04T09:02:26.1713692Z github-token: *** 2025-12-04T09:02:26.1713858Z env: 2025-12-04T09:02:26.1714023Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:02:26.1714297Z HAS_NVIDIA_GPU: false 2025-12-04T09:02:26.1714624Z DOCKER_CONTAINER_ID: f7efe3f2eca996d8a9efcd5effd738cd22ce08409dea3a9ea34cb5639faf9c86 2025-12-04T09:02:26.1714961Z ##[endgroup] 2025-12-04T09:02:26.1742996Z ##[group]Run set -eux 2025-12-04T09:02:26.1743205Z set -eux 2025-12-04T09:02:26.1743369Z  2025-12-04T09:02:26.1743523Z if [[ -n "" ]]; then 2025-12-04T09:02:26.1743714Z  source "" 2025-12-04T09:02:26.1743884Z fi 2025-12-04T09:02:26.1744152Z python3 -mpip install boto3==1.35.33 psutil==7.0.0 pynvml==12.0.0 2025-12-04T09:02:26.1744419Z  2025-12-04T09:02:26.1744573Z DEVICE_NAME="" 2025-12-04T09:02:26.1744751Z DEVICE_TYPE="" 2025-12-04T09:02:26.1744919Z  2025-12-04T09:02:26.1745077Z if command -v nvidia-smi; then 2025-12-04T09:02:26.1745371Z  # NB: I'm using PyTorch here to get the device name, however, it needs to 2025-12-04T09:02:26.1745728Z  # install the correct version of PyTorch manually for now. Any PyTorch 2025-12-04T09:02:26.1746063Z  # version is fine, I just use 2.7.1 to satify PYPIDEP linter 2025-12-04T09:02:26.1746340Z  python3 -mpip install torch==2.7.1 2025-12-04T09:02:26.1746567Z elif command -v rocminfo; then 2025-12-04T09:02:26.1746850Z  # NB: Installing torch on ROCm runner with pip here causes CI to fail 2025-12-04T09:02:26.1747187Z  # with a memoryview is too large error only on MI300 runners. Is pip 2025-12-04T09:02:26.1747614Z  # version on ROCm runner there too old? As a workaround, let's use the 2025-12-04T09:02:26.1747926Z  # GPU device name coming from rocminfo instead 2025-12-04T09:02:26.1748154Z  DEVICE_NAME=rocm 2025-12-04T09:02:26.1748469Z  DEVICE_TYPE=$(rocminfo | grep "Marketing Name" | tail -n1 | awk -F':' '{print $2}' | xargs) 2025-12-04T09:02:26.1748782Z fi 2025-12-04T09:02:26.1748939Z  2025-12-04T09:02:26.1749126Z echo "DEVICE_NAME=$DEVICE_NAME" >> $GITHUB_ENV 2025-12-04T09:02:26.1749396Z echo "DEVICE_TYPE=$DEVICE_TYPE" >> $GITHUB_ENV 2025-12-04T09:02:26.1833089Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:02:26.1833342Z env: 2025-12-04T09:02:26.1833507Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:02:26.1833693Z HAS_NVIDIA_GPU: false 2025-12-04T09:02:26.1833987Z DOCKER_CONTAINER_ID: f7efe3f2eca996d8a9efcd5effd738cd22ce08409dea3a9ea34cb5639faf9c86 2025-12-04T09:02:26.1834319Z ##[endgroup] 2025-12-04T09:02:26.1870200Z + [[ -n '' ]] 2025-12-04T09:02:26.1875800Z + python3 -mpip install boto3==1.35.33 psutil==7.0.0 pynvml==12.0.0 2025-12-04T09:02:26.4897575Z Defaulting to user installation because normal site-packages is not writeable 2025-12-04T09:02:27.2576478Z Collecting boto3==1.35.33 2025-12-04T09:02:27.2927685Z Downloading boto3-1.35.33-py3-none-any.whl (139 kB) 2025-12-04T09:02:27.5885841Z Collecting psutil==7.0.0 2025-12-04T09:02:27.6076986Z Downloading psutil-7.0.0-cp36-abi3-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (277 kB) 2025-12-04T09:02:27.6898593Z Collecting pynvml==12.0.0 2025-12-04T09:02:27.7040358Z Downloading pynvml-12.0.0-py3-none-any.whl (26 kB) 2025-12-04T09:02:27.7453155Z Collecting s3transfer<0.11.0,>=0.10.0 2025-12-04T09:02:27.7604949Z Downloading s3transfer-0.10.4-py3-none-any.whl (83 kB) 2025-12-04T09:02:27.7747895Z Requirement already satisfied: jmespath<2.0.0,>=0.7.1 in /usr/lib/python3.9/site-packages (from boto3==1.35.33) (0.10.0) 2025-12-04T09:02:28.6277178Z Collecting botocore<1.36.0,>=1.35.33 2025-12-04T09:02:28.6438559Z Downloading botocore-1.35.99-py3-none-any.whl (13.3 MB) 2025-12-04T09:02:29.5371371Z Collecting nvidia-ml-py<13.0.0a0,>=12.0.0 2025-12-04T09:02:29.5678023Z Downloading nvidia_ml_py-12.575.51-py3-none-any.whl (47 kB) 2025-12-04T09:02:29.5783886Z Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in /usr/lib/python3.9/site-packages (from botocore<1.36.0,>=1.35.33->boto3==1.35.33) (2.8.1) 2025-12-04T09:02:29.5791494Z Requirement already satisfied: urllib3<1.27,>=1.25.4 in /usr/lib/python3.9/site-packages (from botocore<1.36.0,>=1.35.33->boto3==1.35.33) (1.25.10) 2025-12-04T09:02:29.7142768Z Requirement already satisfied: six>=1.5 in /usr/lib/python3.9/site-packages (from python-dateutil<3.0.0,>=2.1->botocore<1.36.0,>=1.35.33->boto3==1.35.33) (1.15.0) 2025-12-04T09:02:29.8268037Z Installing collected packages: botocore, s3transfer, nvidia-ml-py, pynvml, psutil, boto3 2025-12-04T09:02:30.2022780Z Attempting uninstall: nvidia-ml-py 2025-12-04T09:02:30.2028788Z Found existing installation: nvidia-ml-py 11.525.84 2025-12-04T09:02:30.2044690Z Uninstalling nvidia-ml-py-11.525.84: 2025-12-04T09:02:30.2234386Z Successfully uninstalled nvidia-ml-py-11.525.84 2025-12-04T09:02:30.2797789Z Attempting uninstall: psutil 2025-12-04T09:02:30.2798121Z Found existing installation: psutil 5.9.8 2025-12-04T09:02:30.2851843Z Uninstalling psutil-5.9.8: 2025-12-04T09:02:30.2867229Z Successfully uninstalled psutil-5.9.8 2025-12-04T09:02:30.4247637Z Successfully installed boto3-1.35.33 botocore-1.35.99 nvidia-ml-py-12.575.51 psutil-7.0.0 pynvml-12.0.0 s3transfer-0.10.4 2025-12-04T09:02:30.5353175Z + DEVICE_NAME= 2025-12-04T09:02:30.5353493Z + DEVICE_TYPE= 2025-12-04T09:02:30.5353696Z + command -v nvidia-smi 2025-12-04T09:02:30.5353881Z + command -v rocminfo 2025-12-04T09:02:30.5354052Z + echo DEVICE_NAME= 2025-12-04T09:02:30.5354509Z + echo DEVICE_TYPE= 2025-12-04T09:02:30.5393126Z ##[group]Run set -eux 2025-12-04T09:02:30.5393310Z set -eux 2025-12-04T09:02:30.5393461Z  2025-12-04T09:02:30.5393629Z if [[ -z "${GITHUB_TOKEN}" ]]; then 2025-12-04T09:02:30.5393844Z  echo "Missing github-token input" 2025-12-04T09:02:30.5394041Z  exit 1 2025-12-04T09:02:30.5394187Z fi 2025-12-04T09:02:30.5399148Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:02:30.5399413Z env: 2025-12-04T09:02:30.5399586Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:02:30.5399773Z HAS_NVIDIA_GPU: false 2025-12-04T09:02:30.5400099Z DOCKER_CONTAINER_ID: f7efe3f2eca996d8a9efcd5effd738cd22ce08409dea3a9ea34cb5639faf9c86 2025-12-04T09:02:30.5400461Z DEVICE_NAME: 2025-12-04T09:02:30.5400634Z DEVICE_TYPE: 2025-12-04T09:02:30.5401006Z GITHUB_TOKEN: *** 2025-12-04T09:02:30.5401187Z ##[endgroup] 2025-12-04T09:02:30.5426027Z + [[ -z *** ]] 2025-12-04T09:02:30.5484095Z ##[group]Run pytorch/test-infra/.github/actions/get-workflow-job-id@main 2025-12-04T09:02:30.5484361Z with: 2025-12-04T09:02:30.5484649Z github-token: *** 2025-12-04T09:02:30.5484793Z env: 2025-12-04T09:02:30.5484939Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:02:30.5485118Z HAS_NVIDIA_GPU: false 2025-12-04T09:02:30.5485431Z DOCKER_CONTAINER_ID: f7efe3f2eca996d8a9efcd5effd738cd22ce08409dea3a9ea34cb5639faf9c86 2025-12-04T09:02:30.5485768Z DEVICE_NAME: 2025-12-04T09:02:30.5485937Z DEVICE_TYPE: 2025-12-04T09:02:30.5486100Z ##[endgroup] 2025-12-04T09:02:30.5508242Z ##[group]Run set -eux 2025-12-04T09:02:30.5508443Z set -eux 2025-12-04T09:02:30.5508608Z  2025-12-04T09:02:30.5508937Z python3 "${GITHUB_ACTION_PATH}/../../scripts/get_workflow_job_id.py" "${GITHUB_RUN_ID}" "${RUNNER_NAME}" 2025-12-04T09:02:30.5513251Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:02:30.5513504Z env: 2025-12-04T09:02:30.5513671Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:02:30.5513871Z HAS_NVIDIA_GPU: false 2025-12-04T09:02:30.5514285Z DOCKER_CONTAINER_ID: f7efe3f2eca996d8a9efcd5effd738cd22ce08409dea3a9ea34cb5639faf9c86 2025-12-04T09:02:30.5514626Z DEVICE_NAME: 2025-12-04T09:02:30.5514793Z DEVICE_TYPE: 2025-12-04T09:02:30.5515262Z GITHUB_TOKEN: *** 2025-12-04T09:02:30.5515446Z ##[endgroup] 2025-12-04T09:02:30.5538414Z + python3 /home/ec2-user/actions-runner/_work/_actions/pytorch/test-infra/main/.github/actions/get-workflow-job-id/../../scripts/get_workflow_job_id.py 19921726389 i-02dcdd5317aed8049 2025-12-04T09:02:32.1062749Z setting job-id=57113856426 2025-12-04T09:02:32.1063300Z setting job-name=inductor-cpu-test / test (dynamic_cpu_inductor_huggingface, 1, 1, linux.2xlarge.amx) 2025-12-04T09:02:32.1181833Z ##[group]Run set -eux 2025-12-04T09:02:32.1182039Z set -eux 2025-12-04T09:02:32.1182191Z  2025-12-04T09:02:32.1182351Z if [[ -n "" ]]; then 2025-12-04T09:02:32.1182538Z  source "" 2025-12-04T09:02:32.1182702Z fi 2025-12-04T09:02:32.1182850Z  2025-12-04T09:02:32.1183116Z python3 "${GITHUB_ACTION_PATH}/../../scripts/benchmarks/gather_metadata.py" \ 2025-12-04T09:02:32.1183437Z  --schema-version "${SCHEMA_VERSION}" \ 2025-12-04T09:02:32.1183660Z  --repo "${REPO}" \ 2025-12-04T09:02:32.1183862Z  --head-branch "${HEAD_BRANCH}" \ 2025-12-04T09:02:32.1184077Z  --head-sha "${HEAD_SHA}" \ 2025-12-04T09:02:32.1184288Z  --workflow-id "${WORKFLOW_RUN_ID}" \ 2025-12-04T09:02:32.1184525Z  --run-attempt "${RUN_ATTEMPT}" \ 2025-12-04T09:02:32.1184735Z  --job-id "${JOB_ID}" \ 2025-12-04T09:02:32.1184929Z  --job-name "${JOB_NAME}" 2025-12-04T09:02:32.1189376Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:02:32.1189627Z env: 2025-12-04T09:02:32.1189793Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:02:32.1189976Z HAS_NVIDIA_GPU: false 2025-12-04T09:02:32.1190284Z DOCKER_CONTAINER_ID: f7efe3f2eca996d8a9efcd5effd738cd22ce08409dea3a9ea34cb5639faf9c86 2025-12-04T09:02:32.1190745Z DEVICE_NAME: 2025-12-04T09:02:32.1190911Z DEVICE_TYPE: 2025-12-04T09:02:32.1191075Z SCHEMA_VERSION: v3 2025-12-04T09:02:32.1191256Z REPO: pytorch/pytorch 2025-12-04T09:02:32.1191434Z HEAD_BRANCH: refs/heads/main 2025-12-04T09:02:32.1191660Z HEAD_SHA: ffd9b0fb4355e97af82fc42cf185c3ffa0fc0a32 2025-12-04T09:02:32.1191897Z WORKFLOW_RUN_ID: 19921726389 2025-12-04T09:02:32.1192071Z RUN_ATTEMPT: 1 2025-12-04T09:02:32.1192233Z JOB_ID: 57113856426 2025-12-04T09:02:32.1192540Z JOB_NAME: inductor-cpu-test / test (dynamic_cpu_inductor_huggingface, 1, 1, linux.2xlarge.amx) 2025-12-04T09:02:32.1192880Z ##[endgroup] 2025-12-04T09:02:32.1218331Z + [[ -n '' ]] 2025-12-04T09:02:32.1219519Z + python3 /home/ec2-user/actions-runner/_work/_actions/pytorch/test-infra/main/.github/actions/upload-benchmark-results/../../scripts/benchmarks/gather_metadata.py --schema-version v3 --repo pytorch/pytorch --head-branch refs/heads/main --head-sha ffd9b0fb4355e97af82fc42cf185c3ffa0fc0a32 --workflow-id 19921726389 --run-attempt 1 --job-id 57113856426 --job-name 'inductor-cpu-test / test (dynamic_cpu_inductor_huggingface, 1, 1, linux.2xlarge.amx)' 2025-12-04T09:02:32.1493358Z ##[group]Run set -eux 2025-12-04T09:02:32.1493564Z set -eux 2025-12-04T09:02:32.1493730Z  2025-12-04T09:02:32.1493884Z if [[ -n "" ]]; then 2025-12-04T09:02:32.1494073Z  source "" 2025-12-04T09:02:32.1494239Z fi 2025-12-04T09:02:32.1494396Z  2025-12-04T09:02:32.1494658Z python3 "${GITHUB_ACTION_PATH}/../../scripts/benchmarks/gather_runners_info.py" 2025-12-04T09:02:32.1499329Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:02:32.1499585Z env: 2025-12-04T09:02:32.1499746Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:02:32.1499933Z HAS_NVIDIA_GPU: false 2025-12-04T09:02:32.1500247Z DOCKER_CONTAINER_ID: f7efe3f2eca996d8a9efcd5effd738cd22ce08409dea3a9ea34cb5639faf9c86 2025-12-04T09:02:32.1500557Z DEVICE_NAME: 2025-12-04T09:02:32.1500851Z DEVICE_TYPE: 2025-12-04T09:02:32.1501016Z ##[endgroup] 2025-12-04T09:02:32.1520143Z + [[ -n '' ]] 2025-12-04T09:02:32.1520975Z + python3 /home/ec2-user/actions-runner/_work/_actions/pytorch/test-infra/main/.github/actions/upload-benchmark-results/../../scripts/benchmarks/gather_runners_info.py 2025-12-04T09:02:32.1854855Z INFO:root:Fail to import torch to get the device name 2025-12-04T09:02:32.1963206Z ##[group]Run set -eux 2025-12-04T09:02:32.1963500Z set -eux 2025-12-04T09:02:32.1963656Z  2025-12-04T09:02:32.1963831Z # TODO (huydhn): Implement this part 2025-12-04T09:02:32.1964068Z echo "dependencies={}" >> "${GITHUB_OUTPUT}" 2025-12-04T09:02:32.1968231Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:02:32.1968466Z env: 2025-12-04T09:02:32.1968612Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:02:32.1968792Z HAS_NVIDIA_GPU: false 2025-12-04T09:02:32.1969106Z DOCKER_CONTAINER_ID: f7efe3f2eca996d8a9efcd5effd738cd22ce08409dea3a9ea34cb5639faf9c86 2025-12-04T09:02:32.1969415Z DEVICE_NAME: 2025-12-04T09:02:32.1969563Z DEVICE_TYPE: 2025-12-04T09:02:32.1969715Z ##[endgroup] 2025-12-04T09:02:32.1991720Z + echo 'dependencies={}' 2025-12-04T09:02:32.2023552Z ##[group]Run set -eux 2025-12-04T09:02:32.2023749Z set -eux 2025-12-04T09:02:32.2023909Z  2025-12-04T09:02:32.2024065Z if [[ -n "" ]]; then 2025-12-04T09:02:32.2024246Z  source "" 2025-12-04T09:02:32.2024416Z fi 2025-12-04T09:02:32.2024560Z  2025-12-04T09:02:32.2024735Z if [[ ! -d "${BENCHMARK_RESULTS_DIR}" ]]; then 2025-12-04T09:02:32.2025018Z  echo "${BENCHMARK_RESULTS_DIR} does not exist, skipping" 2025-12-04T09:02:32.2025322Z  # We don't want the job to fail if the directory doesn't exist 2025-12-04T09:02:32.2025559Z  exit 0 2025-12-04T09:02:32.2025713Z fi 2025-12-04T09:02:32.2025917Z  2025-12-04T09:02:32.2026094Z if [[ "${DRY_RUN}" == "true" ]]; then 2025-12-04T09:02:32.2026399Z  python3 "${GITHUB_ACTION_PATH}/../../scripts/upload_benchmark_results.py" \ 2025-12-04T09:02:32.2026752Z  --benchmark-results-dir "${BENCHMARK_RESULTS_DIR}" \ 2025-12-04T09:02:32.2027030Z  --metadata "${BENCHMARK_METADATA}" \ 2025-12-04T09:02:32.2027254Z  --runners "${RUNNER_INFO}" \ 2025-12-04T09:02:32.2027487Z  --dependencies "${DEPENDENCIES}" \ 2025-12-04T09:02:32.2027717Z  --dry-run 2025-12-04T09:02:32.2027887Z else 2025-12-04T09:02:32.2028125Z  python3 "${GITHUB_ACTION_PATH}/../../scripts/upload_benchmark_results.py" \ 2025-12-04T09:02:32.2028458Z  --benchmark-results-dir "${BENCHMARK_RESULTS_DIR}" \ 2025-12-04T09:02:32.2028720Z  --metadata "${BENCHMARK_METADATA}" \ 2025-12-04T09:02:32.2028938Z  --runners "${RUNNER_INFO}" \ 2025-12-04T09:02:32.2029162Z  --dependencies "${DEPENDENCIES}" 2025-12-04T09:02:32.2029370Z fi 2025-12-04T09:02:32.2033208Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:02:32.2033443Z env: 2025-12-04T09:02:32.2033600Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:02:32.2033787Z HAS_NVIDIA_GPU: false 2025-12-04T09:02:32.2034084Z DOCKER_CONTAINER_ID: f7efe3f2eca996d8a9efcd5effd738cd22ce08409dea3a9ea34cb5639faf9c86 2025-12-04T09:02:32.2034404Z DEVICE_NAME: 2025-12-04T09:02:32.2034561Z DEVICE_TYPE: 2025-12-04T09:02:32.2034734Z BENCHMARK_RESULTS_DIR: test/test-reports 2025-12-04T09:02:32.2034943Z DRY_RUN: false 2025-12-04T09:02:32.2035728Z BENCHMARK_METADATA: {"timestamp": 1764838952, "schema_version": "v3", "name": "inductor-cpu-test / test (dynamic_cpu_inductor_huggingface, 1, 1, linux.2xlarge.amx)", "repo": "pytorch/pytorch", "head_branch": "refs/heads/main", "head_sha": "ffd9b0fb4355e97af82fc42cf185c3ffa0fc0a32", "workflow_id": 19921726389, "run_attempt": 1, "job_id": 57113856426} 2025-12-04T09:02:32.2036820Z RUNNER_INFO: [{"cpu_info": "x86_64", "cpu_count": 8, "avail_mem_in_gb": 30, "extra_info": {"hostname": "ip-10-0-12-49.ec2.internal"}, "name": "", "type": ""}] 2025-12-04T09:02:32.2037215Z DEPENDENCIES: {} 2025-12-04T09:02:32.2037377Z ##[endgroup] 2025-12-04T09:02:32.2058632Z + [[ -n '' ]] 2025-12-04T09:02:32.2058865Z + [[ ! -d test/test-reports ]] 2025-12-04T09:02:32.2059074Z + [[ false == \t\r\u\e ]] 2025-12-04T09:02:32.2060776Z + python3 /home/ec2-user/actions-runner/_work/_actions/pytorch/test-infra/main/.github/actions/upload-benchmark-results/../../scripts/upload_benchmark_results.py --benchmark-results-dir test/test-reports --metadata '{"timestamp": 1764838952, "schema_version": "v3", "name": "inductor-cpu-test / test (dynamic_cpu_inductor_huggingface, 1, 1, linux.2xlarge.amx)", "repo": "pytorch/pytorch", "head_branch": "refs/heads/main", "head_sha": "ffd9b0fb4355e97af82fc42cf185c3ffa0fc0a32", "workflow_id": 19921726389, "run_attempt": 1, "job_id": 57113856426}' --runners '[{"cpu_info": "x86_64", "cpu_count": 8, "avail_mem_in_gb": 30, "extra_info": {"hostname": "ip-10-0-12-49.ec2.internal"}, "name": "", "type": ""}]' --dependencies '{}' 2025-12-04T09:02:32.4050766Z INFO:root:Upload test/test-reports/inference_huggingface.json to s3://ossci-benchmarks/v3/pytorch/pytorch/19921726389/57113856426/inference_huggingface.json 2025-12-04T09:02:32.4329409Z INFO:botocore.credentials:Found credentials from IAM Role: gh-ci-github-action-runners-runner-role 2025-12-04T09:02:32.6588289Z ##[group]Run cat test/**/*_toprint.log || true 2025-12-04T09:02:32.6588585Z cat test/**/*_toprint.log || true 2025-12-04T09:02:32.6593518Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:02:32.6593767Z env: 2025-12-04T09:02:32.6593927Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:02:32.6594107Z HAS_NVIDIA_GPU: false 2025-12-04T09:02:32.6594418Z DOCKER_CONTAINER_ID: f7efe3f2eca996d8a9efcd5effd738cd22ce08409dea3a9ea34cb5639faf9c86 2025-12-04T09:02:32.6594824Z DEVICE_NAME: 2025-12-04T09:02:32.6594987Z DEVICE_TYPE: 2025-12-04T09:02:32.6595136Z ##[endgroup] 2025-12-04T09:02:32.6673395Z cat: 'test/**/*_toprint.log': No such file or directory 2025-12-04T09:02:32.6711110Z ##[group]Run kill "$MONITOR_SCRIPT_PID" 2025-12-04T09:02:32.6711366Z kill "$MONITOR_SCRIPT_PID" 2025-12-04T09:02:32.6715734Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:02:32.6715975Z env: 2025-12-04T09:02:32.6716140Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:02:32.6716341Z HAS_NVIDIA_GPU: false 2025-12-04T09:02:32.6716640Z DOCKER_CONTAINER_ID: f7efe3f2eca996d8a9efcd5effd738cd22ce08409dea3a9ea34cb5639faf9c86 2025-12-04T09:02:32.6716973Z DEVICE_NAME: 2025-12-04T09:02:32.6717143Z DEVICE_TYPE: 2025-12-04T09:02:32.6717309Z MONITOR_SCRIPT_PID: 44960 2025-12-04T09:02:32.6717487Z ##[endgroup] 2025-12-04T09:02:32.6737335Z /home/ec2-user/actions-runner/_work/_temp/a680017c-62ca-40c7-9f9d-fac1dff4d236.sh: line 1: kill: (44960) - No such process 2025-12-04T09:02:32.6753739Z ##[error]Process completed with exit code 1. 2025-12-04T09:02:32.6850366Z Prepare all required actions 2025-12-04T09:02:32.6850741Z Getting action download info 2025-12-04T09:02:32.8562352Z Download action repository 'seemethere/upload-artifact-s3@v5' (SHA:baba72d0712b404f646cebe0730933554ebce96a) 2025-12-04T09:02:33.1348061Z Download action repository 'actions/upload-artifact@v4' (SHA:ea165f8d65b6e75b540449e92b4886f43607fa02) 2025-12-04T09:02:33.6727728Z ##[group]Run ./.github/actions/upload-test-artifacts 2025-12-04T09:02:33.6728004Z with: 2025-12-04T09:02:33.6728322Z file-suffix: test-dynamic_cpu_inductor_huggingface-1-1-linux.2xlarge.amx_57113856426 2025-12-04T09:02:33.6728685Z s3-bucket: gha-artifacts 2025-12-04T09:02:33.6728872Z env: 2025-12-04T09:02:33.6729034Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:02:33.6729234Z HAS_NVIDIA_GPU: false 2025-12-04T09:02:33.6729564Z DOCKER_CONTAINER_ID: f7efe3f2eca996d8a9efcd5effd738cd22ce08409dea3a9ea34cb5639faf9c86 2025-12-04T09:02:33.6729932Z DEVICE_NAME: 2025-12-04T09:02:33.6730148Z DEVICE_TYPE: 2025-12-04T09:02:33.6730316Z ##[endgroup] 2025-12-04T09:02:33.6756202Z ##[group]Run # Remove any previous test jsons if they exist 2025-12-04T09:02:33.6756507Z # Remove any previous test jsons if they exist 2025-12-04T09:02:33.6756754Z rm -f test-jsons-*.zip 2025-12-04T09:02:33.6757035Z zip -r "test-jsons-${FILE_SUFFIX}.zip" test/test-reports -i '*.json' 2025-12-04T09:02:33.6761953Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:02:33.6762310Z env: 2025-12-04T09:02:33.6762477Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:02:33.6762662Z HAS_NVIDIA_GPU: false 2025-12-04T09:02:33.6762971Z DOCKER_CONTAINER_ID: f7efe3f2eca996d8a9efcd5effd738cd22ce08409dea3a9ea34cb5639faf9c86 2025-12-04T09:02:33.6763297Z DEVICE_NAME: 2025-12-04T09:02:33.6763451Z DEVICE_TYPE: 2025-12-04T09:02:33.6763727Z FILE_SUFFIX: test-dynamic_cpu_inductor_huggingface-1-1-linux.2xlarge.amx_57113856426 2025-12-04T09:02:33.6764045Z ##[endgroup] 2025-12-04T09:02:33.6983213Z adding: test/test-reports/inference_huggingface.json (deflated 99%) 2025-12-04T09:02:33.7015711Z ##[group]Run # Remove any previous test reports if they exist 2025-12-04T09:02:33.7016032Z # Remove any previous test reports if they exist 2025-12-04T09:02:33.7016283Z rm -f test-reports-*.zip 2025-12-04T09:02:33.7016588Z zip -r "test-reports-${FILE_SUFFIX}.zip" test/test-reports -i '*.xml' -i '*.csv' 2025-12-04T09:02:33.7021438Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:02:33.7021685Z env: 2025-12-04T09:02:33.7021843Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:02:33.7022026Z HAS_NVIDIA_GPU: false 2025-12-04T09:02:33.7022327Z DOCKER_CONTAINER_ID: f7efe3f2eca996d8a9efcd5effd738cd22ce08409dea3a9ea34cb5639faf9c86 2025-12-04T09:02:33.7022653Z DEVICE_NAME: 2025-12-04T09:02:33.7022813Z DEVICE_TYPE: 2025-12-04T09:02:33.7023075Z FILE_SUFFIX: test-dynamic_cpu_inductor_huggingface-1-1-linux.2xlarge.amx_57113856426 2025-12-04T09:02:33.7023456Z ##[endgroup] 2025-12-04T09:02:33.7065259Z adding: test/test-reports/inference_huggingface.csv (deflated 62%) 2025-12-04T09:02:33.7068876Z adding: test/test-reports/inference_huggingface_graph_breaks.csv (deflated 85%) 2025-12-04T09:02:33.7074706Z adding: test/test-reports/inference_huggingface_graph_break_deduped.csv (deflated 65%) 2025-12-04T09:02:33.7099505Z ##[group]Run # Remove any previous usage logs if they exist 2025-12-04T09:02:33.7099816Z # Remove any previous usage logs if they exist 2025-12-04T09:02:33.7100073Z rm -f logs-*.zip 2025-12-04T09:02:33.7100304Z zip "logs-${FILE_SUFFIX}.zip" 'usage_log.txt' || true 2025-12-04T09:02:33.7100632Z zip -r "logs-${FILE_SUFFIX}.zip" test/test-reports -i '*.log' || true 2025-12-04T09:02:33.7104462Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:02:33.7104701Z env: 2025-12-04T09:02:33.7104852Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:02:33.7105094Z HAS_NVIDIA_GPU: false 2025-12-04T09:02:33.7105526Z DOCKER_CONTAINER_ID: f7efe3f2eca996d8a9efcd5effd738cd22ce08409dea3a9ea34cb5639faf9c86 2025-12-04T09:02:33.7105840Z DEVICE_NAME: 2025-12-04T09:02:33.7106000Z DEVICE_TYPE: 2025-12-04T09:02:33.7106274Z FILE_SUFFIX: test-dynamic_cpu_inductor_huggingface-1-1-linux.2xlarge.amx_57113856426 2025-12-04T09:02:33.7106574Z ##[endgroup] 2025-12-04T09:02:33.7155924Z adding: usage_log.txt (deflated 58%) 2025-12-04T09:02:33.7166586Z 2025-12-04T09:02:33.7167210Z zip error: Nothing to do! (logs-test-dynamic_cpu_inductor_huggingface-1-1-linux.2xlarge.amx_57113856426.zip) 2025-12-04T09:02:33.7196330Z ##[group]Run # Remove any previous debugging artifacts if they exist 2025-12-04T09:02:33.7196706Z # Remove any previous debugging artifacts if they exist 2025-12-04T09:02:33.7196975Z rm -f debug-*.zip 2025-12-04T09:02:33.7197171Z if [ -d 'test/debug' ]; then 2025-12-04T09:02:33.7197416Z  zip -r "debug-${FILE_SUFFIX}.zip" test/debug 2025-12-04T09:02:33.7197655Z fi 2025-12-04T09:02:33.7201892Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:02:33.7202170Z env: 2025-12-04T09:02:33.7202349Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:02:33.7202556Z HAS_NVIDIA_GPU: false 2025-12-04T09:02:33.7202895Z DOCKER_CONTAINER_ID: f7efe3f2eca996d8a9efcd5effd738cd22ce08409dea3a9ea34cb5639faf9c86 2025-12-04T09:02:33.7203262Z DEVICE_NAME: 2025-12-04T09:02:33.7203442Z DEVICE_TYPE: 2025-12-04T09:02:33.7203826Z FILE_SUFFIX: test-dynamic_cpu_inductor_huggingface-1-1-linux.2xlarge.amx_57113856426 2025-12-04T09:02:33.7204169Z ##[endgroup] 2025-12-04T09:02:33.7273621Z ##[group]Run seemethere/upload-artifact-s3@v5 2025-12-04T09:02:33.7273835Z with: 2025-12-04T09:02:33.7273996Z s3-bucket: gha-artifacts 2025-12-04T09:02:33.7274216Z s3-prefix: pytorch/pytorch/19921726389/1/artifact 2025-12-04T09:02:33.7274443Z retention-days: 14 2025-12-04T09:02:33.7274704Z if-no-files-found: warn 2025-12-04T09:02:33.7274916Z path: test-jsons-*.zip 2025-12-04T09:02:33.7275108Z name: artifact 2025-12-04T09:02:33.7275286Z region: us-east-1 2025-12-04T09:02:33.7275456Z env: 2025-12-04T09:02:33.7275617Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:02:33.7275818Z HAS_NVIDIA_GPU: false 2025-12-04T09:02:33.7276151Z DOCKER_CONTAINER_ID: f7efe3f2eca996d8a9efcd5effd738cd22ce08409dea3a9ea34cb5639faf9c86 2025-12-04T09:02:33.7276504Z DEVICE_NAME: 2025-12-04T09:02:33.7276669Z DEVICE_TYPE: 2025-12-04T09:02:33.7276835Z ##[endgroup] 2025-12-04T09:02:33.9956757Z NOTE: s3-prefix specified, ignoring name parameter 2025-12-04T09:02:33.9959810Z With the provided path, there will be 1 file uploaded 2025-12-04T09:02:33.9963906Z Uploading to s3 prefix: pytorch/pytorch/19921726389/1/artifact 2025-12-04T09:02:34.0057177Z Starting upload of test-jsons-test-dynamic_cpu_inductor_huggingface-1-1-linux.2xlarge.amx_57113856426.zip 2025-12-04T09:02:34.1044051Z Finished upload of test-jsons-test-dynamic_cpu_inductor_huggingface-1-1-linux.2xlarge.amx_57113856426.zip 2025-12-04T09:02:34.1232675Z ##[group]Run seemethere/upload-artifact-s3@v5 2025-12-04T09:02:34.1232912Z with: 2025-12-04T09:02:34.1233074Z s3-bucket: gha-artifacts 2025-12-04T09:02:34.1233301Z s3-prefix: pytorch/pytorch/19921726389/1/artifact 2025-12-04T09:02:34.1233534Z retention-days: 14 2025-12-04T09:02:34.1233707Z if-no-files-found: error 2025-12-04T09:02:34.1233890Z path: test-reports-*.zip 2025-12-04T09:02:34.1234064Z name: artifact 2025-12-04T09:02:34.1234226Z region: us-east-1 2025-12-04T09:02:34.1234384Z env: 2025-12-04T09:02:34.1234538Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:02:34.1234718Z HAS_NVIDIA_GPU: false 2025-12-04T09:02:34.1235019Z DOCKER_CONTAINER_ID: f7efe3f2eca996d8a9efcd5effd738cd22ce08409dea3a9ea34cb5639faf9c86 2025-12-04T09:02:34.1235342Z DEVICE_NAME: 2025-12-04T09:02:34.1235502Z DEVICE_TYPE: 2025-12-04T09:02:34.1235654Z ##[endgroup] 2025-12-04T09:02:34.4127663Z NOTE: s3-prefix specified, ignoring name parameter 2025-12-04T09:02:34.4128470Z With the provided path, there will be 1 file uploaded 2025-12-04T09:02:34.4128807Z Uploading to s3 prefix: pytorch/pytorch/19921726389/1/artifact 2025-12-04T09:02:34.4158565Z Starting upload of test-reports-test-dynamic_cpu_inductor_huggingface-1-1-linux.2xlarge.amx_57113856426.zip 2025-12-04T09:02:34.5217481Z Finished upload of test-reports-test-dynamic_cpu_inductor_huggingface-1-1-linux.2xlarge.amx_57113856426.zip 2025-12-04T09:02:34.5388209Z ##[group]Run seemethere/upload-artifact-s3@v5 2025-12-04T09:02:34.5388457Z with: 2025-12-04T09:02:34.5388625Z s3-bucket: gha-artifacts 2025-12-04T09:02:34.5388852Z s3-prefix: pytorch/pytorch/19921726389/1/artifact 2025-12-04T09:02:34.5389080Z retention-days: 14 2025-12-04T09:02:34.5389261Z if-no-files-found: ignore 2025-12-04T09:02:34.5389451Z path: logs-*.zip 2025-12-04T09:02:34.5389608Z name: artifact 2025-12-04T09:02:34.5389770Z region: us-east-1 2025-12-04T09:02:34.5389930Z env: 2025-12-04T09:02:34.5390079Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:02:34.5390270Z HAS_NVIDIA_GPU: false 2025-12-04T09:02:34.5390591Z DOCKER_CONTAINER_ID: f7efe3f2eca996d8a9efcd5effd738cd22ce08409dea3a9ea34cb5639faf9c86 2025-12-04T09:02:34.5390915Z DEVICE_NAME: 2025-12-04T09:02:34.5391070Z DEVICE_TYPE: 2025-12-04T09:02:34.5391230Z ##[endgroup] 2025-12-04T09:02:34.7955414Z NOTE: s3-prefix specified, ignoring name parameter 2025-12-04T09:02:34.7955878Z With the provided path, there will be 1 file uploaded 2025-12-04T09:02:34.7956215Z Uploading to s3 prefix: pytorch/pytorch/19921726389/1/artifact 2025-12-04T09:02:34.7988034Z Starting upload of logs-test-dynamic_cpu_inductor_huggingface-1-1-linux.2xlarge.amx_57113856426.zip 2025-12-04T09:02:34.9029188Z Finished upload of logs-test-dynamic_cpu_inductor_huggingface-1-1-linux.2xlarge.amx_57113856426.zip 2025-12-04T09:02:34.9203100Z ##[group]Run seemethere/upload-artifact-s3@v5 2025-12-04T09:02:34.9203460Z with: 2025-12-04T09:02:34.9203702Z s3-bucket: gha-artifacts 2025-12-04T09:02:34.9204060Z s3-prefix: pytorch/pytorch/19921726389/1/artifact 2025-12-04T09:02:34.9204428Z retention-days: 14 2025-12-04T09:02:34.9204699Z if-no-files-found: ignore 2025-12-04T09:02:34.9204991Z path: debug-*.zip 2025-12-04T09:02:34.9205231Z name: artifact 2025-12-04T09:02:34.9205471Z region: us-east-1 2025-12-04T09:02:34.9205707Z env: 2025-12-04T09:02:34.9205922Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:02:34.9206208Z HAS_NVIDIA_GPU: false 2025-12-04T09:02:34.9206717Z DOCKER_CONTAINER_ID: f7efe3f2eca996d8a9efcd5effd738cd22ce08409dea3a9ea34cb5639faf9c86 2025-12-04T09:02:34.9207251Z DEVICE_NAME: 2025-12-04T09:02:34.9207485Z DEVICE_TYPE: 2025-12-04T09:02:34.9207726Z ##[endgroup] 2025-12-04T09:02:35.1665495Z No files were found with the provided path: debug-*.zip. No artifacts will be uploaded. 2025-12-04T09:02:35.1833206Z ##[group]Run # shellcheck disable=SC2156 2025-12-04T09:02:35.1833490Z # shellcheck disable=SC2156 2025-12-04T09:02:35.1833902Z find . -iname "core.[1-9]*" -exec docker exec "${DOCKER_CONTAINER_ID}" sh -c "gdb python {} -ex 'bt' -ex 'q'" \; 2025-12-04T09:02:35.1839219Z shell: /usr/bin/bash -e {0} 2025-12-04T09:02:35.1839428Z env: 2025-12-04T09:02:35.1839620Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:02:35.1839819Z HAS_NVIDIA_GPU: false 2025-12-04T09:02:35.1840267Z DOCKER_CONTAINER_ID: f7efe3f2eca996d8a9efcd5effd738cd22ce08409dea3a9ea34cb5639faf9c86 2025-12-04T09:02:35.1840618Z DEVICE_NAME: 2025-12-04T09:02:35.1840794Z DEVICE_TYPE: 2025-12-04T09:02:35.1840961Z ##[endgroup] 2025-12-04T09:02:35.3801755Z Prepare all required actions 2025-12-04T09:02:35.3802146Z Getting action download info 2025-12-04T09:02:35.5727764Z Download action repository 'actions/setup-python@v6' (SHA:83679a892e2d95755f2dac6acb0bfd1e9ac5d548) 2025-12-04T09:02:36.1062513Z ##[group]Run ./.github/actions/upload-utilization-stats 2025-12-04T09:02:36.1062758Z with: 2025-12-04T09:02:36.1062919Z job_id: 57113856426 2025-12-04T09:02:36.1063217Z job_name: inductor-cpu-test / test (dynamic_cpu_inductor_huggingface, 1, 1, linux.2xlarge.amx) 2025-12-04T09:02:36.1063554Z workflow_name: inductor 2025-12-04T09:02:36.1063732Z workflow_run_id: 19921726389 2025-12-04T09:02:36.1063916Z workflow_attempt: 1 2025-12-04T09:02:36.1064076Z env: 2025-12-04T09:02:36.1064219Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:02:36.1064397Z HAS_NVIDIA_GPU: false 2025-12-04T09:02:36.1064696Z DOCKER_CONTAINER_ID: f7efe3f2eca996d8a9efcd5effd738cd22ce08409dea3a9ea34cb5639faf9c86 2025-12-04T09:02:36.1065046Z DEVICE_NAME: 2025-12-04T09:02:36.1065249Z DEVICE_TYPE: 2025-12-04T09:02:36.1065407Z ##[endgroup] 2025-12-04T09:02:36.1105028Z ##[group]Run actions/setup-python@v6 2025-12-04T09:02:36.1105239Z with: 2025-12-04T09:02:36.1105396Z python-version: 3.10 2025-12-04T09:02:36.1105561Z check-latest: false 2025-12-04T09:02:36.1105810Z token: *** 2025-12-04T09:02:36.1105972Z update-environment: true 2025-12-04T09:02:36.1106156Z allow-prereleases: false 2025-12-04T09:02:36.1106323Z freethreaded: false 2025-12-04T09:02:36.1106489Z env: 2025-12-04T09:02:36.1106633Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:02:36.1106798Z HAS_NVIDIA_GPU: false 2025-12-04T09:02:36.1107090Z DOCKER_CONTAINER_ID: f7efe3f2eca996d8a9efcd5effd738cd22ce08409dea3a9ea34cb5639faf9c86 2025-12-04T09:02:36.1107402Z DEVICE_NAME: 2025-12-04T09:02:36.1107551Z DEVICE_TYPE: 2025-12-04T09:02:36.1107703Z ##[endgroup] 2025-12-04T09:02:36.6044877Z ##[group]Installed versions 2025-12-04T09:02:36.6062911Z Version 3.10 was not found in the local cache 2025-12-04T09:02:36.6328431Z (node:52493) [DEP0040] DeprecationWarning: The `punycode` module is deprecated. Please use a userland alternative instead. 2025-12-04T09:02:36.6329286Z (Use `node --trace-deprecation ...` to show where the warning was created) 2025-12-04T09:02:37.0200218Z ##[error]The version '3.10' with architecture 'x64' was not found for this operating system. The list of all available versions can be found here: https://raw.githubusercontent.com/actions/python-versions/main/versions-manifest.json 2025-12-04T09:02:37.0383115Z ##[group]Run pytorch/test-infra/.github/actions/teardown-linux@main 2025-12-04T09:02:37.0383476Z with: 2025-12-04T09:02:37.0383654Z env: 2025-12-04T09:02:37.0383848Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:02:37.0384078Z HAS_NVIDIA_GPU: false 2025-12-04T09:02:37.0384448Z DOCKER_CONTAINER_ID: f7efe3f2eca996d8a9efcd5effd738cd22ce08409dea3a9ea34cb5639faf9c86 2025-12-04T09:02:37.0384848Z DEVICE_NAME: 2025-12-04T09:02:37.0385038Z DEVICE_TYPE: 2025-12-04T09:02:37.0385250Z ##[endgroup] 2025-12-04T09:02:37.0470753Z ##[group]Run set -eou pipefail 2025-12-04T09:02:37.0471033Z set -eou pipefail 2025-12-04T09:02:37.0471260Z  2025-12-04T09:02:37.0471550Z echo "Holding runner for 2 hours until all ssh sessions have logged out" 2025-12-04T09:02:37.0471878Z for _ in $(seq 1440); do 2025-12-04T09:02:37.0472119Z  # Break if no ssh session exists anymore 2025-12-04T09:02:37.0472367Z  if [ "$(who)" = "" ]; then 2025-12-04T09:02:37.0472691Z  break 2025-12-04T09:02:37.0472874Z  fi 2025-12-04T09:02:37.0473053Z  echo "." 2025-12-04T09:02:37.0473239Z  sleep 5 2025-12-04T09:02:37.0473416Z done 2025-12-04T09:02:37.0478725Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:02:37.0479050Z env: 2025-12-04T09:02:37.0479227Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:02:37.0479437Z HAS_NVIDIA_GPU: false 2025-12-04T09:02:37.0479809Z DOCKER_CONTAINER_ID: f7efe3f2eca996d8a9efcd5effd738cd22ce08409dea3a9ea34cb5639faf9c86 2025-12-04T09:02:37.0480189Z DEVICE_NAME: 2025-12-04T09:02:37.0480372Z DEVICE_TYPE: 2025-12-04T09:02:37.0480555Z ##[endgroup] 2025-12-04T09:02:37.0505213Z Holding runner for 2 hours until all ssh sessions have logged out 2025-12-04T09:02:37.0604223Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2025-12-04T09:02:37.0604601Z # ignore expansion of "docker ps -q" since it could be empty 2025-12-04T09:02:37.0604906Z # shellcheck disable=SC2046 2025-12-04T09:02:37.0605150Z docker stop $(docker ps -q) || true 2025-12-04T09:02:37.0605389Z # Prune all of the docker images 2025-12-04T09:02:37.0605625Z docker system prune -af 2025-12-04T09:02:37.0609618Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:02:37.0609875Z env: 2025-12-04T09:02:37.0610043Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:02:37.0610248Z HAS_NVIDIA_GPU: false 2025-12-04T09:02:37.0610604Z DOCKER_CONTAINER_ID: f7efe3f2eca996d8a9efcd5effd738cd22ce08409dea3a9ea34cb5639faf9c86 2025-12-04T09:02:37.0610960Z DEVICE_NAME: 2025-12-04T09:02:37.0611133Z DEVICE_TYPE: 2025-12-04T09:02:37.0611299Z ##[endgroup] 2025-12-04T09:02:47.6178989Z f7efe3f2eca9 2025-12-04T09:02:48.3025579Z Deleted Containers: 2025-12-04T09:02:48.3026000Z f7efe3f2eca996d8a9efcd5effd738cd22ce08409dea3a9ea34cb5639faf9c86 2025-12-04T09:02:48.3026266Z 2025-12-04T09:03:16.7675863Z Deleted Images: 2025-12-04T09:03:16.7678880Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-f0cd68561080d537ef3d3d6f81b25a6416ad600a 2025-12-04T09:03:16.7679881Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image@sha256:b178ee928adfcab963ebbc6ea05ea1a0f0c605bb095e9076f010bf92d150869d 2025-12-04T09:03:16.7680496Z deleted: sha256:40bd34edc3497e05e2150804a15313235b6426bcc4b8c19c03c3daf5524be913 2025-12-04T09:03:16.7680915Z deleted: sha256:947006dce32e090f6b67d556f519398b957121651577f10819c7b179a5b6b938 2025-12-04T09:03:16.7707714Z deleted: sha256:dca16630fc5508e008012b760ac178fac5af74cb679706a3d9898100501d6f8e 2025-12-04T09:03:16.7708143Z deleted: sha256:a35e2e4cfaaa05e1f193fbe67c4151ded52737cb30c75429050cb7cd7777da23 2025-12-04T09:03:16.7708570Z deleted: sha256:d91e1a0a76e64abd7933fc58b8a553d76ac6cd82269c1bf701ade4e2c5ab58ff 2025-12-04T09:03:16.7709028Z deleted: sha256:f96fff40164ffa27ae535a4e5da2bbf2b18ec293366adc73519c830606f07baa 2025-12-04T09:03:16.7709601Z deleted: sha256:649089a3156b98c2cbf16b0af2b1ebdaa8644f9bf90534d9a1f7657b78b9f90e 2025-12-04T09:03:16.7710004Z deleted: sha256:aebabb7ea7535ad2aca44b23eb308b16d5de44f5b412574f94620d923436f42e 2025-12-04T09:03:16.7710397Z deleted: sha256:bcb08dc52ae55732fc33ab8c011347049a9279cc5f45a50d960c95d3a7cd5a3d 2025-12-04T09:03:16.7710771Z deleted: sha256:aa538cda5b22ea6e81fb69326637459a1356cce39b618657a075ce25d9124876 2025-12-04T09:03:16.7711161Z deleted: sha256:e28671ddaf02cbac7c1dcfdb2b08a40d3b7e2dab69c9a9e6c48ede3d8d550d66 2025-12-04T09:03:16.7711555Z deleted: sha256:ba797a4f759946d2fa7414073a5590cc12337aaa43197c35326ce456a3f84adf 2025-12-04T09:03:16.7711927Z deleted: sha256:25b28b66e2a53c3498a56aff9f543bf3a4641534c9e69741c673e9d018c652d8 2025-12-04T09:03:16.7712314Z deleted: sha256:860240198a7e7dcec697a58bcf979cfde64334033f2e97d4c9609277e051558c 2025-12-04T09:03:16.7712698Z deleted: sha256:39de54a9c331b8da952419605b26c35862b4cbb8c8a0967cc536a02b96eda874 2025-12-04T09:03:16.7713148Z deleted: sha256:6f78957316e0887707bc6534ea2aa43d1cd016130ec6a0d3148370b879696e1a 2025-12-04T09:03:16.7713530Z deleted: sha256:8656f7aa172ef2a37ec1524ac6f3ecd2b3fd9675a78a777b9a902d0f26047ea9 2025-12-04T09:03:16.7713926Z deleted: sha256:255344ff6c764e507ea2f21fbbdc6e0a1aaab0e5e0835935494c9df7d620e687 2025-12-04T09:03:16.7714319Z deleted: sha256:b6a1addb4370e1500d2fd65255b80922ad308c95a846a3460be730442d75fbe4 2025-12-04T09:03:16.7714709Z deleted: sha256:8dafa1eff3c9aada8b6725626ba0dd7de62719fb5ff307be3005cf2c6e72e2df 2025-12-04T09:03:16.7715119Z deleted: sha256:9e3fde92e8ff4f05cadac548766ef5ef9e8beb3c89cb48b4e2a17843ff60d649 2025-12-04T09:03:16.7715512Z deleted: sha256:7169a6f70d1f624733e52107b4913effe35e1e63c99388164b0afd2345658178 2025-12-04T09:03:16.7715900Z deleted: sha256:4c0d4c67766b690c1f1834310cd3898a900cd9742b7157de7e8f64d651600e35 2025-12-04T09:03:16.7716277Z deleted: sha256:7343a9c5c68976b0328278e691ddbcf26f619f989ca2a51933ac8bf637d1d69c 2025-12-04T09:03:16.7716667Z deleted: sha256:b790f29be6db3408a1a53fabdc69e76bb38cef221474a4f3aef1e233bb85d6d3 2025-12-04T09:03:16.7717057Z deleted: sha256:ed0831c232a6a410214e113695ab9288063623b523a6032a05b443fa3e9a68ae 2025-12-04T09:03:16.7717427Z deleted: sha256:554ed49d988a9f7a1381d193479542175c40d702a22d246ecef7958a8662a2eb 2025-12-04T09:03:16.7717833Z deleted: sha256:47b04d260bfb2b1f68a2f12aa70926ae2a2a53be6f4d8811d6fbe17fd7f13e17 2025-12-04T09:03:16.7718344Z deleted: sha256:6eb59e87109db04da60df057ff6b232045b6df4e2729ce7534a7a77fbe663761 2025-12-04T09:03:16.7718785Z deleted: sha256:0d9763e3d6e83eee92752c3a5dcfce74703dfa0aba42042761d7600edd95baa5 2025-12-04T09:03:16.7719213Z deleted: sha256:5811882cf3f14a50f47dc91a87d1be09daf3c20494aaf7e7343554066e9f90b7 2025-12-04T09:03:16.7719652Z deleted: sha256:4a6f4f545e9062949802a6c597609ef772633eae7d6925d41823d2b1a8991d4d 2025-12-04T09:03:16.7720036Z deleted: sha256:69a716cb9dc95d7d58d1416f4d127e49885c5d96882dc593686a6eb97200582a 2025-12-04T09:03:16.7720429Z deleted: sha256:0542610effe51f46308d4570438af559e4f6a263f38714ea04c78e4661d1909a 2025-12-04T09:03:16.7720952Z deleted: sha256:86dcdb716a38a00a400b483f42e4ac58714a70e89f5671b50ca5c45d8f577217 2025-12-04T09:03:16.7721360Z deleted: sha256:a9aaf6de8156c1b1a8ece816adbc7d2ed6edabb95e36732737ca293b26619b40 2025-12-04T09:03:16.7721771Z deleted: sha256:57ce7fcbaaa86795f8f7e6290c0aaa8622e13c9c82191d34952c0a4d2e203326 2025-12-04T09:03:16.7722169Z deleted: sha256:a90408b5f4c06a3ba881facc34fa55beb4ae4e51823e7f8d933e611f68465f4a 2025-12-04T09:03:16.7722589Z deleted: sha256:1ac1aab20bc98902f03ee9c657ff435af431f40a81ac9cebf1cd91d9041bc6b1 2025-12-04T09:03:16.7723020Z deleted: sha256:89e6c1c76b21fbe98c153a32627fa1b2337a8caa744b09b28253b67cb84012cb 2025-12-04T09:03:16.7723411Z deleted: sha256:3765d58e17aa81539a8e1eb3cb45914b48f8fb53f7ac12c1bf06c9390e274a93 2025-12-04T09:03:16.7723789Z deleted: sha256:6056a51594e8137d3425100cc233c8ab0f09dded2dc491ba083e8d7c0e1f1143 2025-12-04T09:03:16.7724177Z deleted: sha256:43a55512a68fc9eb9c622a73682dbf9f3c83f1b499287d9e12151f559edabe85 2025-12-04T09:03:16.7724635Z deleted: sha256:a5ad8e88a7e08768a61feeb801fb5a4c5041587feb2a2c77c35b285e869c25c9 2025-12-04T09:03:16.7725021Z deleted: sha256:55fd389078a5792f567a3592821e38fcfa0b0e114fa43248c4528c4d85cf6459 2025-12-04T09:03:16.7725409Z deleted: sha256:144567b58c97f1b5a75d54c3a1514a26ad5ed2dc7988e166316e324a76e76aa6 2025-12-04T09:03:16.7725797Z deleted: sha256:049fbb3123a92952eba68198938b7b29203fcb3d74dfc74e67f9929999576f19 2025-12-04T09:03:16.7726198Z deleted: sha256:22607d21e148d19a45f3f7afe927a15041b9a5663430f5580cc993c4bf9dd7dc 2025-12-04T09:03:16.7726575Z deleted: sha256:63a29311926064245f6d4a4661513586c8179b1cb460ceb91184c0d1114a742d 2025-12-04T09:03:16.7726957Z deleted: sha256:0072101bc0e6bbe2457f4a2d0de953ae643fc2aa6adc69bb59da9cbf6006c8a9 2025-12-04T09:03:16.7727359Z deleted: sha256:35c8f84104fcf39bc1be4682f46dc8b04cb1c5e4c26f66aac039cb42a3f08128 2025-12-04T09:03:16.7727755Z deleted: sha256:c6da65c085b8e72b2e240a3a7d79c4aa166c247737f5916587e51d40acff1ab8 2025-12-04T09:03:16.7728165Z deleted: sha256:b8b5ced5e8550748e26855aad0bb30864c4c435c64694097e63a9a686ef90ba0 2025-12-04T09:03:16.7728590Z deleted: sha256:9905f824759cfae96d29353305bc8a6130a7316db29dfc3efeb8ba9c7cbb5172 2025-12-04T09:03:16.7728991Z deleted: sha256:2b93a73a0fc57cdcfab2a04a30c9b313d84e5e447d076575212f198508242a00 2025-12-04T09:03:16.7729379Z deleted: sha256:baa73842b47530e5d1421010d895ed9059a567516a7e85b10943a6b5fe6fbe41 2025-12-04T09:03:16.7729754Z deleted: sha256:db79627a360d2322d58fb57c98268676011788961d4682ba034f17c9f5fdd855 2025-12-04T09:03:16.7730144Z deleted: sha256:8439f69d5eab9d42b15df43b938f56c690b18cd5e3884f4b80ddc9edf9e43822 2025-12-04T09:03:16.7730549Z deleted: sha256:e95ccdb10d43bbb6f2d6416bca8e5ad7d76a39985431aa5e40ff192ebeb52883 2025-12-04T09:03:16.7730939Z deleted: sha256:d878081c591fd25e8e494f4565e67f341cb3c459b21596ad8e8a707ff9ca63e6 2025-12-04T09:03:16.7731332Z deleted: sha256:2d594947cf73e58a20e36d52acf22f3d31b7a43291d7017c1ac71ea38c4aa9d1 2025-12-04T09:03:16.7731722Z deleted: sha256:933d315b5f20ac941653d01f2b896cc7710564c9b9c4bf9fd8e850115180c6f9 2025-12-04T09:03:16.7732121Z deleted: sha256:0edebbabed274b9e777a5933aaa21ed58569a412c97423c365eff4645e773981 2025-12-04T09:03:16.7732513Z deleted: sha256:744e1cebe181659b0a93ac857d4c16af4c2ce29f3227368eacd7ef18b69e06a7 2025-12-04T09:03:16.7732907Z deleted: sha256:71393f74ab60b183866b9f5995f85aaa65613968c611c8dc15d043befbac662b 2025-12-04T09:03:16.7733300Z deleted: sha256:741a07a29d03bdecf1fb4dfc8f1654a136064f14e5ded89a1497f476ff24623d 2025-12-04T09:03:16.7733690Z deleted: sha256:83df0b5b82e0926d6046012c3b5098dae9aa37177323a33d6eda09e392844b86 2025-12-04T09:03:16.7734083Z deleted: sha256:65b3d6509a37164c67b36965f4ca77ccb1a8e62bdd8f058e2d0886c82ba0940f 2025-12-04T09:03:16.7734488Z deleted: sha256:a38d6cb5dacc51c3e614f4ed42c05cdc53fc489f6e3dafc3b7d0eb39d3250908 2025-12-04T09:03:16.7734892Z deleted: sha256:c3de4f46ac5273a8218c6c5325247b0bd17f12037cfb49f5101a2b5085746704 2025-12-04T09:03:16.7735279Z deleted: sha256:7f3448d255710b28f79767ded93dcf11e0c1a941c7579d8752f4f0bc589aa0b5 2025-12-04T09:03:16.7735671Z deleted: sha256:fd33fe89098b4a8897870a9a316cd91868ab19f42f800d5b512e679356791e62 2025-12-04T09:03:16.7736082Z deleted: sha256:df2c12f62f5ceb2670bf7cb8aa53f268d11dbbfbf25f4b4d567673702cc3fc30 2025-12-04T09:03:16.7736512Z deleted: sha256:a3d7900a6180e4612708429f9c0edba3e999bb43fd67fd3406ae088ea7016a47 2025-12-04T09:03:16.7736899Z deleted: sha256:f592639a1c1bf8da30a1c21866ee22e76b33ae029ee236c7fd20b1ecc4716775 2025-12-04T09:03:16.7737293Z deleted: sha256:e11e4a9756b2bd4a6e62cde476dc36c6528d71a74e8e7570b271784bc494ab40 2025-12-04T09:03:16.7737709Z deleted: sha256:4c03ab513fc4cf698dd61fc37312e57abf7c4e44d4d4ddab126f38804fddec75 2025-12-04T09:03:16.7738124Z deleted: sha256:b4d3d8b5232e0bac87fa78b9c77997cbe8798faffce2aac259b820ab387ce4ad 2025-12-04T09:03:16.7738551Z deleted: sha256:dfda7f6e86c7217a3eb1fe089352248b5d84ca4f62734abce70beb4a6fc90779 2025-12-04T09:03:16.7738965Z deleted: sha256:00f41b846c66bc26a6381fe3663c611a2f539ae3b7e46e007436afd3335a7517 2025-12-04T09:03:16.7739409Z deleted: sha256:0f35c2638249a4525b29668586b079b4ee01f73bbe33732c9da7cfa7b6afb480 2025-12-04T09:03:16.7739812Z deleted: sha256:7441da072ac3269ddb67441b265fdf82904871335b5c19bb381d3622a95b7745 2025-12-04T09:03:16.7740195Z deleted: sha256:ba70851a3aedb994e8d7f8657f8f108bd90a19272843f389e6a8af48a61db720 2025-12-04T09:03:16.7740585Z deleted: sha256:1984ce6f5cabbd8c86386a8d9b5365923b92bc7ccb9aace9b4fe9570213bbbe1 2025-12-04T09:03:16.7740970Z deleted: sha256:73974f74b436f39a2fdb6461b1e3f7c3e41c73325776fa71d16b942a5b4a365b 2025-12-04T09:03:16.7741204Z 2025-12-04T09:03:16.7789505Z Total reclaimed space: 53.33GB 2025-12-04T09:03:16.7867556Z Post job cleanup. 2025-12-04T09:03:16.8108954Z Post job cleanup. 2025-12-04T09:03:16.9152705Z (node:52597) [DEP0040] DeprecationWarning: The `punycode` module is deprecated. Please use a userland alternative instead. 2025-12-04T09:03:16.9154337Z (Use `node --trace-deprecation ...` to show where the warning was created) 2025-12-04T09:03:16.9326136Z Post job cleanup. 2025-12-04T09:03:16.9392884Z Post job cleanup. 2025-12-04T09:03:17.0270621Z [command]/usr/bin/git version 2025-12-04T09:03:17.0311183Z git version 2.50.1 2025-12-04T09:03:17.0348723Z Copying '/home/ec2-user/.gitconfig' to '/home/ec2-user/actions-runner/_work/_temp/19f890e0-3178-4dff-a7e8-77bad5ddebe2/.gitconfig' 2025-12-04T09:03:17.0375183Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/19f890e0-3178-4dff-a7e8-77bad5ddebe2' before making global git config changes 2025-12-04T09:03:17.0376098Z Adding repository directory to the temporary git global config as a safe directory 2025-12-04T09:03:17.0385792Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2025-12-04T09:03:17.0442235Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2025-12-04T09:03:17.0476274Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2025-12-04T09:03:17.0879660Z Entering 'android/libs/fbjni' 2025-12-04T09:03:17.0941113Z Entering 'third_party/FP16' 2025-12-04T09:03:17.0999497Z Entering 'third_party/FXdiv' 2025-12-04T09:03:17.1050822Z Entering 'third_party/NNPACK' 2025-12-04T09:03:17.1108681Z Entering 'third_party/NVTX' 2025-12-04T09:03:17.1163998Z Entering 'third_party/VulkanMemoryAllocator' 2025-12-04T09:03:17.1212407Z Entering 'third_party/XNNPACK' 2025-12-04T09:03:17.1285693Z Entering 'third_party/aiter' 2025-12-04T09:03:17.1337520Z Entering 'third_party/aiter/3rdparty/composable_kernel' 2025-12-04T09:03:17.1401464Z Entering 'third_party/benchmark' 2025-12-04T09:03:17.1466342Z Entering 'third_party/composable_kernel' 2025-12-04T09:03:17.1528160Z Entering 'third_party/cpp-httplib' 2025-12-04T09:03:17.1581246Z Entering 'third_party/cpuinfo' 2025-12-04T09:03:17.1634188Z Entering 'third_party/cudnn_frontend' 2025-12-04T09:03:17.1692444Z Entering 'third_party/cutlass' 2025-12-04T09:03:17.1757304Z Entering 'third_party/fbgemm' 2025-12-04T09:03:17.1815451Z Entering 'third_party/fbgemm/external/asmjit' 2025-12-04T09:03:17.1863406Z Entering 'third_party/fbgemm/external/composable_kernel' 2025-12-04T09:03:17.1918395Z Entering 'third_party/fbgemm/external/cpuinfo' 2025-12-04T09:03:17.1973151Z Entering 'third_party/fbgemm/external/cutlass' 2025-12-04T09:03:17.2033446Z Entering 'third_party/fbgemm/external/googletest' 2025-12-04T09:03:17.2086122Z Entering 'third_party/fbgemm/external/hipify_torch' 2025-12-04T09:03:17.2142484Z Entering 'third_party/fbgemm/external/json' 2025-12-04T09:03:17.2196990Z Entering 'third_party/flash-attention' 2025-12-04T09:03:17.2254200Z Entering 'third_party/flash-attention/csrc/composable_kernel' 2025-12-04T09:03:17.2315370Z Entering 'third_party/flash-attention/csrc/cutlass' 2025-12-04T09:03:17.2378034Z Entering 'third_party/flatbuffers' 2025-12-04T09:03:17.2434363Z Entering 'third_party/fmt' 2025-12-04T09:03:17.2487932Z Entering 'third_party/gemmlowp/gemmlowp' 2025-12-04T09:03:17.2544505Z Entering 'third_party/gloo' 2025-12-04T09:03:17.2606569Z Entering 'third_party/googletest' 2025-12-04T09:03:17.2663170Z Entering 'third_party/ideep' 2025-12-04T09:03:17.2719783Z Entering 'third_party/ideep/mkl-dnn' 2025-12-04T09:03:17.2787348Z Entering 'third_party/ittapi' 2025-12-04T09:03:17.2838975Z Entering 'third_party/kineto' 2025-12-04T09:03:17.2897363Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2025-12-04T09:03:17.2952298Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2025-12-04T09:03:17.3011668Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2025-12-04T09:03:17.3065656Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2025-12-04T09:03:17.3114287Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2025-12-04T09:03:17.3170629Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2025-12-04T09:03:17.3226704Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2025-12-04T09:03:17.3288712Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2025-12-04T09:03:17.3340828Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2025-12-04T09:03:17.3394563Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2025-12-04T09:03:17.3455613Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp' 2025-12-04T09:03:17.3506408Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/civetweb' 2025-12-04T09:03:17.3564450Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/googletest' 2025-12-04T09:03:17.3632781Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2025-12-04T09:03:17.3686694Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2025-12-04T09:03:17.3741128Z Entering 'third_party/kleidiai' 2025-12-04T09:03:17.3799946Z Entering 'third_party/mimalloc' 2025-12-04T09:03:17.3858953Z Entering 'third_party/nlohmann' 2025-12-04T09:03:17.3914197Z Entering 'third_party/onnx' 2025-12-04T09:03:17.3988409Z Entering 'third_party/onnx/third_party/pybind11' 2025-12-04T09:03:17.4045626Z Entering 'third_party/opentelemetry-cpp' 2025-12-04T09:03:17.4105443Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2025-12-04T09:03:17.4156896Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2025-12-04T09:03:17.4215754Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2025-12-04T09:03:17.4271175Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2025-12-04T09:03:17.4320136Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2025-12-04T09:03:17.4378395Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2025-12-04T09:03:17.4432657Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2025-12-04T09:03:17.4482532Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2025-12-04T09:03:17.4540725Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2025-12-04T09:03:17.4596644Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2025-12-04T09:03:17.4667438Z Entering 'third_party/pocketfft' 2025-12-04T09:03:17.4727758Z Entering 'third_party/protobuf' 2025-12-04T09:03:17.4782808Z Entering 'third_party/protobuf/third_party/benchmark' 2025-12-04T09:03:17.4839017Z Entering 'third_party/protobuf/third_party/googletest' 2025-12-04T09:03:17.4896798Z Entering 'third_party/psimd' 2025-12-04T09:03:17.4949024Z Entering 'third_party/pthreadpool' 2025-12-04T09:03:17.5005156Z Entering 'third_party/pybind11' 2025-12-04T09:03:17.5058071Z Entering 'third_party/python-peachpy' 2025-12-04T09:03:17.5113694Z Entering 'third_party/sleef' 2025-12-04T09:03:17.5178395Z Entering 'third_party/tensorpipe' 2025-12-04T09:03:17.5234018Z Entering 'third_party/tensorpipe/third_party/googletest' 2025-12-04T09:03:17.5288097Z Entering 'third_party/tensorpipe/third_party/libnop' 2025-12-04T09:03:17.5337383Z Entering 'third_party/tensorpipe/third_party/libuv' 2025-12-04T09:03:17.5393293Z Entering 'third_party/tensorpipe/third_party/pybind11' 2025-12-04T09:03:17.5445568Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2025-12-04T09:03:17.5526651Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2025-12-04T09:03:17.5556320Z http.https://github.com/.extraheader 2025-12-04T09:03:17.5567479Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader 2025-12-04T09:03:17.5596408Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2025-12-04T09:03:17.5907031Z Entering 'android/libs/fbjni' 2025-12-04T09:03:17.5941960Z http.https://github.com/.extraheader 2025-12-04T09:03:17.5980405Z Entering 'third_party/FP16' 2025-12-04T09:03:17.6009952Z http.https://github.com/.extraheader 2025-12-04T09:03:17.6046318Z Entering 'third_party/FXdiv' 2025-12-04T09:03:17.6082248Z http.https://github.com/.extraheader 2025-12-04T09:03:17.6116744Z Entering 'third_party/NNPACK' 2025-12-04T09:03:17.6152133Z http.https://github.com/.extraheader 2025-12-04T09:03:17.6191276Z Entering 'third_party/NVTX' 2025-12-04T09:03:17.6227821Z http.https://github.com/.extraheader 2025-12-04T09:03:17.6269684Z Entering 'third_party/VulkanMemoryAllocator' 2025-12-04T09:03:17.6304666Z http.https://github.com/.extraheader 2025-12-04T09:03:17.6338004Z Entering 'third_party/XNNPACK' 2025-12-04T09:03:17.6372323Z http.https://github.com/.extraheader 2025-12-04T09:03:17.6418029Z Entering 'third_party/aiter' 2025-12-04T09:03:17.6453950Z http.https://github.com/.extraheader 2025-12-04T09:03:17.6488491Z Entering 'third_party/aiter/3rdparty/composable_kernel' 2025-12-04T09:03:17.6519049Z http.https://github.com/.extraheader 2025-12-04T09:03:17.6566662Z Entering 'third_party/benchmark' 2025-12-04T09:03:17.6599214Z http.https://github.com/.extraheader 2025-12-04T09:03:17.6635174Z Entering 'third_party/composable_kernel' 2025-12-04T09:03:17.6671467Z http.https://github.com/.extraheader 2025-12-04T09:03:17.6714345Z Entering 'third_party/cpp-httplib' 2025-12-04T09:03:17.6754616Z http.https://github.com/.extraheader 2025-12-04T09:03:17.6786995Z Entering 'third_party/cpuinfo' 2025-12-04T09:03:17.6824507Z http.https://github.com/.extraheader 2025-12-04T09:03:17.6861085Z Entering 'third_party/cudnn_frontend' 2025-12-04T09:03:17.6897228Z http.https://github.com/.extraheader 2025-12-04T09:03:17.6935410Z Entering 'third_party/cutlass' 2025-12-04T09:03:17.6967431Z http.https://github.com/.extraheader 2025-12-04T09:03:17.7011118Z Entering 'third_party/fbgemm' 2025-12-04T09:03:17.7045965Z http.https://github.com/.extraheader 2025-12-04T09:03:17.7080080Z Entering 'third_party/fbgemm/external/asmjit' 2025-12-04T09:03:17.7113886Z http.https://github.com/.extraheader 2025-12-04T09:03:17.7153783Z Entering 'third_party/fbgemm/external/composable_kernel' 2025-12-04T09:03:17.7184458Z http.https://github.com/.extraheader 2025-12-04T09:03:17.7226186Z Entering 'third_party/fbgemm/external/cpuinfo' 2025-12-04T09:03:17.7266111Z http.https://github.com/.extraheader 2025-12-04T09:03:17.7301850Z Entering 'third_party/fbgemm/external/cutlass' 2025-12-04T09:03:17.7340066Z http.https://github.com/.extraheader 2025-12-04T09:03:17.7385238Z Entering 'third_party/fbgemm/external/googletest' 2025-12-04T09:03:17.7418040Z http.https://github.com/.extraheader 2025-12-04T09:03:17.7457585Z Entering 'third_party/fbgemm/external/hipify_torch' 2025-12-04T09:03:17.7489258Z http.https://github.com/.extraheader 2025-12-04T09:03:17.7521740Z Entering 'third_party/fbgemm/external/json' 2025-12-04T09:03:17.7555523Z http.https://github.com/.extraheader 2025-12-04T09:03:17.7596501Z Entering 'third_party/flash-attention' 2025-12-04T09:03:17.7632043Z http.https://github.com/.extraheader 2025-12-04T09:03:17.7667985Z Entering 'third_party/flash-attention/csrc/composable_kernel' 2025-12-04T09:03:17.7701332Z http.https://github.com/.extraheader 2025-12-04T09:03:17.7741057Z Entering 'third_party/flash-attention/csrc/cutlass' 2025-12-04T09:03:17.7780829Z http.https://github.com/.extraheader 2025-12-04T09:03:17.7829988Z Entering 'third_party/flatbuffers' 2025-12-04T09:03:17.7871237Z http.https://github.com/.extraheader 2025-12-04T09:03:17.7918509Z Entering 'third_party/fmt' 2025-12-04T09:03:17.7956219Z http.https://github.com/.extraheader 2025-12-04T09:03:17.7995420Z Entering 'third_party/gemmlowp/gemmlowp' 2025-12-04T09:03:17.8030623Z http.https://github.com/.extraheader 2025-12-04T09:03:17.8063220Z Entering 'third_party/gloo' 2025-12-04T09:03:17.8098888Z http.https://github.com/.extraheader 2025-12-04T09:03:17.8134859Z Entering 'third_party/googletest' 2025-12-04T09:03:17.8168780Z http.https://github.com/.extraheader 2025-12-04T09:03:17.8203075Z Entering 'third_party/ideep' 2025-12-04T09:03:17.8243255Z http.https://github.com/.extraheader 2025-12-04T09:03:17.8280807Z Entering 'third_party/ideep/mkl-dnn' 2025-12-04T09:03:17.8313109Z http.https://github.com/.extraheader 2025-12-04T09:03:17.8363746Z Entering 'third_party/ittapi' 2025-12-04T09:03:17.8393905Z http.https://github.com/.extraheader 2025-12-04T09:03:17.8430723Z Entering 'third_party/kineto' 2025-12-04T09:03:17.8467340Z http.https://github.com/.extraheader 2025-12-04T09:03:17.8503061Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2025-12-04T09:03:17.8536015Z http.https://github.com/.extraheader 2025-12-04T09:03:17.8571915Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2025-12-04T09:03:17.8603157Z http.https://github.com/.extraheader 2025-12-04T09:03:17.8639457Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2025-12-04T09:03:17.8676935Z http.https://github.com/.extraheader 2025-12-04T09:03:17.8711834Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2025-12-04T09:03:17.8746982Z http.https://github.com/.extraheader 2025-12-04T09:03:17.8787902Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2025-12-04T09:03:17.8818930Z http.https://github.com/.extraheader 2025-12-04T09:03:17.8853153Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2025-12-04T09:03:17.8884880Z http.https://github.com/.extraheader 2025-12-04T09:03:17.8916012Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2025-12-04T09:03:17.8951356Z http.https://github.com/.extraheader 2025-12-04T09:03:17.8989787Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2025-12-04T09:03:17.9026038Z http.https://github.com/.extraheader 2025-12-04T09:03:17.9062788Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2025-12-04T09:03:17.9093259Z http.https://github.com/.extraheader 2025-12-04T09:03:17.9128643Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2025-12-04T09:03:17.9160462Z http.https://github.com/.extraheader 2025-12-04T09:03:17.9200268Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp' 2025-12-04T09:03:17.9232890Z http.https://github.com/.extraheader 2025-12-04T09:03:17.9264524Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/civetweb' 2025-12-04T09:03:17.9303641Z http.https://github.com/.extraheader 2025-12-04T09:03:17.9338722Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/googletest' 2025-12-04T09:03:17.9371333Z http.https://github.com/.extraheader 2025-12-04T09:03:17.9404923Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2025-12-04T09:03:17.9440210Z http.https://github.com/.extraheader 2025-12-04T09:03:17.9476962Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2025-12-04T09:03:17.9513792Z http.https://github.com/.extraheader 2025-12-04T09:03:17.9553824Z Entering 'third_party/kleidiai' 2025-12-04T09:03:17.9591821Z http.https://github.com/.extraheader 2025-12-04T09:03:17.9629578Z Entering 'third_party/mimalloc' 2025-12-04T09:03:17.9664079Z http.https://github.com/.extraheader 2025-12-04T09:03:17.9705801Z Entering 'third_party/nlohmann' 2025-12-04T09:03:17.9740001Z http.https://github.com/.extraheader 2025-12-04T09:03:17.9784314Z Entering 'third_party/onnx' 2025-12-04T09:03:17.9814586Z http.https://github.com/.extraheader 2025-12-04T09:03:17.9867385Z Entering 'third_party/onnx/third_party/pybind11' 2025-12-04T09:03:17.9899996Z http.https://github.com/.extraheader 2025-12-04T09:03:17.9942921Z Entering 'third_party/opentelemetry-cpp' 2025-12-04T09:03:17.9977947Z http.https://github.com/.extraheader 2025-12-04T09:03:18.0012759Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2025-12-04T09:03:18.0051690Z http.https://github.com/.extraheader 2025-12-04T09:03:18.0088476Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2025-12-04T09:03:18.0118473Z http.https://github.com/.extraheader 2025-12-04T09:03:18.0154596Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2025-12-04T09:03:18.0190689Z http.https://github.com/.extraheader 2025-12-04T09:03:18.0221891Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2025-12-04T09:03:18.0257780Z http.https://github.com/.extraheader 2025-12-04T09:03:18.0292024Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2025-12-04T09:03:18.0331675Z http.https://github.com/.extraheader 2025-12-04T09:03:18.0371218Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2025-12-04T09:03:18.0402137Z http.https://github.com/.extraheader 2025-12-04T09:03:18.0438376Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2025-12-04T09:03:18.0470273Z http.https://github.com/.extraheader 2025-12-04T09:03:18.0503141Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2025-12-04T09:03:18.0546987Z http.https://github.com/.extraheader 2025-12-04T09:03:18.0586080Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2025-12-04T09:03:18.0624175Z http.https://github.com/.extraheader 2025-12-04T09:03:18.0660917Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2025-12-04T09:03:18.0692838Z http.https://github.com/.extraheader 2025-12-04T09:03:18.0747380Z Entering 'third_party/pocketfft' 2025-12-04T09:03:18.0782453Z http.https://github.com/.extraheader 2025-12-04T09:03:18.0819499Z Entering 'third_party/protobuf' 2025-12-04T09:03:18.0853546Z http.https://github.com/.extraheader 2025-12-04T09:03:18.0892103Z Entering 'third_party/protobuf/third_party/benchmark' 2025-12-04T09:03:18.0928226Z http.https://github.com/.extraheader 2025-12-04T09:03:18.0968613Z Entering 'third_party/protobuf/third_party/googletest' 2025-12-04T09:03:18.1005138Z http.https://github.com/.extraheader 2025-12-04T09:03:18.1044335Z Entering 'third_party/psimd' 2025-12-04T09:03:18.1085100Z http.https://github.com/.extraheader 2025-12-04T09:03:18.1115940Z Entering 'third_party/pthreadpool' 2025-12-04T09:03:18.1154316Z http.https://github.com/.extraheader 2025-12-04T09:03:18.1185002Z Entering 'third_party/pybind11' 2025-12-04T09:03:18.1223702Z http.https://github.com/.extraheader 2025-12-04T09:03:18.1266722Z Entering 'third_party/python-peachpy' 2025-12-04T09:03:18.1298847Z http.https://github.com/.extraheader 2025-12-04T09:03:18.1338838Z Entering 'third_party/sleef' 2025-12-04T09:03:18.1376534Z http.https://github.com/.extraheader 2025-12-04T09:03:18.1409178Z Entering 'third_party/tensorpipe' 2025-12-04T09:03:18.1452247Z http.https://github.com/.extraheader 2025-12-04T09:03:18.1490346Z Entering 'third_party/tensorpipe/third_party/googletest' 2025-12-04T09:03:18.1519262Z http.https://github.com/.extraheader 2025-12-04T09:03:18.1557070Z Entering 'third_party/tensorpipe/third_party/libnop' 2025-12-04T09:03:18.1594343Z http.https://github.com/.extraheader 2025-12-04T09:03:18.1629674Z Entering 'third_party/tensorpipe/third_party/libuv' 2025-12-04T09:03:18.1662704Z http.https://github.com/.extraheader 2025-12-04T09:03:18.1702396Z Entering 'third_party/tensorpipe/third_party/pybind11' 2025-12-04T09:03:18.1733533Z http.https://github.com/.extraheader 2025-12-04T09:03:18.1770126Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2025-12-04T09:03:18.1800855Z http.https://github.com/.extraheader 2025-12-04T09:03:18.1866685Z [command]/usr/bin/git config --local --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:03:18.1900632Z [command]/usr/bin/git submodule foreach --recursive git config --local --show-origin --name-only --get-regexp remote.origin.url 2025-12-04T09:03:18.2211199Z Entering 'android/libs/fbjni' 2025-12-04T09:03:18.2237266Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config remote.origin.url 2025-12-04T09:03:18.2257212Z Entering 'third_party/FP16' 2025-12-04T09:03:18.2283502Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config remote.origin.url 2025-12-04T09:03:18.2296862Z Entering 'third_party/FXdiv' 2025-12-04T09:03:18.2318269Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config remote.origin.url 2025-12-04T09:03:18.2336072Z Entering 'third_party/NNPACK' 2025-12-04T09:03:18.2355498Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config remote.origin.url 2025-12-04T09:03:18.2382355Z Entering 'third_party/NVTX' 2025-12-04T09:03:18.2402214Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NVTX/config remote.origin.url 2025-12-04T09:03:18.2419003Z Entering 'third_party/VulkanMemoryAllocator' 2025-12-04T09:03:18.2440876Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config remote.origin.url 2025-12-04T09:03:18.2464357Z Entering 'third_party/XNNPACK' 2025-12-04T09:03:18.2488601Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config remote.origin.url 2025-12-04T09:03:18.2516158Z Entering 'third_party/aiter' 2025-12-04T09:03:18.2540329Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/aiter/config remote.origin.url 2025-12-04T09:03:18.2557720Z Entering 'third_party/aiter/3rdparty/composable_kernel' 2025-12-04T09:03:18.2583764Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/aiter/modules/3rdparty/composable_kernel/config remote.origin.url 2025-12-04T09:03:18.2611059Z Entering 'third_party/benchmark' 2025-12-04T09:03:18.2631658Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config remote.origin.url 2025-12-04T09:03:18.2653347Z Entering 'third_party/composable_kernel' 2025-12-04T09:03:18.2673232Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/composable_kernel/config remote.origin.url 2025-12-04T09:03:18.2700276Z Entering 'third_party/cpp-httplib' 2025-12-04T09:03:18.2719408Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpp-httplib/config remote.origin.url 2025-12-04T09:03:18.2742417Z Entering 'third_party/cpuinfo' 2025-12-04T09:03:18.2767467Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config remote.origin.url 2025-12-04T09:03:18.2779167Z Entering 'third_party/cudnn_frontend' 2025-12-04T09:03:18.2801564Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config remote.origin.url 2025-12-04T09:03:18.2814449Z Entering 'third_party/cutlass' 2025-12-04T09:03:18.2836527Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config remote.origin.url 2025-12-04T09:03:18.2864300Z Entering 'third_party/fbgemm' 2025-12-04T09:03:18.2883118Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config remote.origin.url 2025-12-04T09:03:18.2902821Z Entering 'third_party/fbgemm/external/asmjit' 2025-12-04T09:03:18.2921245Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/asmjit/config remote.origin.url 2025-12-04T09:03:18.2935706Z Entering 'third_party/fbgemm/external/composable_kernel' 2025-12-04T09:03:18.2962570Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/composable_kernel/config remote.origin.url 2025-12-04T09:03:18.2986600Z Entering 'third_party/fbgemm/external/cpuinfo' 2025-12-04T09:03:18.3008581Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/cpuinfo/config remote.origin.url 2025-12-04T09:03:18.3025398Z Entering 'third_party/fbgemm/external/cutlass' 2025-12-04T09:03:18.3047894Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/cutlass/config remote.origin.url 2025-12-04T09:03:18.3075069Z Entering 'third_party/fbgemm/external/googletest' 2025-12-04T09:03:18.3096052Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/googletest/config remote.origin.url 2025-12-04T09:03:18.3112184Z Entering 'third_party/fbgemm/external/hipify_torch' 2025-12-04T09:03:18.3137838Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/hipify_torch/config remote.origin.url 2025-12-04T09:03:18.3155038Z Entering 'third_party/fbgemm/external/json' 2025-12-04T09:03:18.3178934Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/json/config remote.origin.url 2025-12-04T09:03:18.3197618Z Entering 'third_party/flash-attention' 2025-12-04T09:03:18.3223252Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flash-attention/config remote.origin.url 2025-12-04T09:03:18.3238747Z Entering 'third_party/flash-attention/csrc/composable_kernel' 2025-12-04T09:03:18.3261959Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flash-attention/modules/csrc/composable_kernel/config remote.origin.url 2025-12-04T09:03:18.3281671Z Entering 'third_party/flash-attention/csrc/cutlass' 2025-12-04T09:03:18.3303378Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flash-attention/modules/csrc/cutlass/config remote.origin.url 2025-12-04T09:03:18.3329159Z Entering 'third_party/flatbuffers' 2025-12-04T09:03:18.3355095Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config remote.origin.url 2025-12-04T09:03:18.3373153Z Entering 'third_party/fmt' 2025-12-04T09:03:18.3397854Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config remote.origin.url 2025-12-04T09:03:18.3414386Z Entering 'third_party/gemmlowp/gemmlowp' 2025-12-04T09:03:18.3440193Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config remote.origin.url 2025-12-04T09:03:18.3457919Z Entering 'third_party/gloo' 2025-12-04T09:03:18.3482084Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config remote.origin.url 2025-12-04T09:03:18.3499913Z Entering 'third_party/googletest' 2025-12-04T09:03:18.3523009Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config remote.origin.url 2025-12-04T09:03:18.3545796Z Entering 'third_party/ideep' 2025-12-04T09:03:18.3565651Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config remote.origin.url 2025-12-04T09:03:18.3580126Z Entering 'third_party/ideep/mkl-dnn' 2025-12-04T09:03:18.3602132Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config remote.origin.url 2025-12-04T09:03:18.3626557Z Entering 'third_party/ittapi' 2025-12-04T09:03:18.3652406Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config remote.origin.url 2025-12-04T09:03:18.3665609Z Entering 'third_party/kineto' 2025-12-04T09:03:18.3690332Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config remote.origin.url 2025-12-04T09:03:18.3707897Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2025-12-04T09:03:18.3735187Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/config remote.origin.url 2025-12-04T09:03:18.3748820Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2025-12-04T09:03:18.3769579Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/DCGM/config remote.origin.url 2025-12-04T09:03:18.3787627Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2025-12-04T09:03:18.3808083Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/cpr/config remote.origin.url 2025-12-04T09:03:18.3825157Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2025-12-04T09:03:18.3851576Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/fmt/config remote.origin.url 2025-12-04T09:03:18.3867086Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2025-12-04T09:03:18.3896149Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/config remote.origin.url 2025-12-04T09:03:18.3906272Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2025-12-04T09:03:18.3927306Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/modules/doc/config remote.origin.url 2025-12-04T09:03:18.3948053Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2025-12-04T09:03:18.3967693Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/glog/config remote.origin.url 2025-12-04T09:03:18.3988421Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2025-12-04T09:03:18.4008707Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/googletest/config remote.origin.url 2025-12-04T09:03:18.4024661Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2025-12-04T09:03:18.4049636Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/json/config remote.origin.url 2025-12-04T09:03:18.4066698Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2025-12-04T09:03:18.4093141Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/pfs/config remote.origin.url 2025-12-04T09:03:18.4111247Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp' 2025-12-04T09:03:18.4137967Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/prometheus-cpp/config remote.origin.url 2025-12-04T09:03:18.4148767Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/civetweb' 2025-12-04T09:03:18.4171378Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/prometheus-cpp/modules/civetweb/config remote.origin.url 2025-12-04T09:03:18.4188233Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/googletest' 2025-12-04T09:03:18.4212170Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/prometheus-cpp/modules/googletest/config remote.origin.url 2025-12-04T09:03:18.4239512Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2025-12-04T09:03:18.4259468Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config remote.origin.url 2025-12-04T09:03:18.4280113Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2025-12-04T09:03:18.4301904Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config remote.origin.url 2025-12-04T09:03:18.4320209Z Entering 'third_party/kleidiai' 2025-12-04T09:03:18.4344393Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kleidiai/config remote.origin.url 2025-12-04T09:03:18.4362807Z Entering 'third_party/mimalloc' 2025-12-04T09:03:18.4386691Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/mimalloc/config remote.origin.url 2025-12-04T09:03:18.4406980Z Entering 'third_party/nlohmann' 2025-12-04T09:03:18.4428989Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config remote.origin.url 2025-12-04T09:03:18.4453048Z Entering 'third_party/onnx' 2025-12-04T09:03:18.4470915Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config remote.origin.url 2025-12-04T09:03:18.4503588Z Entering 'third_party/onnx/third_party/pybind11' 2025-12-04T09:03:18.4524962Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config remote.origin.url 2025-12-04T09:03:18.4546538Z Entering 'third_party/opentelemetry-cpp' 2025-12-04T09:03:18.4568505Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/config remote.origin.url 2025-12-04T09:03:18.4590035Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2025-12-04T09:03:18.4609275Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/benchmark/config remote.origin.url 2025-12-04T09:03:18.4633526Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2025-12-04T09:03:18.4656655Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/googletest/config remote.origin.url 2025-12-04T09:03:18.4670311Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2025-12-04T09:03:18.4689913Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/ms-gsl/config remote.origin.url 2025-12-04T09:03:18.4706451Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2025-12-04T09:03:18.4725901Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/nlohmann-json/config remote.origin.url 2025-12-04T09:03:18.4742829Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2025-12-04T09:03:18.4768462Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentelemetry-proto/config remote.origin.url 2025-12-04T09:03:18.4787741Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2025-12-04T09:03:18.4807984Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentracing-cpp/config remote.origin.url 2025-12-04T09:03:18.4821230Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2025-12-04T09:03:18.4842618Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/config remote.origin.url 2025-12-04T09:03:18.4859825Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2025-12-04T09:03:18.4879711Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/civetweb/config remote.origin.url 2025-12-04T09:03:18.4904274Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2025-12-04T09:03:18.4921265Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/googletest/config remote.origin.url 2025-12-04T09:03:18.4938046Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2025-12-04T09:03:18.4959169Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/tools/vcpkg/config remote.origin.url 2025-12-04T09:03:18.4990844Z Entering 'third_party/pocketfft' 2025-12-04T09:03:18.5012276Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config remote.origin.url 2025-12-04T09:03:18.5028521Z Entering 'third_party/protobuf' 2025-12-04T09:03:18.5051253Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config remote.origin.url 2025-12-04T09:03:18.5066926Z Entering 'third_party/protobuf/third_party/benchmark' 2025-12-04T09:03:18.5092430Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config remote.origin.url 2025-12-04T09:03:18.5109087Z Entering 'third_party/protobuf/third_party/googletest' 2025-12-04T09:03:18.5129941Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config remote.origin.url 2025-12-04T09:03:18.5147903Z Entering 'third_party/psimd' 2025-12-04T09:03:18.5171288Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config remote.origin.url 2025-12-04T09:03:18.5185537Z Entering 'third_party/pthreadpool' 2025-12-04T09:03:18.5210107Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config remote.origin.url 2025-12-04T09:03:18.5233082Z Entering 'third_party/pybind11' 2025-12-04T09:03:18.5255490Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config remote.origin.url 2025-12-04T09:03:18.5273233Z Entering 'third_party/python-peachpy' 2025-12-04T09:03:18.5295994Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config remote.origin.url 2025-12-04T09:03:18.5315227Z Entering 'third_party/sleef' 2025-12-04T09:03:18.5335827Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config remote.origin.url 2025-12-04T09:03:18.5358258Z Entering 'third_party/tensorpipe' 2025-12-04T09:03:18.5379935Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config remote.origin.url 2025-12-04T09:03:18.5391200Z Entering 'third_party/tensorpipe/third_party/googletest' 2025-12-04T09:03:18.5415639Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config remote.origin.url 2025-12-04T09:03:18.5432703Z Entering 'third_party/tensorpipe/third_party/libnop' 2025-12-04T09:03:18.5462569Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config remote.origin.url 2025-12-04T09:03:18.5475228Z Entering 'third_party/tensorpipe/third_party/libuv' 2025-12-04T09:03:18.5501286Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config remote.origin.url 2025-12-04T09:03:18.5516930Z Entering 'third_party/tensorpipe/third_party/pybind11' 2025-12-04T09:03:18.5541627Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config remote.origin.url 2025-12-04T09:03:18.5558912Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2025-12-04T09:03:18.5580006Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config remote.origin.url 2025-12-04T09:03:18.5619488Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:03:18.5655242Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:03:18.5692301Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:03:18.5716250Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:03:18.5747980Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NVTX/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:03:18.5785267Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:03:18.5805368Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:03:18.5837312Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/aiter/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:03:18.5870716Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/aiter/modules/3rdparty/composable_kernel/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:03:18.5893225Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:03:18.5914861Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/composable_kernel/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:03:18.5945807Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpp-httplib/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:03:18.5981587Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:03:18.5993870Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:03:18.6014456Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:03:18.6041429Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:03:18.6077978Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/asmjit/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:03:18.6104385Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/composable_kernel/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:03:18.6130839Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/cpuinfo/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:03:18.6163291Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/cutlass/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:03:18.6194343Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/googletest/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:03:18.6213580Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/hipify_torch/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:03:18.6238751Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/json/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:03:18.6260455Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flash-attention/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:03:18.6289254Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flash-attention/modules/csrc/composable_kernel/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:03:18.6328170Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flash-attention/modules/csrc/cutlass/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:03:18.6357821Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:03:18.6385666Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:03:18.6406873Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:03:18.6430304Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:03:18.6460235Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:03:18.6499267Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:03:18.6518684Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:03:18.6546963Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:03:18.6582176Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:03:18.6607219Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:03:18.6636575Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/DCGM/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:03:18.6665823Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/cpr/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:03:18.6696731Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/fmt/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:03:18.6724426Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:03:18.6773096Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/modules/doc/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:03:18.6795165Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/glog/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:03:18.6814463Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/googletest/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:03:18.6842213Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/json/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:03:18.6866732Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/pfs/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:03:18.6897271Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/prometheus-cpp/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:03:18.6917952Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/prometheus-cpp/modules/civetweb/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:03:18.6947301Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/prometheus-cpp/modules/googletest/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:03:18.6980668Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:03:18.7009260Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:03:18.7035859Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kleidiai/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:03:18.7064917Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/mimalloc/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:03:18.7095140Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:03:18.7117972Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:03:18.7141542Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:03:18.7178654Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:03:18.7204031Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/benchmark/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:03:18.7226388Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/googletest/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:03:18.7252049Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/ms-gsl/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:03:18.7284461Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/nlohmann-json/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:03:18.7308818Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentelemetry-proto/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:03:18.7336027Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentracing-cpp/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:03:18.7371357Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:03:18.7404971Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/civetweb/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:03:18.7432159Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/googletest/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:03:18.7459164Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/tools/vcpkg/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:03:18.7490402Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:03:18.7513891Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:03:18.7540433Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:03:18.7571208Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:03:18.7597757Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:03:18.7626952Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:03:18.7655229Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:03:18.7681362Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:03:18.7710229Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:03:18.7741197Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:03:18.7774020Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:03:18.7800789Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:03:18.7831037Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:03:18.7863659Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:03:18.7886763Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T09:03:18.7999710Z A job completed hook has been configured by the self-hosted runner administrator 2025-12-04T09:03:18.8040390Z ##[group]Run '/home/ec2-user/runner-scripts/after_job.sh' 2025-12-04T09:03:18.8044912Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:03:18.8045211Z ##[endgroup] 2025-12-04T09:03:27.3304108Z Cleaning up orphan processes